WAS Issues and Solution

404 Page Not Found

500 Internal Server Error

Out Of Memory Exception

Connection TimeOut

Slow Response

Unable to Connect to DataBase

Stale Connection Exception

Deployment Related Issues

Server Hung/Creash

Component Isolation

Ping Webserver Machine whether it is up & running

if yes → Try to access IHS Home Page with http://localhost:80/

if No → Check whether IHS is running with ps -ef | grep httpd

If IHS not running Strat it 

if IHS is running Analyse log Files 

   <IHS_HOME>/logs/error.log , 

   <IHS_HOME>/logs/access.log

   <Plugin”_Home>/logs/<webserver>/http_plugin.log

Then Generate & Propagate plugin_cfg.xml

Ping Appserver Machine whether it is running or not

If No → Start it

Access Application using Proper url

Still Unable to access Application , Analyse Server logs and Application logs

Once Problematic Server is Identified

Collect SystemOut/Err.log files

Take Dumps with kill -3 <PID>

Note: For PID, ps -ef | grep java or from Console

Record CPU and Memory Utilization

Colect Output of netstat into a file using netstat -an | grep 9080 >> netstat9080.txt

Try to Google the Error Code

JDBC Connection Problems

Perform Test Connection

If Successful ;;

Failure might be because of failed JNDI Lookup

check SystemOut.log for NameNotFoundException

If True, the root cause of problem is that the JNDI Lookup of DataSource by Application Fails

Check the DataSource JNDI Name and Ensure it is bound to Application Server, when it starts

You can see it in SystemOut.log like below: WSVRR0049I: Binding datasource as jdbc/ds

Also Check Application Code and ensure that it is Looking Up the correct JNDI Name

If you are doing an indirect JNDI Lookup (i.e java:comp/env/jdbc/ds ), ensure that the binding is correct in the datasource resource reference

The Resource reference is configured in the Web, EJB or Application Deployment Descriptor

If Not ;;

Catch the Error Message shown in the Console when TestConnection Performed

a) Does the Error Message Contains, SQLException, Indicates problem with database credentials

If Yes, Check J2C Authentication Alias and correct UserID and Password

If Not , Check with DBA

b) NO SQLException

If you are using Type2 or Type3 JDBC Driver,

java.land.unsatisfiedlinkerror indicates JVM is unable to load a native library that is needed by the JDBC Driver, Ensure All Environment Entries are Set like ORACLE_HOME, LIBPATH

Try to Connect to the database server using a standalone java program to determine whether problem is with DataBase or WAS

a) If Direct JDBC Connection also fails, work with DBA to resolve the issue

b) If Successful Gather for DataBase Connection, and Raise PMR to IBM

Connection Wait Issue

a) Connection Slow

b) Application Connection Leak

c) Performance Issues related to Connection Pooling

If you see ConnectionWaitTimeOutException in SystemOut.log

→ Slower Performanceor Delays when application tries to get a conection

→ Threads hanging while waiting for a free connection ( This can be seen in JavaCore )

If Yes → Connection Pool is at its Maximum Size

Results All Of Connections are in use and the application requests more connections.

So, the application must wait for a free connection to become available

a) Does the Problem eventually clears up or persists until server restart

If the Problem clears up Eventually, the root cause of problem is likely that maximum connections is not set hight enough for the amount of load on the application

To Increase Value, Perform the load test. Use PMI and Monitor Connection pool counters in TPV for tuning

b) If it Persists until server restart, cause is application is not closing connections

Each App should Follow get/use/close

After Close each connection returns to free pool

If the Connection is leaked and never returns to pool, eventually the pool might become filed with leaked Connections causing ConnectionWait Problems

To determine application is leaking connections,

Enable Tracing, The Trace output shows whether a connection leak exists and it also shows what part of application code is leaking connections

PMI helps you to determine connection leaks

If the application is failing to call close() on all conection objects, that is root cause and fix the application code to resolve the issue

If No Connection Leak

The Connection Wait problem and slow application performance, when using connection pool occur when the WebContainer is not Configured Properly

If the webcontainer thread pool size is set too high relative to maximum connections of pool, resource contention for available connections could occur

It is also strongly recommended not to check isGrowable checkbox for webcontainer threadpool

Is WC Threadpool is set to high or Is isGrowable checkbox checked ?

If Yes → correct it & test to see if problem fixed

If No → Other Issues that cause wait Execution

Even If the get/use/close pattern is being followed by the app, connections will not be returned to the pol until trancation in which the connection obtaine is commited

This Frequently occurs in servlets, when a Sharable Connection is obtained in a local transaction Containment(LTC)

If you face any slow performance, without connection wait check SystemOut.log for WSVR0605W warinings,

If Yes → There is a thread that has been running as long as threshold defined in hang detection poloicy

generate javacores and analyse , why thread was hung.

If None of above occurs Raise PMR