Note: Taking only one or two Thread Dumps is not very useful. Always make sure that you collect at least 5-6 Thread Dumps taken in the interval of 10-12 Seconds….Which will be really helpful to investigate the Hang Situation.
In older versions of Linux, kill -3 does not work. In such cases kill -QUIT would be the best option to take thread dumps.
Thread Dump:
When we trigger a thread dump on an application server, all active thread information will be dumped into file which is normally known as thread dump. In this dump file, we can get information about thread like active thread, hung thread, dead lock, runnable and, inactive thread info.
Command to generate thread dump: kill -3 PID_OF_SERVER
when our application server reports the thread hung conditions in SystemOut.log file then we generate thread dump to analyze the reason behind thread hung. We are generally having thread hung situation when any threads coming to system becomes unresponded.
When Websphere Application Server detects that a thread has been active longer than the time defined by the thread monitor threshold then it write below error code in SystemOut.log file:
WSVR0605W: Thread threadname has been active for hangtime and may be hung. There are totalthreads threads in total in the server that may be hung.
Where: thread name is the name that appears in a JVM thread dump, hangtime gives an approximation of how long the thread has been active and totalthreads gives an overall assessment of the system threads.
If the above thread gets released by itself then we get below entry in our log file:
WSVR0606W: Thread threadname was previously reported to be hung but has completed. It was active for approximately hangtime. There are totalthreads threads in total in the server that still may be hung.
Note: If an application server spontaneously dies, look for a file. The JVM creates the file in the product directory structure, with a name like javacore[number].txt
What is a Thread Dump?
A thread represents a work item or task, such as a servlet request. Java processes are usually multi-threaded. This means there can be many tasks occurring simultaneously (multi-threading) within one JVM process.
Java thread dumps provide a Java view of a failing JVM process.
A Java dump provides information about the executing Java classes and allows the problem determination process to reference the Java source code.
When to generate ?
If you get unexplained server hangs under WebSphere, you can obtain, from the WebSphere server, a thread dump to help diagnose the problem.
In the case of a server hang, you can force an application to create a thread dump.
If an application server spontaneously dies, look for a file. The JVM creates the file in the product directory structure, with a name like javacore[number].txt.
How to generate ?
On unix/Linux machines find the process id (PID) of the hung JVM and issue kill -3 PID. Look for an output file in the installation root directory with a name like javacore.date.time.id.txt.
Using wasadminprompt:
get the handle of the server wsadmin>set jvm [$AdminControlcompleteObjectName type=JVM,process=server1,*]
then execute wsadmin>$AdminControl invoke $jvmdumpThreads
Analysis
Download thread analyzer from IBM website to analyze the generated thread dumps.
- http://www.alphaworks.ibm.com/tech/jca
Heap Dump:
A heap dump is a snapshot of JVM memory. It shows the live objects on the heap along with references between objects. It is used to determine memory usage patterns and memory leak suspects.
Heap dump is a text file which keeps record of all objects in the java heap. It contains the information like size and address of all objects, as well as the addresses of all objects that it references.
Before generating the heap dump, we need to enable the garbage collection to get only live objects information in dump.
By Default, whenever OutOfMemoryException occurs Websphere application Server create heap dump in profile's home directory.
Heap dump is nothing but a snap shot of our JVM’s Heap memory, so here we can have all active java objects and their activities which object doing what and how much memory it is occupying.
Commands to generate heap dump and core dump:
For getting handle:
set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
Where server1 is the name of application server for which we want heapdump.
For generating heap dump:
$AdminControl invoke $jvm generateHeapDump
For generating Core dump:
$AdminControl invoke $jvm dumpThreads
Generate heap dump automatically?
To enable automated heap dump generation support, perform the following steps in the administrative console: (heap dump will be generated upon receiving the out.of.memoryexceptios)
1. Click Servers > Application servers in the administrative console navigation tree. 2. Click server_name >Runtime Performance Advisor Configuration. 3. Click the Runtime tab. 4. Select the Enable automatic heap dump collection check box. 5. Click OK.
To generate manually
kill -3 PID on unix/Linux machines. heap dumps can be found under profile_rootprofile-name with naming heapdump. <date>.<timestamp><pid>.phd
Use HeapAnalyzer or MDD4J for analyzing heap dumps
2). Heap Dump
A heapdump is a snapshot of JVM memory. It shows the live objects on the heap along with references between objects. It is used to determine memory usage patterns and memory leak suspects.
When to generate?
Memory leaks in the Java heap produce java.lang.OutOfMemoryError exceptions in log files. However, not all out-of-memory errors are caused by Java heap memory leaks. Out-of-memory errors can also be caused by the following conditions:
Java heap fragmentation. This fragmentation occurs when no contiguous chunk of free Java heap space is available from which to allocate Java objects. Various causes for this problem exist, including the presence of pinned or dosed objects or because of the repeated allocation of large objects.
Memory leaks in native heap. This problem occurs when a native component, like DB2 connections, is leaking.
How to generate?
To enable automated heap dump generation support, perform the following steps in the administrative console: (heap dump will be generated upon receiving the out.of.memoryexceptios)
1. Click Servers > Application servers in the administrative console navigation tree. 2. Click server_name >Runtime Performance Advisor Configuration. 3. Click the Runtime tab. 4. Select the Enable automatic heap dump collection check box. 5. Click OK.
To generate manually
use kill -3 PID on unix/Linux machines.
Using WSADMIN tool
Invoke the generateHeapDump operation on a JVMMBean, for example,
* Finding JVMobjectName: <wsadmin> set objectName [$AdminControlqueryNamesWebSphere:type=JVM,process=<servername>,node=<nodename>,*]
* Invoking the generateHeapDump operation on JVMMBean: <wsadmin> $AdminControl invoke $objectNamegenerateHeapDump
Analysis
heap dumps can be found under profile_rootprofile-name with naming heapdump. <date>.<timestamp><pid>.phd
Use HeapAnalyzer or MDD4J for analyzing heap dumps
++++++++++++++++++++
What is the actual difference between heap dump, thread dump and core dump and in which situations we will take these dumps?
Heap dump - Collection of objects that are in memory (JVM)
Thread dump - Shows what each thread in a process is doing at a given point in time along with the stack trace.
Core dump - O/S level dump file which has O/S level info in addition to the heap dump.
Heap dump - Is useful to analyze OOM situations.
Thread dump - To troubleshoot slow running of your application.
Core dump - When your JVM has crashed abruptly. To find details about native calls and so on.
Could you please elaborate about core dump? How to take core dump, when server was crashed?
Core dump will be generated automatically if a program crashed due to segmentation fault or some other reason. But core dump will not be generated if an application halts and been terminate by control or conventional way of kill or pkill.
In order to force a core dump generation you could run kill -6 <PID>
You should also make sure that you have enabled it and have provided enough space and permissions for it to get generated.
For Linux, the core dump is turned off by default on all systems. For Red Hat Advanced Server 2.1, it should be under /etc/security. There should be a self-explanatory file called limits.conf and look for the word "core". If set to "0", then core dump is disabled.
For Solaris, you can also make sure core files are enabled with the coreadm command.
Check ulimit -a to see whether your environment allows core files to be produced.
ulimit -c (This is the size limit of the core file. Fix it with ulimit -c unlimited).
Kernel limitation (hard limit for ulimit -c).
Available disk space for the user (e.g., is there disk quota?).
There are different types of core dumps available and the process to take core dump varies from OS to OS. See below article for Solaris
http://developers.sun.com/solaris/articles/manage_core_dump.html
###################################
Automatic Heapdump
``````````````````````````````
Goto Servers --> ApplicationServers --> server1 --> Java & Process Management --> Process Definition --> Environment Variables
Set Following Properties ;
IBM_HEAPDUMP = true
IBM_HEAP_DUMP = true
IBM_JAVA_HEAPDUMP_TEXT = true
IBM_HEAPDUMP_OUTOFMEMORY = false
JAVA_DUMP_OPTS = ONANYSIGNAL(JAVADUMP[5], HEAPDUMP[5])
Whenever kill -3 <PID> executed, it will generate a maximum of 5 heapdumps & 5 Threaddumps.
Note: We can specify the path of dumps also.
No comments:
Post a Comment