Diagnosing OutOfMemoryError or Memory Leaks in JRockit

When you run into OutOfMemoryError or other memory-leak issues, generating a heap histogram or heap dump can help you diagnose the memory-bloating issues.

In [1], it lists the following Java Heap related problems:

• Exceeding max heap 
• The heap is full and cannot fit a new object
• Large allocation
• The new object is too large for the contiguous free space
• Native exhaustion
• There is not enough native heap for the requested object^[10]
• GC Starvation
• The heap is almost full and causing frequent garbage collection^[9]
• Optimization
• Heap utilization is higher than expected for the current number of users

In this article, we will discuss the following topics:

1. Heap histogram vs. heap dump (see also [8])
2. How to generate heap histogram or heap dump in JRockit
3. JVM options that are useful for heap analysis

Heap Histogram vs. Heap Dump

A heap dump is a snapshot of all the objects in the Java Virtual Machine (JVM) heap at a certain point in time. The JVM software allocates memory for objects from the heap for all class instances and arrays. The garbage collector reclaims the heap memory when an object is no longer needed and there are no references to the object. By examining the heap you can locate where objects are created and find the references to those objects in the source.  However, dumping of Java heap is time-consuming and lengthy in size.

On the other hand, heap histogram gives a very good summary of heap objects used in the application without doing a full heap dump. It can help you quickly narrow down a memory leak. This information can be obtained in several way:

• Attach a running process using the command jrcmd.
• Generate from a core file or heap dump

Note that we refer to heap histogram, heap summary, or heap diagnostics interchangeably in this article.

Generating Heap Histogram

A heap histogram can be obtained from a running process using the command:

• jrcmd 20488 heap_diagnostics

--------- Detailed Heap Statistics: ---------
30.0% 65027k   672176     -1k [C
10.2% 22119k   943754     -2k java/lang/String
10.0% 21592k   183036     -3k [Ljava/lang/Object;
 4.7% 10185k   434587     +0k java/util/HashMap$Entry
 4.7% 10114k    27029  -1254k [B
 4.5% 9783k   111539     +0k [Ljava/util/HashMap$Entry;
 1.9% 4075k    34777     +0k java/lang/Class
 1.9% 4058k    86590     +0k java/util/HashMap
 1.6% 3448k   147156     +0k javax/management/ObjectName$Property
 1.2% 2593k    82994     +0k java/util/LinkedHashMap$Entry
 1.1% 2398k    76765     +0k java/util/concurrent/ConcurrentHashMap$Segment
 1.0% 2215k     9311     +0k [I
 0.9% 1975k    18469     +0k [J

In the output, there is a "Detailed Heap Statistics" section, which shows the total size and instance count for each class type in the heap:

• The first column corresponds to the Class object type contribution to the Java Heap footprint in %
• The second column correponds to the Class object type memory footprint in K
• The third column correponds to the # of Class instances of a particular type
• The fourth column correponds to the delta - / + memory footprint of a particular type

As you can see from the above snapshot, the biggest data type is [C (i.e., character array) and java.lang.String. In order to see which data types are leaking, you will probably need to generate several snapshots, which you might be able to observe a trend that can lead to further analysis.

Generating Heap Dump

Heap dump is a file containing all the memory contents of a Java application. It can be generated via 

• jrcmd 20488 hprofdump

• Wrote dump to /.../appmgr/APPTOP/instance/debug/jrockit_20488.hprof

Then you can use various tools to load that file and look at various things in the heap: how much each kind of object is using, what things are holding onto the most amount of memory, and so on.  The size of heap dump file is proportional to the size of Java Heap and can be large.

Three of the most common tools are:

• jhat
• This is the original heap analyzer tool, which reads the heap dump and runs a small HTTP server that lets you look at the dump through a series of web page links.
• VisualVM ^[3]
• MAT ^[4,5]

Heap-Related JVM Options

When your JVM runs into OutOfMemoryError, you can set:

• -XX:+HeapDumpOnOutOfMemoryError

to get a heap dump after the heap is big and bloated just before the JVM dies.  Also, you can provide the following flags:

• -XX:HeapDumpPath=<path to the destination>
• -XX:+ExitOnOutOfMemoryError

Similarly, you can get a heap histogram instead of a full heap dump using^[7]:

• -XX:+HeapDiagnosticsOnOutOfMemoryError 
• -XX:HeapDiagnosticsPath=<path to the destination>

References

1. Java™ Memory: Understanding and Analyzing the Java Heap
2. Getting histogram from .hprof heap dump file
3. VisualVM
4. Memory Analyzer (MAT)
5. Diagnosing Java.lang.OutOfMemoryError (XML and More)
6. Troubleshooting Memory Leaks
7. Default Values of JRockit's VM Options (XML and More)
8. Diagnosing Heap Stress in HotSpot (XML and More)
9. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded (XML and More)
10. Memory Mapped File and IO in Java
• Memory used to load Memory mapped file is outside of Java heap Space.  If OOM is caused by memory-mapped files, you may want to reduce your Java heap allocation (i.e., reducing -Xmx).

How to Debug Native OutOfMemory in JRockit

This is the first time that I have seen the following messages:

Caused By: java.lang.OutOfMemoryError: CG #210992 (2) weblogic/management/configuration/DomainMBeanImpl$Helper.getChildren()Ljava/util/Iterator; in generate_code (compilerfrontend.c:537).

Attempting to allocate 6G bytes

There is insufficient native memory for the Java Runtime Environment to continue.

In this article, we will discuss what native memory is and how to debug running out of native memory in JRockit.

Native Memory vs. Heap Memory

There are two types of memory used by JVM and its applications, all of which are allocated from system memory:

• Java Heap
• Java heap is the area of memory used by the JVM to do dynamic memory allocation.
• The amount of memory used for the heap can be controlled by the following command options:
• –Xms2g
• –Xmx2g
• Heap memory can be garbage collected^[4].

• Native Memory
• Internal JVM memory management is, to a large extent, kept off the Java heap and allocated natively in the operating system, through system calls like malloc.   This non-heap system memory allocated by the JVM is referred to as native memory. 
• For JRockit, increasing the amount of available native memory is done implicitly by lowering the maximum Java heap size using –Xmx.

If the heap is too large, it may well be the case that not enough native memory is left for JVM internal usage—bookkeeping, code optimizations, and so on. In that case, the JVM may have no other choice than to throw an OutOfMemoryError from native code (for example, from line 537 of compilerfrontend.c in the previous example).

One example is when several parallel threads perform code optimizations in the JVM. Code optimization typically is one of the JVM operations that consumes the largest amounts of native memory, though only when the optimizing JIT is running and only on a per-method basis.

There are also mechanisms that allow the Java program, and not just the JVM, to allocate native memory, for example through JNI calls. If a JNI call executes a native malloc to reserve a large amount of memory, this memory will be unavailable to the JVM until it is freed.

Code Buffers

JRockit is unique in that it has no bytecode interpreter^[1].  The native code is emitted into a code buffer and executed whenever the function it represents is called.  There are two main problems associated with this compile-only strategy:

• Larger compile-code size
• This problem is mitigated by garbage collecting code buffers with methods no longer in use.
• Long compilation time for large methods
• This problem is solved by having a sloppy mode for the JIT.
• Sometimes JRockit will use a lot of time generating a relatively large method, the typical example being a JSP.
• However, once finished, the response time for accessing that JSP will be better than that of an interpreted version.

The problem of running out of memory for metadata in JRockit is not that different from the one in HotSpot, except for that it is native memory instead of heap memory. There are, however, two differences:

• Cleaning up stale metadata is always enabled by default in JRockit
• UseCodeGC = true (default)
• Allow GC of discarded compiled code
• FreeEmptyCodeBlocks = true (default)
• Free unused code memory
• There is no fixed size limit, be default, for the space used to store metadata

JRCMD^[2]

When JRockit runs out of native memory and throws an OOM exception, JRCMD can be used for debugging.  JRCMD is a small command-line tool that can be used to interact with a running JRockit instance.  And it can be used to track native memory usage.

There is no need to pre-configure the JVM or the application to be able to later attach the tool. Also, the tool add virtually no overhead, making it suitable for use in live production environments.

The tools.jar in the JDK contains an API for attaching to a running JVM—the Java Attach API. This framework is utilized by JRCMD to invoke diagnostic commands.

For debugging OOM, you can invoke jrcmd with print_memusage command with displayMap argument:

$ ./jrcmd 411 print_memusage displayMap
411:
Total mapped                  3641460KB           (reserved=178564KB)
-              Java heap      2097152KB           (reserved=0KB)
-              GC tables        70156KB
-          Thread stacks        45876KB           (#threads=132)
-          Compiled code        65536KB           (used=45010KB)
-               Internal         1672KB
-                     OS       394836KB
-                  Other       544088KB
-            Classblocks        27392KB           (malloced=26718KB #62502)
-        Java class data       393728KB           (malloced=388547KB #294025 in 62502 classes)
- Native memory tracking         1024KB           (malloced=168KB #10)


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    OS                          *java    r x 0x0000000000400000.(     76KB)
    OS                          *java    rw  0x0000000000612000.(      4KB)
    OS                        *[heap]    rw  0x000000001e8a0000.( 284976KB)
THREAD                      Stack 457    rwx 0x000000004007c000 (      8KB)
THREAD                      Stack 457        0x000000004007e000 (     12KB)

In the header section, the first column contains the name of a memory space (i.e., "Java Heap") and the second column shows how much memory is mapped for that space. The third column contains details.

In the map section, the first column shows the category of memory chunks:

• THREAD: Thread related, for example thread stacks.
• INT: Internal use, for example pointer pages.
• HEAP: Chunk used by JRockit for the Java heap.
• OS: Mapped directly from the operating system, such as third party DLLs or shared objects.
• MSP: Memory space. A memory space is a native heap with a specific purpose, for example native memory allocation inside the JVM.
• GC: Garbage collection related, for example live bits.
• CODE: compiled code

When tracking native memory leaks, it is useful to look at how much the memory usage changes over time.  You can do by establishing a baseline first:

$jrcmd 411 print_memusage scale=M baseline

The argument baseline is used to establish a point from which to start measuring.  The scale argument modifies the unit of the amounts of memory in the printout (default is KB).  Once print_memusage is executed with the baseline argument, subsequent calls will include differentials against the baseline.  This can facilitate the monitoring of memory usage changes over time.

References

1. Oracle JRockit - The Definitive Guide by Marcus Hirt and Marcus Lagergren
2. Diagnostic Commands (JRCMD)
3. JNI calls (Wikipedia)
4. Understanding Garbage Collection (XML and More)
5. Where did all of these ConstPoolWrapper objects come from?!

Understanding JVM Thread States

To investigate CPU issues in Java applications, one approach is to diagnose monitor locks and thread activities^[1].

In this article, we will show you:

• How to generate thread dumps on different VMs
• What to know about thread states

JVM Thread States

A thread can be in only one state at a given point in time. These states are virtual machine states which do not reflect any operating system thread states. As an example, here shows different threads in different states on a HotSpot VM:

$ cat thread.tmp | grep "java.lang.Thread.State" | sort | uniq -c
      3    java.lang.Thread.State: BLOCKED (on object monitor)
     18    java.lang.Thread.State: RUNNABLE
      6    java.lang.Thread.State: TIMED_WAITING (on object monitor)
      2    java.lang.Thread.State: TIMED_WAITING (sleeping)
     13    java.lang.Thread.State: WAITING (on object monitor)
      3    java.lang.Thread.State: WAITING (parking)

Below we describe some of the thread states that can be found in a thread dump:

• NEW - this state represents a new thread which is not yet started.
• RUNNABLE - this state represents a thread which is executing in the underlying JVM. Here executing in JVM doesn't mean that the thread is always executing in the OS as well - it may wait for a resource from the Operating system like the processor while being in this state.
• BLOCKED (on object monitor)- this state represents a thread which has been blocked and is waiting for a moniotor to enter/re-enter a synchronized block/method. A thread gets into this state after calling Object.wait method.
• WAITING - this state represnts a thread in the waiting state and this wait is over only when some other thread performs some appropriate action. A thread can get into this state either by calling - Object.wait (without timeout), Thread.join (without timeout), or LockSupport.park methods.
• TIMED_WAITING - this state represents a thread which is required to wait at max for a specified time limit. A thread can get into this state by calling either of these methods: Thread.sleep, Object.wait (with timeout specified), Thread.join (with timeout specified), LockSupport.parkNanos, LockSupport.parkUntil
• TERMINATED - this state reprents a thread which has completed its execution either by returning from the run() method after completing the execution OR by throwing an exception which propagated from the run() method and hence caused the termination of the thread.
• WAITING (parking)- it means a wait state after being parked. A thread can suspend its execution until permit is available (or thread is interrupted, or timeout expired, etc) by calling park(). You can give permit to a thread by calling unpark(). When permit is available, the parked thread consumes it and exits a park() method. Unlike Semaphore's permits, permits of LockSupport are associated with threads (i.e. permit is given to a particular thread) and doesn't accumulate (i.e. there can be only one permit per thread, when thread consumes the permit, it disappears).

Notes:

1. It's hard to investigate any CPU issue just from one thread dump. So, you need to prepare a series of thread dumps for investigation. For an example of analyzing hanging problems, see [9].
2. The above thread state information is for HotSpot.  To understand JRockit's thread dump contents, read [4].

Generating Thread Dumps in HotSpot

$jcmd 15679 Thread.print >thread.tmp

where 15679 is the process ID. For example, it will print the following messages:

15679:
2012-08-24 11:27:27
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.0-b21-internal mixed mode):

"Attach Listener" daemon prio=10 tid=0x000000000528e000 nid=0x4653 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Thread-29" daemon prio=10 tid=0x00002aaab9447000 nid=0x3fe3 runnable [0x000000004408b000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00000000f5ae4738> (a java.io.BufferedInputStream)
        at com.sun.jndi.ldap.Connection.run(Connection.java:849)
        at java.lang.Thread.run(Thread.java:722)

Generating Thread Dumps in JRockit^[4]

$jrcmd 1286 print_threads

where 1286 is the process ID. For example, it will print the following messages:

1286:

===== FULL THREAD DUMP ===============
Thu Aug 23 17:36:30 2012
Oracle JRockit(R) R28.1.3-11-141760-1.6.0_24-20110301-1432-linux-x86_64

"Main Thread" id=1 idx=0x4 tid=1287 prio=5 alive, waiting, native_blocked
    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xe48b4598[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:981)
    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xe48b4598[fat lock]
    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:490)
    at weblogic/Server.main(Server.java:71)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace

"(Signal Handler)" id=2 idx=0x8 tid=1288 prio=5 alive, native_blocked, daemon

"(OC Main Thread)" id=3 idx=0xc tid=1289 prio=5 alive, native_waiting, daemon
...

References

1. Understanding Threads and Locks
2. Thread States Diagram
3. Useful tool: jrcmd
4. Using Thread Dumps
5. Understanding a Java thread dump
6. Fun with JStack 
7. Java 2 Platform, Standard Edition 5.0 "Trouobingshooting and Diagnostic Guide"
8. HotSpot VM Performance Tuning Tips
9. Analyze Hanging Programs Using Java Thread Traces (XML and More)
10. Analyzing Thread Dumps in Middleware - Part 1
11. Analyzing Thread Dumps in Middleware - Part 2