How to Debug Native OutOfMemory in JRockit

This is the first time that I have seen the following messages:

Caused By: java.lang.OutOfMemoryError: CG #210992 (2) weblogic/management/configuration/DomainMBeanImpl$Helper.getChildren()Ljava/util/Iterator; in generate_code (compilerfrontend.c:537).

Attempting to allocate 6G bytes

There is insufficient native memory for the Java Runtime Environment to continue.

In this article, we will discuss what native memory is and how to debug running out of native memory in JRockit.

Native Memory vs. Heap Memory

There are two types of memory used by JVM and its applications, all of which are allocated from system memory:

• Java Heap
• Java heap is the area of memory used by the JVM to do dynamic memory allocation.
• The amount of memory used for the heap can be controlled by the following command options:
• –Xms2g
• –Xmx2g
• Heap memory can be garbage collected^[4].

• Native Memory
• Internal JVM memory management is, to a large extent, kept off the Java heap and allocated natively in the operating system, through system calls like malloc.   This non-heap system memory allocated by the JVM is referred to as native memory. 
• For JRockit, increasing the amount of available native memory is done implicitly by lowering the maximum Java heap size using –Xmx.

If the heap is too large, it may well be the case that not enough native memory is left for JVM internal usage—bookkeeping, code optimizations, and so on. In that case, the JVM may have no other choice than to throw an OutOfMemoryError from native code (for example, from line 537 of compilerfrontend.c in the previous example).

One example is when several parallel threads perform code optimizations in the JVM. Code optimization typically is one of the JVM operations that consumes the largest amounts of native memory, though only when the optimizing JIT is running and only on a per-method basis.

There are also mechanisms that allow the Java program, and not just the JVM, to allocate native memory, for example through JNI calls. If a JNI call executes a native malloc to reserve a large amount of memory, this memory will be unavailable to the JVM until it is freed.

Code Buffers

JRockit is unique in that it has no bytecode interpreter^[1].  The native code is emitted into a code buffer and executed whenever the function it represents is called.  There are two main problems associated with this compile-only strategy:

• Larger compile-code size
• This problem is mitigated by garbage collecting code buffers with methods no longer in use.
• Long compilation time for large methods
• This problem is solved by having a sloppy mode for the JIT.
• Sometimes JRockit will use a lot of time generating a relatively large method, the typical example being a JSP.
• However, once finished, the response time for accessing that JSP will be better than that of an interpreted version.

The problem of running out of memory for metadata in JRockit is not that different from the one in HotSpot, except for that it is native memory instead of heap memory. There are, however, two differences:

• Cleaning up stale metadata is always enabled by default in JRockit
• UseCodeGC = true (default)
• Allow GC of discarded compiled code
• FreeEmptyCodeBlocks = true (default)
• Free unused code memory
• There is no fixed size limit, be default, for the space used to store metadata

JRCMD^[2]

When JRockit runs out of native memory and throws an OOM exception, JRCMD can be used for debugging.  JRCMD is a small command-line tool that can be used to interact with a running JRockit instance.  And it can be used to track native memory usage.

There is no need to pre-configure the JVM or the application to be able to later attach the tool. Also, the tool add virtually no overhead, making it suitable for use in live production environments.

The tools.jar in the JDK contains an API for attaching to a running JVM—the Java Attach API. This framework is utilized by JRCMD to invoke diagnostic commands.

For debugging OOM, you can invoke jrcmd with print_memusage command with displayMap argument:

$ ./jrcmd 411 print_memusage displayMap
411:
Total mapped                  3641460KB           (reserved=178564KB)
-              Java heap      2097152KB           (reserved=0KB)
-              GC tables        70156KB
-          Thread stacks        45876KB           (#threads=132)
-          Compiled code        65536KB           (used=45010KB)
-               Internal         1672KB
-                     OS       394836KB
-                  Other       544088KB
-            Classblocks        27392KB           (malloced=26718KB #62502)
-        Java class data       393728KB           (malloced=388547KB #294025 in 62502 classes)
- Native memory tracking         1024KB           (malloced=168KB #10)


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    OS                          *java    r x 0x0000000000400000.(     76KB)
    OS                          *java    rw  0x0000000000612000.(      4KB)
    OS                        *[heap]    rw  0x000000001e8a0000.( 284976KB)
THREAD                      Stack 457    rwx 0x000000004007c000 (      8KB)
THREAD                      Stack 457        0x000000004007e000 (     12KB)

In the header section, the first column contains the name of a memory space (i.e., "Java Heap") and the second column shows how much memory is mapped for that space. The third column contains details.

In the map section, the first column shows the category of memory chunks:

• THREAD: Thread related, for example thread stacks.
• INT: Internal use, for example pointer pages.
• HEAP: Chunk used by JRockit for the Java heap.
• OS: Mapped directly from the operating system, such as third party DLLs or shared objects.
• MSP: Memory space. A memory space is a native heap with a specific purpose, for example native memory allocation inside the JVM.
• GC: Garbage collection related, for example live bits.
• CODE: compiled code

When tracking native memory leaks, it is useful to look at how much the memory usage changes over time.  You can do by establishing a baseline first:

$jrcmd 411 print_memusage scale=M baseline

The argument baseline is used to establish a point from which to start measuring.  The scale argument modifies the unit of the amounts of memory in the printout (default is KB).  Once print_memusage is executed with the baseline argument, subsequent calls will include differentials against the baseline.  This can facilitate the monitoring of memory usage changes over time.

References

1. Oracle JRockit - The Definitive Guide by Marcus Hirt and Marcus Lagergren
2. Diagnostic Commands (JRCMD)
3. JNI calls (Wikipedia)
4. Understanding Garbage Collection (XML and More)
5. Where did all of these ConstPoolWrapper objects come from?!