HotSpot: Understanding Metaspace in JDK 8

In JDK 8, it will print out the following information when JVM exits:

Heap

 <snipped> 

 Metaspace       used 2425K, capacity 4498K, committed 4864K, reserved 1056768K
  class space    used 262K, capacity 386K, committed 512K, reserved 1048576K

if you have enabled GC printing using:

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Note that if PrintGCDetails is enabled, PrintGC^[1] is also enabled.

In this article, we will examine what the above lines mean.

Metaspace

Internal JVM memory management is, to a large extent, kept off the Java heap and allocated natively in the operating system, through system calls like malloc or mmap. This non-heap system memory allocated by the JVM is referred to as native memory. Similar to the Oracle JRockit and IBM JVM's, the JDK 8 HotSpot JVM is now using native memory for the representation of class metadata, which is called Metaspace.^[2,4]

In JDK 8, Hotspot explicitly manages the space used for metadata. Space is requested from the OS and then divided into chunks. A class loader will allocate space for metadata from its chunks (a chunk is to bound to a specific class loader). When classes are unloaded for a class loader, its chunks are recycled for reuse or returned to the OS.   Note that metadata uses mmap'ed space and not malloc'ed space.

Used/Capacity/Committed/Reserved

When JVM exits, it prints the following information if you have enabled GC printing:

 Metaspace       used 2425K, capacity 4498K, committed 4864K, reserved 1056768K
  class space    used 262K, capacity 386K, committed 512K, reserved 1048576K

In the “Metaspace” line, the “used” is the amount of space used for loaded classes and other metadata. The “capacity” is the space available for metadata in currently allocated chunks. The “committed” value is the amount of space available for chunks. The “reserved” is the amount of space reserved (but not necessarily committed) for metadata. On the “class space” line, those values are the corresponding values for the class area in Metaspace when compressed class pointers are used.

Mataspace is dynamically managed by HotSpot.  Metadata are deallocated when their class loader are garbage collected (GC'ed).  A high water mark is used for inducing a GC—when committed memory of all metaspaces reaches this level, a GC is triggered.  You can use the following flag:

-XX:MetaspaceSize=<size>

to specify the initial high water level. 

The interaction between metaspace growth and GC can be summarized as follows:

Metaspaces expand the native memory it is using until it gets to some level (starts at MetaspaceSize). When it hits that level, it does a GC to see if classes can be unloaded. After the GC, it can use freed space in it for metadata. If not enough space has been freed, it uses more native memory.

After the GC it also decides what the next level is for doing a GC to unload classes.  The level mostly increases to have fewer GC's. It sometimes will decrease if lots of space has been freed due to class unloading.  If MetaspaceSize is set higher, then fewer GC will be done early.

Acknowledgement

Some writings here are based on the feedback from Jon Masamitsu. However, the author would assume the full responsibility for the content himself.

References

1. Java HotSpot VM Options
2. HotSpot: Monitoring and Tuning Metaspace in JDK 8
3. Latest JDK 8 and JDK 7 available here
4. JavaOne Wednesday: Permanent Generation Removal Overview   
5. jstat Tool: New Metaspace Statistics from -gc Option 
6. Metatspace in JDK 8 (good) 
7. VM Class Loading
8. Learn More About Performance Improvements in JDK 8 

HotSpot: Monitoring and Tuning Metaspace in JDK 8

If you upgrade from JDK 7 to JDK 8, there are some changes in JVM that you should pay attention to.  One of them is:

• The removal of Permanent Generation (PermGen) space.^[1]  

Similar to the Oracle JRockit and IBM JVM's, the JDK 8 HotSpot JVM is now using native memory for the representation of class metadata, which is called Metaspace.  This may have removed the old OOM Error.^[2] But ...

As described in [3], proper monitoring and tuning of the Metaspace is still required in order to limit the frequency or delay of metaspace garbage collections.  In this article, we will show how to achieve that.

Tuning Metaspace

When space (either PermGen or Metaspace) is filled up, class need to be unloaded.  In the PermGen era, when PermGen is filled up, a Garbage Collection (or GC) would occur and unload classes.  Without PermGen, we still need to have some way to know when to do a GC to unload classes.  This is when

-XX:MetaspaceSize=<size>

comes into play.   When the amount of space used for classes reaching MetaspaceSize, a GC is done to see if there are classes to be unloaded.  If you know that you are going to have lots of classes loaded and use more than 20M (this is the default value for my platform) class data, increasing MetaspaceSize can delay the GC's that are done to check for class unloading.  During your upgrade from JDK 7 to 8, as a first estimate, you can use the old value of PermSize to be your new MetaspaceSize.  You can also read [4] for more details.

Since Metaspace uses native memory, class metadata allocation theoretically is limited by amount of available native memory (capacity will of course depend if you use a 32-bit JVM vs. 64-bit along with OS virtual memory availability).  However, if you want to limit the amount of native memory used for class metadata, you can set:

-XX:MaxMetaspaceSize=<size>

which can restrict the extension of metaspace to the maximum at runtime.

How Many Classes Are Loaded?

There are multiple way^[6,7] to find out how many classes are loaded and how much space they would take.  Here we introduce one way of finding that information.

jps is a Java tool that  you can use to  list the instrumented HotSpot Java Virtual Machines (JVMs) on the target system.  For example, here are the list of JVMs that are running on our Linux.

$ jps -l

32520 weblogic.Server
8936 weblogic.Server
11598 weblogic.Server
32646 weblogic.Server
7591 weblogic.Server
27429 weblogic.Server

To gather class information, you can use jstat tool which displays performance statistics for an instrumented HotSpot Java virtual machine (JVM).  For example, jstat has attached to process 27429 and displayed the statistics on the behavior of the class loader as below:

$ jstat -class 27429 1000 6

Loaded  Bytes  Unloaded  Bytes     Time
 46167 98340.6      406  1494.6      54.22
 46167 98340.6      406  1494.6      54.22
 46167 98340.6      406  1494.6      54.22
 46167 98340.6      406  1494.6      54.22
 46167 98340.6      406  1494.6      54.22
 46167 98340.6      406  1494.6      54.22

Class Loader Statistics

Column	Description
Loaded	Number of classes loaded.
Bytes	Number of Kbytes loaded.
Unloaded	Number of classes unloaded.
Bytes	Number of Kbytes unloaded.
Time	Time spent performing class load and unload operations.

  

From the above output, we know that 46167 classes were loaded and they have taken 96M.  At the time of snapshots, HotSpot has also unloaded 406 classes from which 1.46 M were freed.  Based on this information, you can then decide how to set your MetaspaceSize and MaxMetaspaceSize (not advised) as described above.  However, be warned that once you have set MaxMetaspaceSize, you may still run into OOM error if metaspace cannot be extended further.

Acknowledgement

Some writings here are based on the feedback from Jon Masamitsu. However, the author would assume the full responsibility for the content himself.

References

1. Java 7 features - PermGen removal
2. 64-bit java.lang.OutOfMemoryError: PermGen space
3. Java 8: From PermGen to Metaspace
4. HotSpot: Understanding Metaspace in JDK 8 
5. Latest JDK 8 and JDK 7 available here
6. JavaOne Wednesday: Permanent Generation Removal Overview  
7. jstat Tool: New Metaspace Statistics from -gc Option 
8. VM Class Loading
9. Other JDK 8 articles on Xml and More
10. Tuning that was great in old JRockit versions might not be so good anymore
    
    • Trying to bring over each and every tuning option from a JR configuration to an HS one is probably a bad idea.
    • Even when moving between major versions of the same JVM, we usually recommend going back to the default (just pick a collector and heap size) and then redoing any tuning work from scratch (if even necessary).

How to Debug Native OutOfMemory in JRockit

This is the first time that I have seen the following messages:

Caused By: java.lang.OutOfMemoryError: CG #210992 (2) weblogic/management/configuration/DomainMBeanImpl$Helper.getChildren()Ljava/util/Iterator; in generate_code (compilerfrontend.c:537).

Attempting to allocate 6G bytes

There is insufficient native memory for the Java Runtime Environment to continue.

In this article, we will discuss what native memory is and how to debug running out of native memory in JRockit.

Native Memory vs. Heap Memory

There are two types of memory used by JVM and its applications, all of which are allocated from system memory:

• Java Heap
• Java heap is the area of memory used by the JVM to do dynamic memory allocation.
• The amount of memory used for the heap can be controlled by the following command options:
• –Xms2g
• –Xmx2g
• Heap memory can be garbage collected^[4].

• Native Memory
• Internal JVM memory management is, to a large extent, kept off the Java heap and allocated natively in the operating system, through system calls like malloc.   This non-heap system memory allocated by the JVM is referred to as native memory. 
• For JRockit, increasing the amount of available native memory is done implicitly by lowering the maximum Java heap size using –Xmx.

If the heap is too large, it may well be the case that not enough native memory is left for JVM internal usage—bookkeeping, code optimizations, and so on. In that case, the JVM may have no other choice than to throw an OutOfMemoryError from native code (for example, from line 537 of compilerfrontend.c in the previous example).

One example is when several parallel threads perform code optimizations in the JVM. Code optimization typically is one of the JVM operations that consumes the largest amounts of native memory, though only when the optimizing JIT is running and only on a per-method basis.

There are also mechanisms that allow the Java program, and not just the JVM, to allocate native memory, for example through JNI calls. If a JNI call executes a native malloc to reserve a large amount of memory, this memory will be unavailable to the JVM until it is freed.

Code Buffers

JRockit is unique in that it has no bytecode interpreter^[1].  The native code is emitted into a code buffer and executed whenever the function it represents is called.  There are two main problems associated with this compile-only strategy:

• Larger compile-code size
• This problem is mitigated by garbage collecting code buffers with methods no longer in use.
• Long compilation time for large methods
• This problem is solved by having a sloppy mode for the JIT.
• Sometimes JRockit will use a lot of time generating a relatively large method, the typical example being a JSP.
• However, once finished, the response time for accessing that JSP will be better than that of an interpreted version.

The problem of running out of memory for metadata in JRockit is not that different from the one in HotSpot, except for that it is native memory instead of heap memory. There are, however, two differences:

• Cleaning up stale metadata is always enabled by default in JRockit
• UseCodeGC = true (default)
• Allow GC of discarded compiled code
• FreeEmptyCodeBlocks = true (default)
• Free unused code memory
• There is no fixed size limit, be default, for the space used to store metadata

JRCMD^[2]

When JRockit runs out of native memory and throws an OOM exception, JRCMD can be used for debugging.  JRCMD is a small command-line tool that can be used to interact with a running JRockit instance.  And it can be used to track native memory usage.

There is no need to pre-configure the JVM or the application to be able to later attach the tool. Also, the tool add virtually no overhead, making it suitable for use in live production environments.

The tools.jar in the JDK contains an API for attaching to a running JVM—the Java Attach API. This framework is utilized by JRCMD to invoke diagnostic commands.

For debugging OOM, you can invoke jrcmd with print_memusage command with displayMap argument:

$ ./jrcmd 411 print_memusage displayMap
411:
Total mapped                  3641460KB           (reserved=178564KB)
-              Java heap      2097152KB           (reserved=0KB)
-              GC tables        70156KB
-          Thread stacks        45876KB           (#threads=132)
-          Compiled code        65536KB           (used=45010KB)
-               Internal         1672KB
-                     OS       394836KB
-                  Other       544088KB
-            Classblocks        27392KB           (malloced=26718KB #62502)
-        Java class data       393728KB           (malloced=388547KB #294025 in 62502 classes)
- Native memory tracking         1024KB           (malloced=168KB #10)


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    OS                          *java    r x 0x0000000000400000.(     76KB)
    OS                          *java    rw  0x0000000000612000.(      4KB)
    OS                        *[heap]    rw  0x000000001e8a0000.( 284976KB)
THREAD                      Stack 457    rwx 0x000000004007c000 (      8KB)
THREAD                      Stack 457        0x000000004007e000 (     12KB)

In the header section, the first column contains the name of a memory space (i.e., "Java Heap") and the second column shows how much memory is mapped for that space. The third column contains details.

In the map section, the first column shows the category of memory chunks:

• THREAD: Thread related, for example thread stacks.
• INT: Internal use, for example pointer pages.
• HEAP: Chunk used by JRockit for the Java heap.
• OS: Mapped directly from the operating system, such as third party DLLs or shared objects.
• MSP: Memory space. A memory space is a native heap with a specific purpose, for example native memory allocation inside the JVM.
• GC: Garbage collection related, for example live bits.
• CODE: compiled code

When tracking native memory leaks, it is useful to look at how much the memory usage changes over time.  You can do by establishing a baseline first:

$jrcmd 411 print_memusage scale=M baseline

The argument baseline is used to establish a point from which to start measuring.  The scale argument modifies the unit of the amounts of memory in the printout (default is KB).  Once print_memusage is executed with the baseline argument, subsequent calls will include differentials against the baseline.  This can facilitate the monitoring of memory usage changes over time.

References

1. Oracle JRockit - The Definitive Guide by Marcus Hirt and Marcus Lagergren
2. Diagnostic Commands (JRCMD)
3. JNI calls (Wikipedia)
4. Understanding Garbage Collection (XML and More)
5. Where did all of these ConstPoolWrapper objects come from?!