Diagnosing Heap Stress in HotSpot

Heap stress is characterized as OutOfMemory conditions or frequent Full GCs accounting for a certain percentage of CPU time^[6].  To diagnose heap stress, either heap dumps or heap histograms can help.

In this article, we will discuss the following topics:

1. Heap histogram vs. heap dump^[1]
2. How to generate heap histogram or heap dump in HotSpot

Heap Histogram vs. Heap Dump 

Without much ado, read this companion article for the comparison.  For heap analysis, you can use either jmap or jcmd to do the job^[5].  Here we focus only on using jmap.

$ jdk-hs/bin/jmap -help
Usage:
    jmap [option] 
        (to connect to running process)
    jmap [option] 
        (to connect to a core file)
    jmap [option] [server_id@]
        (to connect to remote debug server)

where <option> is one of:
    <none>               to print same info as Solaris pmap
    -heap                to print java heap summary
    -histo[:live]        to print histogram of java object heap; if the "live"
                         suboption is specified, only count live objects
    -permstat            to print permanent generation statistics
    -finalizerinfo       to print information on objects awaiting finalization
    -dump:<dump-options> to dump java heap in hprof binary format
                         dump-options:
                           live         dump only live objects; if not specified,
                                        all objects in the heap are dumped.
                           format=b     binary format
                           file=  dump heap to 
                         Example: jmap -dump:live,format=b,file=heap.bin <pid>
    -F                   force. Use with -dump:<dump-options> <pid> or -histo
                         to force a heap dump or histogram when <pid> does not
                         respond. The "live" suboption is not supported
                         in this mode.
    -h | -help           to print this help message
    -J<flag>             to pass <flag> directly to the runtime system

Generating Heap Histogram

Heap histograms can be obtained by using jmap (note that you need to use jmap from the same JDK installation which is used to run your applications):

$~/JVMs/jdk-hs/bin/jmap -histo:live 7891 >hs_jmap_7891.txt

 num     #instances         #bytes  class name
----------------------------------------------
   1:       2099805      195645632  [C
   2:        347553       49534472  <constMethodKlass>
   3:       2055692       49336608  java.lang.String
   4:        347553       44501600  <methodKlass>
   5:         30089       36612792  <constantPoolKlass>
   6:       1044560       33425920  java.util.HashMap$Entry
   7:         90868       24909264  [B
   8:         30089       23289072  <instanceKlassKlass>
   9:         22323       18194144  <constantPoolCacheKlass>
  10:        177458       15661816  [Ljava.util.HashMap$Entry;
  11:        642260       15414240  javax.management.ObjectName$Property
  12:        159785       15405144  [Ljava.lang.Object;

In the output, it shows the total size and instance count for each class type in the heap.  For example, there are 2099805 instances of character arrays (i..e, [C), which has a total size of 195645632 bytes.  Because the suboption live was specified, only live objects were counted (i.e., a full GC was forced before histogram was collected).

Generating Heap Dump

Heap dump is a file containing all the memory contents of a Java application. It can be generated via:

$ ~/JVMs/jdk-hs/bin/jmap -dump:live,file=/tmp/hs_jmap_dump.hprof 7891

Dumping heap to /tmp/hs_jmap_dump.hprof ...

Heap dump file created

Including the live option in jmap will force a full GC to occur before the heap is dumped so that it contains only live objects.  We recommend taking multiple heap dumps.  For example, 30 minutes and 1 hour into the run.  Then use Eclipse MAT^[2,3] to examine the heap dumps.

Finally, there are other ways to generate a java heap dump:

• Use jconsole option to obtain a heap dump via HotSpotDiagnosticMXBean at runtime
• Heap dump will be generated when OutOfMemoryError is thrown by specifying
• -XX:+HeapDumpOnOutOfMemoryError VM option
• Use hprof^[7]

References

1. Diagnosing OutOfMemoryError or Memory Leaks in JRockit (XML and More)
2. Memory Analyzer (MAT)
3. Diagnosing Java.lang.OutOfMemoryError (XML and More)
4. MAT Documentation
5. JRockit JDK and HotSpot JDK: Tool Equivalents (XML and More)
6. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded (XML and More)
7. hprof - Heap and CPU profiling tool
8. What do those strange class names in a java heap dump mean?
9. StackOverflowError when serializing an object in Java
• A good place to check when you run into OOM: object serialization which can grow memory.
10. I Bet You Have a Memory Leak in Your Application by Nikita Salnikov-Tarnovski
• Classloader leak is the most common leak in web applications
11. Diagnose Leaks in Java Language Code (Java SE 8)