Eclipse MAT: Understand Incoming and Outgoing References

In [1], we have shown how to use OQL to query String instances starting with a specified substring (i.e., our objects of interest) from a heap dump.^[7,8] To determine who is creating these objects, or find out what the purpose of some structures are, an object's incoming and outgoing references become handy.

In this article, we will examine the following topics:

• What are incoming references or outgoing references of an object?

Then look at three topics related to incoming references:

• Garbage Collection Roots (GC Roots)
• Path To GC Roots
• Immediate Dominators

Outgoing references for the 1st String instance

Outgoing References

Using the following OQL statement, we have identified total 7 entries (see Figure above) as our objects of interest:

SELECT * FROM java.lang.String WHERE toString().startsWith("http://xmlns.oracle.com/apps/fnd/applcore/log/service/types")

After expanding the first entry, it shows two outgoing references:

1. a reference to the Class instance for the String object
2. a reference to an array of char values

Outgoing References show the actual contents of the instances, helping to find out their purpose. In our String instance, it holds two references. The memory overhead of this String instance is shown in two values: ^[3]

• Shallow Heap
• Is the memory consumed by that object alone
• Retained Heap
• Is the sum of shallow sizes of all objects in the retained set of that object

These sizes of String instances depends on the internal implementation of the JVM. Read [2,4] for more details.

Incoming References

To get incoming references of the first entry, choose List Objects with Incoming References from the context menu.

Now a tree structure is displayed, showing all instances with all incoming references (note the different icon highlighted in red). These references have kept the object alive and prevented it from being garbage collected.  

Incoming references for String object are from QName object whose incoming references are, in turn, from HashMap.Entry object and class WebServiceOperation object.  See [9] for more details.

Immediate Dominators

Similarly, from the context menu, you can display immediate dominators of the first entry (see Figure below). An Object X is said to dominate an Object Y if every path from the GC Root to Y must pass through X. So, immediate dominators is a very effective way to find out who is keeping a set of objects alive. For example, the immediate dominators of our first String entry in the OQL query (note that we have used "java.*|com\.sun\..*" as our filter) is:

• oracle.j2ee.ws.server.deployment.WebServiceEndpoint

The immediate donimator of String instance is WebServieEndpoint instance

Garbage Collection Roots (GC Roots)

GC roots are objects accessible from outside the heap. GC algorithms build a tree of live objects starting from these GC roots.

The below list shows some of the GC roots: ^[9,10]

• System Class
• Class loaded by bootstrap/system class loader.
• Thread Block
• Objects referred to from currently active thread blocks
• Basically all objects in active thread blocks when a GC is happening are GC roots
• Thread
• Active Threads
• Java Local
• All local variables (parameters, objects or methods in thread stacks)
• JNI Local
• Local variable or parameter of JNI method
• JNI Global
• Global JNI reference
• Monitor Used
• Objects used as a monitor for synchronization

Path To GC Roots

From context menu, you can also show "Path to GC Roots" of the first entry (see Figure below). Path to GC Roots shows the path to GC roots which should be found for a given object. As you can expect, its immediate dominators must also be on this path. Note that, when you display Path to GC Roots, you can specify which fields of certain classes to be ignored when finding paths. For example, we have specified that paths through Weak or Soft Reference referents to be excluded.

GC root of instance of class String is the HashMap.Entry

Live Data Set

Now we know

• oracle.j2ee.ws.server.deployment.WebServiceEndpoint

is keeping our String instance alive. Instead of viewing Path to GC Roots, it is easier to see it the other way around. So, we have chosen to display the outgoing references of WebServiceEndpoint instance (see Figure below). As you can see, our String instance is displayed as the leaf node of the tree structure.

References

1. Eclipse MAT: Querying Heap Objects Using OQL (Xml and More)
2. Java memory usage of simple data structure
3. Shallow vs. Retained Heap
4. Create and Understand Java Heapdumps (Act 4)
5. Diagnosing Java.lang.OutOfMemoryError (Xml and More)
6. I Bet You Have a Memory Leak in Your Application by Nikita Salnikov-Tarnovski
• Classloader leak is the most common leak in web applications
7. How to analyze heap dumps
• Leak can be induced
• Per call (or a class of objects)
• Per object 
8. Diagnosing Heap Stress in HotSpot (Xml and More)
9. Basic Concepts of Java Heap Dump Analysis with MAT (good)
10. What are the GC roots?

Diagnosing OutOfMemoryError or Memory Leaks in JRockit

When you run into OutOfMemoryError or other memory-leak issues, generating a heap histogram or heap dump can help you diagnose the memory-bloating issues.

In [1], it lists the following Java Heap related problems:

• Exceeding max heap 
• The heap is full and cannot fit a new object
• Large allocation
• The new object is too large for the contiguous free space
• Native exhaustion
• There is not enough native heap for the requested object^[10]
• GC Starvation
• The heap is almost full and causing frequent garbage collection^[9]
• Optimization
• Heap utilization is higher than expected for the current number of users

In this article, we will discuss the following topics:

1. Heap histogram vs. heap dump (see also [8])
2. How to generate heap histogram or heap dump in JRockit
3. JVM options that are useful for heap analysis

Heap Histogram vs. Heap Dump

A heap dump is a snapshot of all the objects in the Java Virtual Machine (JVM) heap at a certain point in time. The JVM software allocates memory for objects from the heap for all class instances and arrays. The garbage collector reclaims the heap memory when an object is no longer needed and there are no references to the object. By examining the heap you can locate where objects are created and find the references to those objects in the source.  However, dumping of Java heap is time-consuming and lengthy in size.

On the other hand, heap histogram gives a very good summary of heap objects used in the application without doing a full heap dump. It can help you quickly narrow down a memory leak. This information can be obtained in several way:

• Attach a running process using the command jrcmd.
• Generate from a core file or heap dump

Note that we refer to heap histogram, heap summary, or heap diagnostics interchangeably in this article.

Generating Heap Histogram

A heap histogram can be obtained from a running process using the command:

• jrcmd 20488 heap_diagnostics

--------- Detailed Heap Statistics: ---------
30.0% 65027k   672176     -1k [C
10.2% 22119k   943754     -2k java/lang/String
10.0% 21592k   183036     -3k [Ljava/lang/Object;
 4.7% 10185k   434587     +0k java/util/HashMap$Entry
 4.7% 10114k    27029  -1254k [B
 4.5% 9783k   111539     +0k [Ljava/util/HashMap$Entry;
 1.9% 4075k    34777     +0k java/lang/Class
 1.9% 4058k    86590     +0k java/util/HashMap
 1.6% 3448k   147156     +0k javax/management/ObjectName$Property
 1.2% 2593k    82994     +0k java/util/LinkedHashMap$Entry
 1.1% 2398k    76765     +0k java/util/concurrent/ConcurrentHashMap$Segment
 1.0% 2215k     9311     +0k [I
 0.9% 1975k    18469     +0k [J

In the output, there is a "Detailed Heap Statistics" section, which shows the total size and instance count for each class type in the heap:

• The first column corresponds to the Class object type contribution to the Java Heap footprint in %
• The second column correponds to the Class object type memory footprint in K
• The third column correponds to the # of Class instances of a particular type
• The fourth column correponds to the delta - / + memory footprint of a particular type

As you can see from the above snapshot, the biggest data type is [C (i.e., character array) and java.lang.String. In order to see which data types are leaking, you will probably need to generate several snapshots, which you might be able to observe a trend that can lead to further analysis.

Generating Heap Dump

Heap dump is a file containing all the memory contents of a Java application. It can be generated via 

• jrcmd 20488 hprofdump

• Wrote dump to /.../appmgr/APPTOP/instance/debug/jrockit_20488.hprof

Then you can use various tools to load that file and look at various things in the heap: how much each kind of object is using, what things are holding onto the most amount of memory, and so on.  The size of heap dump file is proportional to the size of Java Heap and can be large.

Three of the most common tools are:

• jhat
• This is the original heap analyzer tool, which reads the heap dump and runs a small HTTP server that lets you look at the dump through a series of web page links.
• VisualVM ^[3]
• MAT ^[4,5]

Heap-Related JVM Options

When your JVM runs into OutOfMemoryError, you can set:

• -XX:+HeapDumpOnOutOfMemoryError

to get a heap dump after the heap is big and bloated just before the JVM dies.  Also, you can provide the following flags:

• -XX:HeapDumpPath=<path to the destination>
• -XX:+ExitOnOutOfMemoryError

Similarly, you can get a heap histogram instead of a full heap dump using^[7]:

• -XX:+HeapDiagnosticsOnOutOfMemoryError 
• -XX:HeapDiagnosticsPath=<path to the destination>

References

1. Java™ Memory: Understanding and Analyzing the Java Heap
2. Getting histogram from .hprof heap dump file
3. VisualVM
4. Memory Analyzer (MAT)
5. Diagnosing Java.lang.OutOfMemoryError (XML and More)
6. Troubleshooting Memory Leaks
7. Default Values of JRockit's VM Options (XML and More)
8. Diagnosing Heap Stress in HotSpot (XML and More)
9. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded (XML and More)
10. Memory Mapped File and IO in Java
• Memory used to load Memory mapped file is outside of Java heap Space.  If OOM is caused by memory-mapped files, you may want to reduce your Java heap allocation (i.e., reducing -Xmx).