Eclipse MAT: Understand Incoming and Outgoing References

In [1], we have shown how to use OQL to query String instances starting with a specified substring (i.e., our objects of interest) from a heap dump.^[7,8] To determine who is creating these objects, or find out what the purpose of some structures are, an object's incoming and outgoing references become handy.

In this article, we will examine the following topics:

• What are incoming references or outgoing references of an object?

Then look at three topics related to incoming references:

• Garbage Collection Roots (GC Roots)
• Path To GC Roots
• Immediate Dominators

Outgoing references for the 1st String instance

Outgoing References

Using the following OQL statement, we have identified total 7 entries (see Figure above) as our objects of interest:

SELECT * FROM java.lang.String WHERE toString().startsWith("http://xmlns.oracle.com/apps/fnd/applcore/log/service/types")

After expanding the first entry, it shows two outgoing references:

1. a reference to the Class instance for the String object
2. a reference to an array of char values

Outgoing References show the actual contents of the instances, helping to find out their purpose. In our String instance, it holds two references. The memory overhead of this String instance is shown in two values: ^[3]

• Shallow Heap
• Is the memory consumed by that object alone
• Retained Heap
• Is the sum of shallow sizes of all objects in the retained set of that object

These sizes of String instances depends on the internal implementation of the JVM. Read [2,4] for more details.

Incoming References

To get incoming references of the first entry, choose List Objects with Incoming References from the context menu.

Now a tree structure is displayed, showing all instances with all incoming references (note the different icon highlighted in red). These references have kept the object alive and prevented it from being garbage collected.  

Incoming references for String object are from QName object whose incoming references are, in turn, from HashMap.Entry object and class WebServiceOperation object.  See [9] for more details.

Immediate Dominators

Similarly, from the context menu, you can display immediate dominators of the first entry (see Figure below). An Object X is said to dominate an Object Y if every path from the GC Root to Y must pass through X. So, immediate dominators is a very effective way to find out who is keeping a set of objects alive. For example, the immediate dominators of our first String entry in the OQL query (note that we have used "java.*|com\.sun\..*" as our filter) is:

• oracle.j2ee.ws.server.deployment.WebServiceEndpoint

The immediate donimator of String instance is WebServieEndpoint instance

Garbage Collection Roots (GC Roots)

GC roots are objects accessible from outside the heap. GC algorithms build a tree of live objects starting from these GC roots.

The below list shows some of the GC roots: ^[9,10]

• System Class
• Class loaded by bootstrap/system class loader.
• Thread Block
• Objects referred to from currently active thread blocks
• Basically all objects in active thread blocks when a GC is happening are GC roots
• Thread
• Active Threads
• Java Local
• All local variables (parameters, objects or methods in thread stacks)
• JNI Local
• Local variable or parameter of JNI method
• JNI Global
• Global JNI reference
• Monitor Used
• Objects used as a monitor for synchronization

Path To GC Roots

From context menu, you can also show "Path to GC Roots" of the first entry (see Figure below). Path to GC Roots shows the path to GC roots which should be found for a given object. As you can expect, its immediate dominators must also be on this path. Note that, when you display Path to GC Roots, you can specify which fields of certain classes to be ignored when finding paths. For example, we have specified that paths through Weak or Soft Reference referents to be excluded.

GC root of instance of class String is the HashMap.Entry

Live Data Set

Now we know

• oracle.j2ee.ws.server.deployment.WebServiceEndpoint

is keeping our String instance alive. Instead of viewing Path to GC Roots, it is easier to see it the other way around. So, we have chosen to display the outgoing references of WebServiceEndpoint instance (see Figure below). As you can see, our String instance is displayed as the leaf node of the tree structure.

References

1. Eclipse MAT: Querying Heap Objects Using OQL (Xml and More)
2. Java memory usage of simple data structure
3. Shallow vs. Retained Heap
4. Create and Understand Java Heapdumps (Act 4)
5. Diagnosing Java.lang.OutOfMemoryError (Xml and More)
6. I Bet You Have a Memory Leak in Your Application by Nikita Salnikov-Tarnovski
• Classloader leak is the most common leak in web applications
7. How to analyze heap dumps
• Leak can be induced
• Per call (or a class of objects)
• Per object 
8. Diagnosing Heap Stress in HotSpot (Xml and More)
9. Basic Concepts of Java Heap Dump Analysis with MAT (good)
10. What are the GC roots?

Diagnosing Java.lang.OutOfMemoryError

The heap is one of the foremost components that should be monitored to trace performance issues^{[18, 19,22]}. Heap pressure is created when the heap usage approaches the maximum heap size permitted. This leads to frequent full garbage collection events. This steals CPU cycles available for processing and the overall response times degrade. Extreme cases can lead to OutOfMemory conditions, which are not recoverable without a JVM restart.

OutOfMemoryError

When I ran my application, the following exceptions have been thrown in sequence:

• java.lang.OutOfMemoryError: GC overhead limit exceeded^[7,9,15]
• java.lang.OutOfMemoryError: Java heap space

The first message means that, for some reason, the garbage collector is taking an excessive amount of time and recovers very little memory in each run. After I removed the following statement:

• System.gc();

The 1st message was gone. However, the system threw the 2nd message. So, obviously my heap space issue remains. Here are the steps that I took to investigate it:

1. Add the following Java Options
   • -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
     • System will generate a gc.log file
   • -XX:+HeapDumpOnOutOfMemoryError
     • System will generate a heap dump file
     • There is an additional HotSpot VM command line option that allows a user to specify a path where the heap dump will be placed
     • -XX:HeapDumpPath=
2. Analyze the log files:
   • Use regular text editor to examine gc.log file
   • Use Eclipse Memory Analyzer to examine heap dump file (i.e., java_xxx.hprof)

Sometimes your Java process could have run into  "GC overhead limit exceeded" and remain alive.  In this case, you may gather extra information using jstat:

jstat -gcutil
jstat -gccapacity

without restarting the Java process.  Note that all command options discussed in this article applied to Hotspot VM.

JVM Options

The command line options specify:

• -XX:+PrintGCDetails
  • prints more details at garbage collection.
• -XX:+PrintGCTimeStamps
  • prints a time stamp representing the number of seconds since the HotSpot VM was launched until the garbage collection occurred.
• -Xloggc:gc.log
  • causes information about the heap and garbage collection to be printed at each collection.

To set Java Options in JDeveloper, do the following:

1. Right select your project (i.e., ViewController) and bring up the context menu
2. Select Project Properties...
3. Select Run/Debug/Profile
4. Select your Run Configuration (i.e., Default)
5. Click Edit button
6. Specify -Xloggc:gc.log -XX:-PrintGCDetails in the Java Options field

Run your application and reproduce the out-of-memory exception. A log file named gc.log will be generated. I've found mine in the following default location:

• .../system11.1.1.5.37.60.13/DefaultDomain

because my web application was deployed to the Integrated WLS^[4] and run from DefaultDomain. To understand the format of gc.log, read [5,15] for details.

However, gc.log file was not really helpful because it simply pointed out there was a heap issue. But, it didn't say where.

The next step I have taken is running my server with the following flag:

-XX:+HeapDumpOnOutOfMemoryError

it generated a java_pid30835.hprof file when my server encountered a heap error.

Eclipse Memory Analyzer

The heap dump file (i.e., java_pid30835.hprof) is generated by HPROF—a heap and cpu profiling tool. My heap dump file was generated in binary format. Therefore I need to use Eclpse Memory Analyzer to examine it.

You can install Eclipse MAT via the Eclipse Update manager . Select "General Purpose Tools " and install "Memory Analyser (Incubation)" and "Memory Analyser (Charts)".
After installation, double-click your heap dump file and select "Leak Suspects Report".
Eclipse MAT will show a diagram:
and problem suspects:
You can click on the "Details" link to investigate.

Heap Size Adjustment

If you observe many full GCs, try to determine if your old generation is sized too small to hold all the live objects collected from the Survivor and Eden spaces. Alternatively, there may be too many live objects that do not fit into the configured heap size. If it is the latter, increase the overall heap size.

Based on whether the old generation space or the permanent generation space is running out of memory, you may adjust the sizes of heap and meta spaces in this way^[10]:

• For old generation space OutOfMemoryErrors
• Increase -Xms and -Xmx
• For permanent generation OutOfMemoryErrors
• Increase -XX:PermSize and -XX:MaxPermSize

References

1. Eclipse Update Manager
2. Eclipse Memory Analyzer
3. Java Hotspot VM Options
4. Integrated WebLogic Server (WLS)
5. Diagnosing a Garbage Collection problem
6. Frequently Asked Questions about Garbage Collection
7. GC Overhead Limit Exceeded
8. HPROF: A Heap/CPU Profiling Tool in J2SE 5.0
9. Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning
10. Java Performance by Charlie Hunt and Binu John
11. Understanding Garbage Collection
12. Java HotSpot VM Options
13. GCViewer (a free open source tool)
14. Understanding Garbage Collector Output of Hotspot VM
15. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded
16. Memory Analyzer Downloads
• The stand-alone Memory Analyzer is based on Eclipse RCP.
• Can find the update site here too.
17. Shallow vs. Retained Heap (MAT)
18. Diagnosing Heap Stress in HotSpot (XML and More)
19. Diagnosing OutOfMemoryError or Memory Leaks in JRockit (XML and More)
20. MAT Documentation
21. Out of Memory Error while Running the Memory Analyzer
• I need to modify -Xmx512m to -Xmx4g in the file eclipse.ini to analyze a 2GB hprof file.
22. Eclipse MAT: Querying Heap Objects Using OQL (Xml and More)
23. Eclipse MAT: Understand Incoming and Outgoing References  (Xml and More)
24. Memory Mapped File and IO in Java
• Memory used to load Memory mapped file is outside of Java heap Space.  If OOM is caused by memory-mapped files, you may want to reduce your Java heap allocation (i.e., reducing -Xmx).
• java.io.IOException: Map failed
25. jstat - Java Virtual Machine Statistics Monitoring Tool (XML and More)
26. Eclipse MAT — Incoming, Outgoing References (good)

Split Editor Window in Eclipse

This is how to open the same file (for example, RecurrenceImpl.java) in a split editor window in Eclipse:

1. Right click your file name (i.e. the tab title)
2. Select New Editor
• Now you have two files with the same name opened in the same editor window
  
3. Left-click (don't release) on a RecurrenceImpl.java's tab, drag the cursor to the lower edge of the active editor, wait a second so that the cursor changes into a black filled arrow, and release the mouse button.

Learn More

1. Eclipse IDE Tutorial
   

How to Fix SVN Issue "Attempted to lock an already-locked dir"

Using FlexBuilder 3 and attempting to commit a file to SVN Repository, I bumped into the following error:
commit -m "Updated fr_FR version" D:/Adobe/.../fr_FR/search.properties
Attempted to lock an already-locked dir
svn: Working copy 'D:\Adobe\...\fr_FR' locked

Based on SVN help, it says that "svn cleanup" can do the following:
Recursively clean up the working copy, removing locks, resumingunfinished operations, etc.

When cleanup command was issued, I bumped into another issue:
svn: Error replacing text-base of 'search.properties'svn: Can't move 'src\assets\search\locale\fr_FR\.svn\tmp\text-base\search.properties.svn-base' to 'src\assets\search\locale\fr_FR\.svn\text-base\search.properties.svn-base': The system cannot find the path specified.

To fix this issue, I manually added a new folder named "text-base" in ...\fr_FR\.svn subfolder. Finally, I was able to commit my file to the SVN repository.

Learn More

1. Eclipse IDE Tutorial

Installing Eclipse and Its Plug-ins (FB, Jupiter, Subclipse)

This article documents the procedure of setting up Eclipse to use the following plug-ins:

• Flex Builder plug-in 3.0.2 (i.e., FB3_WWEJ_Plugin.exe)
• Jupiter plug-in 3.4.1.1 (i.e., edu.hawaii.ics.csdl.jupiter_3.4.1.1.jar)
  
• Subclipse plug-in
  

To read more about these plug-ins, refer to the following documents:

• Download Adobe Flex BuildIner 3
  
• jupiter-eclipse-plugin
• Install Subclipse in Eclipse 3.x
  

The Eclipse platform we use is "Eclipse IDE for Java EE Developers" and you can download it from here. Our version is 3.4.1.

Follow these steps to install Eclipse and its plug-ins:

1. Install Eclipse by unzipping the archive to a folder, say E:\MyApplications.
2. Click on E:\MyApplications\eclipse\eclipse.exe to start Eclipse.
3. Install Flex Builder plug-in by clicking on FB3_WWEJ_Plugin.exe
• Adobe Flex Builder 3 Installer will be launched.
• Choose an install folder for Flex Builder (e.g., E:\Program Files\Adobe\Flex Builder 3 Plug-in)
  
• Choose an Existing Eclipse Folder (e.g., E:\MyApplications\eclipse)
• If you already have Flash Player installed, you can skip additional installations
• Close all instances of Eclipse or Flex Builder before proceed with the installation
  
• If you install the plug-in on a machine that already has a registered Flex Builder installation, it will be automatically registered. Otherwise, you need to register.
  
4. Install Jupiter plug-in by copying its jar file into the plugins directory of Eclipse (e.g., E:\MyApplications\eclipse\plugins).
5. Install the Subclipse plug-in
• In Eclipse, open Help -> Software Updates
• Software Updates and Add-ons wizard will be launched
• If you try to install the Subclipse plug-in (or any Eclipse plug-in) from the intranet, you'll need to set the Proxy in Eclipse first. You do that in Window->Preferences->General->Network Connectivity.
• Click on Available Software tab
• Click on Add Site button
• Add the following location: http://subclipse.tigris.org/update_1.4.x
• Expand http://subclipse.tigris.org/update_1.4.x site
• Select "Subclipse" and "JavaHL Adapter"and then click on "Install" button

As you might have noticed, there are 3 different ways of installing Eclipse plug-ins:

1. Use InstallAnywhere installer
2. Copy jar file into plugins directory
3. Use Eclipse's Software Update

Final Words

When features and plug-ins are manually installed on top of an Eclipse-based product install located on a FAT file system that has already been run at least once, the product must be explicitly restarted with -clean. That is,

eclipse.exe -clean

Learn More

1. Eclipse IDE Tutorial