HotSpot: Sizing System Dictionary in JDK 8

The system dictionary is an internal JVM data structure which holds all the classes loaded by the system.  As described in JDK-7114376,^[1]

The System Dictionary hashtable bucket array size is fixed at 1009. This value is too small for large programs with many names, too large for small programs and just about right for medium sized ones. The default should remain at 1009, but it should be possible to override it on the command line.

Finally, this has happened in JDK 8 and you can set hashtable bucket array size to be larger if you have more classes loaded in your applications by using:^[2]

• -XX:+UnlockExperimentalVMOptions  -XX:PredictedLoadedClassCount=<#>

This can help calls like Class.forName(), which do lookups into this data structure.

In this article, we will look into sizing system dictionary in more details.

Performance of System Dictionary

In 1.4.2, there were some performance enhancements.  One of them is making system dictionary reads lock-free.^[3] From this enhancement, you probably can guess that tuning system dictionary could be important to the JVM's performance.

The current number of buckets in hash table for system dictionary is set to be 1009 by default, which is relatively small for an application which has 49987 classes loaded as shown below:

Loaded   Bytes Unloaded   Bytes       Time
49987 110488.8     2453  7743.8     105.24

From the experience, we know:^[5]

In a good hash table, each bucket has zero or one entries, and sometimes two or three, but rarely more than that.

 Assuming the average ideal length of buckets is three, the hashtable bucket array size then should be:

Ideal hashtable bucket array size = 49987 / 3 = 16662

 

System Dictionary Tuning

Seeing 49987classes loaded in our application at end of its run, we decided to set:

-XX:PredictedLoadedClassCount=16661 (a prime)

Note that PredictedLoadedClassCount is an experimental flag.  So, you also need to set:

-XX:+UnlockExperimentalVMOptions

before it in the command line.

When the JVM allocates memory, the largest chunk is the java heap. The system dictionary is in C heap and it is the loaded class cache. The goal of setting PredictedLoadedClassCount flag is to increase the size of the system dictionary in order to make lookups of loaded classes faster. Before every class being loaded, it requires a checking to see if the class is already loaded.  Larger system dictionary will improve JVM performance during this class loading and resolution phase.

Note that the total number of entries in the hashtable does not change— that is based on the number of loaded classes. What sizing of system dictionary do is to increase the spread of the entries so that the average length of the buckets under a single hash result could be reduced. This would reduce the time it takes to find a given entry.

How to Find Number of Loaded Classes?

There are multiple ways to find out number of loaded classes in an application.  For example, you can use -Xverbose:class to determine what classes are loaded.   Another way to find out is what we have done in this article by setting -XX:+PerfDataSaveToFile and then use "jstat -class" command to decipher the class statistics offline:

$<snipped>/jdk-hs/bin/jstat -class file:////slot/myserver/appmgr/APPTOP/instance/domains/slcaf977.us.oracle.com/CRMDomain/hsperfdata_9872

Loaded   Bytes Unloaded   Bytes       Time
49987 110488.8     2453  7743.8     105.24

Acknowledgement

Some writings here are based on the feedback from Mikael Gerdin and Karen Kinnear. However, the author would assume the full responsibility for the content himself.

References

1. Make system dictionary hashtable bucket array size configurable  (JDK-7114376)
2. Tuning the JVM (at 43:02 mark)
3. Java 2 Platform, Standard Edition (J2SE Platform), version 1.4.2 Performance White Paper 
4. The Class Loader Subsystem 
5. Hash table (Wikipedia) 
6. VM Class Loading
• Actually, the HotSpot VM maintains three main hash tables to track class loading
• SystemDictionary  
• PlaceholderTable 
• LoaderConstraintTable
7. Garbage-First Garbage Collector Tuning

HotSpot: What Does {pd product} Mean?

In [1], we have discussed how to find out default HotSpot JVM values.  In the output, you may have spotted:

{pd product}

What does it mean? In this article, we will examine that in depth.

Platform Dependent

{pd product} means platform-dependent product option.  First of all, if a flag is a product option, maybe you want to pay attention to it and may want to change its value for better performance.  If it's an experimental or diagnostic flag, you may want to ignore it.

If an option is {pd product}, it means that its default value will be dependent on which platform its implementation would be compiled on.  Currently, these platforms are supported, but not limited to:

• ARM
• PPC
• Solaris
• x86

Examples

Two of the {pd product} flags, we would like to discuss here further.  In JDK 7u51,^[2] they have the following default values on x86 platforms:

    uintx ReservedCodeCacheSize     = 50331648        {pd product}
    bool  TieredCompilation         = false           {pd product}

These default values have been changed in later JDK 8 builds.  For example, in JDK 8u20, they have the following default values on the same server:

    uintx ReservedCodeCacheSize     = 251658240        {pd product}
    bool  TieredCompilation         = true             {pd product}

As discussed in [3,4], you may want to tune these flags for better performance.

References

1. What Are the Default HotSpot JVM Values?
2. Java™ SE Development Kit 7, Update 51 (JDK 7u51)
3. Performance Tuning with Hotspot VM Option: -XX:+TieredCompilation (XML and More)
4. A Case Study of Using Tiered Compilation in HotSpot (XML and More)  

Diagnosing OutOfMemoryError or Memory Leaks in JRockit

When you run into OutOfMemoryError or other memory-leak issues, generating a heap histogram or heap dump can help you diagnose the memory-bloating issues.

In [1], it lists the following Java Heap related problems:

• Exceeding max heap 
• The heap is full and cannot fit a new object
• Large allocation
• The new object is too large for the contiguous free space
• Native exhaustion
• There is not enough native heap for the requested object^[10]
• GC Starvation
• The heap is almost full and causing frequent garbage collection^[9]
• Optimization
• Heap utilization is higher than expected for the current number of users

In this article, we will discuss the following topics:

1. Heap histogram vs. heap dump (see also [8])
2. How to generate heap histogram or heap dump in JRockit
3. JVM options that are useful for heap analysis

Heap Histogram vs. Heap Dump

A heap dump is a snapshot of all the objects in the Java Virtual Machine (JVM) heap at a certain point in time. The JVM software allocates memory for objects from the heap for all class instances and arrays. The garbage collector reclaims the heap memory when an object is no longer needed and there are no references to the object. By examining the heap you can locate where objects are created and find the references to those objects in the source.  However, dumping of Java heap is time-consuming and lengthy in size.

On the other hand, heap histogram gives a very good summary of heap objects used in the application without doing a full heap dump. It can help you quickly narrow down a memory leak. This information can be obtained in several way:

• Attach a running process using the command jrcmd.
• Generate from a core file or heap dump

Note that we refer to heap histogram, heap summary, or heap diagnostics interchangeably in this article.

Generating Heap Histogram

A heap histogram can be obtained from a running process using the command:

• jrcmd 20488 heap_diagnostics

--------- Detailed Heap Statistics: ---------
30.0% 65027k   672176     -1k [C
10.2% 22119k   943754     -2k java/lang/String
10.0% 21592k   183036     -3k [Ljava/lang/Object;
 4.7% 10185k   434587     +0k java/util/HashMap$Entry
 4.7% 10114k    27029  -1254k [B
 4.5% 9783k   111539     +0k [Ljava/util/HashMap$Entry;
 1.9% 4075k    34777     +0k java/lang/Class
 1.9% 4058k    86590     +0k java/util/HashMap
 1.6% 3448k   147156     +0k javax/management/ObjectName$Property
 1.2% 2593k    82994     +0k java/util/LinkedHashMap$Entry
 1.1% 2398k    76765     +0k java/util/concurrent/ConcurrentHashMap$Segment
 1.0% 2215k     9311     +0k [I
 0.9% 1975k    18469     +0k [J

In the output, there is a "Detailed Heap Statistics" section, which shows the total size and instance count for each class type in the heap:

• The first column corresponds to the Class object type contribution to the Java Heap footprint in %
• The second column correponds to the Class object type memory footprint in K
• The third column correponds to the # of Class instances of a particular type
• The fourth column correponds to the delta - / + memory footprint of a particular type

As you can see from the above snapshot, the biggest data type is [C (i.e., character array) and java.lang.String. In order to see which data types are leaking, you will probably need to generate several snapshots, which you might be able to observe a trend that can lead to further analysis.

Generating Heap Dump

Heap dump is a file containing all the memory contents of a Java application. It can be generated via 

• jrcmd 20488 hprofdump

• Wrote dump to /.../appmgr/APPTOP/instance/debug/jrockit_20488.hprof

Then you can use various tools to load that file and look at various things in the heap: how much each kind of object is using, what things are holding onto the most amount of memory, and so on.  The size of heap dump file is proportional to the size of Java Heap and can be large.

Three of the most common tools are:

• jhat
• This is the original heap analyzer tool, which reads the heap dump and runs a small HTTP server that lets you look at the dump through a series of web page links.
• VisualVM ^[3]
• MAT ^[4,5]

Heap-Related JVM Options

When your JVM runs into OutOfMemoryError, you can set:

• -XX:+HeapDumpOnOutOfMemoryError

to get a heap dump after the heap is big and bloated just before the JVM dies.  Also, you can provide the following flags:

• -XX:HeapDumpPath=<path to the destination>
• -XX:+ExitOnOutOfMemoryError

Similarly, you can get a heap histogram instead of a full heap dump using^[7]:

• -XX:+HeapDiagnosticsOnOutOfMemoryError 
• -XX:HeapDiagnosticsPath=<path to the destination>

References

1. Java™ Memory: Understanding and Analyzing the Java Heap
2. Getting histogram from .hprof heap dump file
3. VisualVM
4. Memory Analyzer (MAT)
5. Diagnosing Java.lang.OutOfMemoryError (XML and More)
6. Troubleshooting Memory Leaks
7. Default Values of JRockit's VM Options (XML and More)
8. Diagnosing Heap Stress in HotSpot (XML and More)
9. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded (XML and More)
10. Memory Mapped File and IO in Java
• Memory used to load Memory mapped file is outside of Java heap Space.  If OOM is caused by memory-mapped files, you may want to reduce your Java heap allocation (i.e., reducing -Xmx).

java.lang.NoClassDefFoundError: sun/io/CharacterEncoding

sun.io.CharacterEncoding will be removed in JDK 8.  In our Java test using JDK 8, we have found the following message:

• java.lang.NoClassDefFoundError: sun/io/CharacterEncoding

when WebLogic Server (WLS) started.

In this article, we will discuss how to trouble shoot and resolve this issue.

JVM Option -verbose

When it comes to java.lang.NoClassDefFoundError exception, one thing we need to find out this who has thrown that exception.  To trouble shoot class loading issues, the following JVM option comes in handy:

• -verbose

After we add -verbose option, we have found the following messages from the WLS output:

[Loaded oracle.webservices.annotations.PortableWebService from file:/scratch/perfgrp/mt/rup1/fusionapps/oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar]
[Loaded oracle.j2ee.ws.common.util.TestPageUtils from file:/scratch/perfgrp/mt/rup1/fusionapps/oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar]


[Loaded java.lang.UnsatisfiedLinkError from /scratch/perfgrp/JVMs/jdk1.8.0/jre/lib/rt.jar]
[Loaded java.lang.VerifyError from /scratch/perfgrp/JVMs/jdk1.8.0/jre/lib/rt.jar]
[Loaded com.bea.logging.ThrowableWrapper from file:/scratch/perfgrp/mt/rup1/fusionapps/modules/com.bea.core.logging_1.9.0.0.jar]

java.lang.NoClassDefFoundError: sun/io/CharacterEncoding
        at oracle.j2ee.ws.common.util.TestPageUtils.encode(TestPageUtils.java:216)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.createTestPagePath(WebServiceOperation.java:276)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.initialize(WebServiceOperation.java:253)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.(WebServiceOperation.java:244)
        at oracle.j2ee.ws.server.provider.GenericProviderInterceptorPipeline.registerWebServiceOperationMBean(GenericProviderInterceptorPipeline.java:175)
        Truncated. see log file for complete stack trace

After some investigation, we finally identify that TestPageUtils class from wsclient-rt.jar is the culprit. In JDK 8, the following class:

• sun.io.CharacterEncoding 

will be removed.  It should be rewritten using:

• java.nio.charset.Charset.forName()

For example, here are our changes:

//String encoding = sun.io.CharacterEncoding.aliasName(givenEncoding.toUpperCase());

String encoding = java.nio.charset.Charset.forName(givenEncoding.toUpperCase()).name();

How to Patch wsclient-rt.jar with New Class?

After we got a fix for TestPageUtils class, we loaded the jar with new class.  Note that from the -verbose output, we also know where TestPageUtils was loaded from.  For instance, it was loaded from:

• /.../oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar

Here are the steps that we have taken to apply the patch:

1. cd /.../oracle_common/modules/oracle.webservices_11.1.1
2. mkdir tmpdir
3. cd tmpdir
4. cp ../wsclient-rt.jar .
5. unzip wsclient-rt.jar
6. cp <location of new patch>/TestPageUtils.class oracle/j2ee/ws/common/util
7. jar uf /.../oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar oracle/j2ee/ws/common/util/TestPageUtils.class
8. cd ..
9. rm -rf tmpdir

References

1. The Java Archive Tool (JAR) Examples