Friday, August 22, 2014

HotSpot: Sizing System Dictionary in JDK 8

The system dictionary is an internal JVM data structure which holds all the classes loaded by the system.  As described in JDK-7114376,[1]
The System Dictionary hashtable bucket array size is fixed at 1009. This value is too small for large programs with many names, too large for small programs and just about right for medium sized ones. The default should remain at 1009, but it should be possible to override it on the command line.
Finally, this has happened in JDK 8 and you can set hashtable bucket array size to be larger if you have more classes loaded in your applications by using:[2]
  • -XX:+UnlockExperimentalVMOptions  -XX:PredictedLoadedClassCount=<#>
This can help calls like Class.forName(), which do lookups into this data structure.

In this article, we will look into sizing system dictionary in more details.

Performance of System Dictionary


In 1.4.2, there were some performance enhancements.  One of them is making system dictionary reads lock-free.[3] From this enhancement, you probably can guess that tuning system dictionary could be important to the JVM's performance.

The current number of buckets in hash table for system dictionary is set to be 1009 by default, which is relatively small for an application which has 49987 classes loaded as shown below:

Loaded   Bytes Unloaded   Bytes       Time
49987 110488.8     2453  7743.8     105.24

From the experience, we know:[5]
In a good hash table, each bucket has zero or one entries, and sometimes two or three, but rarely more than that.
 Assuming the average ideal length of buckets is three, the hashtable bucket array size then should be:
Ideal hashtable bucket array size = 49987 / 3 = 16662

 

System Dictionary Tuning


Seeing 49987classes loaded in our application at end of its run, we decided to set:
-XX:PredictedLoadedClassCount=16661 (a prime)

Note that PredictedLoadedClassCount is an experimental flag.  So, you also need to set:
-XX:+UnlockExperimentalVMOptions
before it in the command line.

When the JVM allocates memory, the largest chunk is the java heap. The system dictionary is in C heap and it is the loaded class cache. The goal of setting PredictedLoadedClassCount flag is to increase the size of the system dictionary in order to make lookups of loaded classes faster. Before every class being loaded, it requires a checking to see if the class is already loaded.  Larger system dictionary will improve JVM performance during this class loading and resolution phase.

Note that the total number of entries in the hashtable does not change— that is based on the number of loaded classes. What sizing of system dictionary do is to increase the spread of the entries so that the average length of the buckets under a single hash result could be reduced. This would reduce the time it takes to find a given entry.

How to Find Number of Loaded Classes?


There are multiple ways to find out number of loaded classes in an application.  For example, you can use -Xverbose:class to determine what classes are loaded.   Another way to find out is what we have done in this article by setting -XX:+PerfDataSaveToFile and then use "jstat -class" command to decipher the class statistics offline:

$<snipped>/jdk-hs/bin/jstat -class file:////slot/myserver/appmgr/APPTOP/instance/domains/slcaf977.us.oracle.com/CRMDomain/hsperfdata_9872

Loaded   Bytes Unloaded   Bytes       Time
49987 110488.8     2453  7743.8     105.24

Acknowledgement


Some writings here are based on the feedback from Mikael Gerdin and Karen Kinnear. However, the author would assume the full responsibility for the content himself.

References

  1. Make system dictionary hashtable bucket array size configurable  (JDK-7114376)
  2. Tuning the JVM (at 43:02 mark)
  3. Java 2 Platform, Standard Edition (J2SE Platform), version 1.4.2 Performance White Paper 
  4. The Class Loader Subsystem 
  5. Hash table (Wikipedia)

Wednesday, August 20, 2014

HotSpot: A Case Study of MetaspaceSize Tuning in JDK 8

This article is one of the Metaspace series in JDK 8 on Xml and More.  Please read previous articles (i.e. [1, 2, 3]) for the background information.

Here we use a case study to show you what we mean by:
Proper monitoring and tuning of the Metaspace is still required in order to limit the frequency or delay the garbage collections of metaspace.

 

JVM Setup


We have used the following print options:
  •  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
to generate GC log files.  Besides that, we have set:
  •  -XX:MetaspaceSize=128m
to specify the initial high water mark to trigger metaspace cleanup by inducing a GC event.

In the GC log file, you can find the following entries:

[Metaspace: 21013K->21013K(1069056K)]

The first value shows you the previously used size before GC, the second value shows you the current used size and the last value shows you the current reserved size.

Before Tuning


Below is the experiment running with all default values.  As you can see, there are six Full GC events triggered by "Metadata GC Threshold" being reached.  The initial high water mark was set by the default MetaspaceSize (i.e.,  21807104 Bytes).  So, the first Full GC event was induced when committed memory of all metaspaces reaches the initial high water mark.  To find HotSpot's default option values, read [4].

3.112: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 21013K->21013K(1069056K)]
5.440: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 34645K->34645K(1081344K)]
11.921: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 58159K->58152K(1101824K)]
18.321: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 97038K->97038K(1136640K)]
51.761: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 160475K->155432K(1193984K)]
319.406: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 267854K->267854K(1288192K)]



Note that the unit on X-axis is seconds and Y-axis KBytes.

After Tuning


Here we have set the experiment with the following extra option:
  • -XX:MetaspaceSize=128m
Note that 128M is bigger than default value (i.e., 21807104 Bytes).  Because of the setting, we have reduced the frequency and delayed garbage collections due to "Metadata GC Threshold" being reached.  


25.863: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 127978K->126460K(1165312K)]
81.254: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 213481K->209918K(1241088K)]
3486.808: [Full GC (Allocation Failure) ...]
   [Eden: ..., [Metaspace: 306929K->298136K(1327104K)]
3631.001: [Full GC (Allocation Failure) ...]
   [Eden: ..., [Metaspace: 299979K->298792K(1329152K)]



Why to Tune?


So, you may ask what's the deal with tuning MetaspaceSize.  Yes, it's just setting the initial high water mark.  Later HotSpot will adjust high water mark based on ergonomics.  Hopefully, when HotSpot's ergonomics becomes more matured, no tuning is necessary at all.  But, even in that case, you may still want to set MetaspaceSize for some occasions.  One of them is when you run a benchmark in a short period and your focus may be on other GC activities than meta data being loaded or unloaded.

References

  1. HotSpot: Understanding Metaspace in JDK 8
  2. HotSpot: Monitoring and Tuning Metaspace in JDK 8
  3. jstat Tool: New Metaspace Statistics from -gc Option
  4. What Are the Default HotSpot JVM Values?