HotSpot: A Case Study of MetaspaceSize Tuning in JDK 8

This article is one of the Metaspace series in JDK 8 on Xml and More.  Please read previous articles (i.e. [1, 2, 3]) for the background information.

Here we use a case study to show you what we mean by:

Proper monitoring and tuning of the Metaspace is still required in order to limit the frequency or delay the garbage collections of metaspace.

 

JVM Setup

We have used the following print options:

•  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

to generate GC log files.  Besides that, we have set:

•  -XX:MetaspaceSize=128m

to specify the initial high water mark to trigger metaspace cleanup by inducing a GC event.

In the GC log file, you can find the following entries:

[Metaspace: 21013K->21013K(1069056K)]

The first value shows you the previously used size before GC, the second value shows you the current used size and the last value shows you the current reserved size.

Before Tuning

Below is the experiment running with all default values.  As you can see, there are six Full GC events triggered by "Metadata GC Threshold" being reached.  The initial high water mark was set by the default MetaspaceSize (i.e.,  21807104 Bytes).  So, the first Full GC event was induced when committed memory of all metaspaces reaches the initial high water mark.  To find HotSpot's default option values, read [4].

3.112: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 21013K->21013K(1069056K)]
5.440: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 34645K->34645K(1081344K)]
11.921: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 58159K->58152K(1101824K)]
18.321: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 97038K->97038K(1136640K)]
51.761: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 160475K->155432K(1193984K)]
319.406: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 267854K->267854K(1288192K)]

Note that the unit on X-axis is seconds and Y-axis KBytes.

After Tuning

Here we have set the experiment with the following extra option:

• -XX:MetaspaceSize=128m

Note that 128M is bigger than default value (i.e., 21807104 Bytes).  Because of the setting, we have reduced the frequency and delayed garbage collections due to "Metadata GC Threshold" being reached.  

25.863: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 127978K->126460K(1165312K)]
81.254: [Full GC (Metadata GC Threshold) ...]
   [Eden: ..., [Metaspace: 213481K->209918K(1241088K)]
3486.808: [Full GC (Allocation Failure) ...]
   [Eden: ..., [Metaspace: 306929K->298136K(1327104K)]
3631.001: [Full GC (Allocation Failure) ...]
   [Eden: ..., [Metaspace: 299979K->298792K(1329152K)]

Why to Tune?

So, you may ask what's the deal with tuning MetaspaceSize.  Yes, it's just setting the initial high water mark.  Later HotSpot will adjust high water mark based on ergonomics.  Hopefully, when HotSpot's ergonomics becomes more matured, no tuning is necessary at all.  But, even in that case, you may still want to set MetaspaceSize for some occasions.  One of them is when you run a benchmark in a short period and your focus may be on other GC activities than meta data being loaded or unloaded.

References

1. HotSpot: Understanding Metaspace in JDK 8
2. HotSpot: Monitoring and Tuning Metaspace in JDK 8
3. jstat Tool: New Metaspace Statistics from -gc Option
4. What Are the Default HotSpot JVM Values?
5. VM Class Loading 

Java Throwable: ClassNotFoundException vs. NoClassDefFoundError

Many times we have confused ourselves with the following two Java Throwable messages:

• java.lang.ClassNotFoundException
• java.lang.NoClassDefFoundError

Although both of them are related to Java Classpath,^[7] they are different.^[1] In a nutshell, they differ in this way:

• ClassNotFoundException
• Thrown when an application tries to load a class at run-time and name was provided during runtime not at compile time
• NoClassDefFoundError^[11,12]
• When JVM or a ClassLoader instance is not able to find a particular class at runtime which was available during compile time

ClassNotFoundException

ClassNotFoundException is thrown when an application tries to load in a class through its string name using:

• The forName method in class Class.
• The findSystemClass method in class ClassLoader .
• The loadClass method in class ClassLoader.

but no definition for the class with the specified name could be found. See How-To section below for solutions.

NoClassDefFoundError

The searched-for class definition existed when the currently executing class was compiled, but the definition can no longer be found. One way to debug NoClassDefFoundError is going back to the design-time environment. Using an IDE, you might be able to find where the class is coming from at compile time. Then use that as a clue to find why that class cannot be found at runtime. See How-To section for more details.

What Could Go Wrong?

Classpath^[7] in Java is path to directory or list of directory which is used by ClassLoaders^[3] to find and load class in Java program. If a class cannot be found at runtime, it may be due to:

• Classloaders are not set up correctly^[3]
• Class is corrupted
• Java compiler is not backwards compatible. For example, bytecode generated with JDK 7 won't run in Java 1.6 JVM.^[2]
• Jar file could be renamed in the runtime environment
• Your startup script may have overridden Classpath environment variable
• You might be running your program using jar command and class was not defined in manifest file's ClassPath attribute.

How to Resolve it?

The application that triggered the request to load a class receives a ClassNotFoundException or NoClassDefFoundError if neither the classloader nor any of its ancestors can locate the class.^[3] In that case, you can take the following actions:

• You can use System.getproperty("java.class.path") to get the class path used by your Java application at runtime.^[4]
• Try to run with -classpath option using the classpath you think would work: if it works, then it's a sign that some one is overriding java classpath.
• Check the permission of your jar files.  Your application may not be able to access them.
• Enable class loading traces at JVM level. For example, you can specify -verbose:class for both JRockit and HotSpot.^[5]
• If your application is deployed in WebLogic server, read [3] and enable classloader debugging. 
•  For example, you may want to set:
• -Dweblogic.utils.classloaders.GenericClassLoader.Verbose=true 
• -Dweblogic.utils.classloaders.ChangeAwareClassLoader.Verbose=true
• You can also use Classloader Analysis Tool (http://localhost:port/wls-cat/) which is deployed by default on admin servers of domains in development mode.^[6]
• If your application runs in one environment and not in another,
• Try adding the CLASSPATH explicitly pointing to your jars in setDomainEnv script 
• You can also set "EXT_PRE_CLASSPATH=...." or "EXT_POST_CLASSPATH=..." where "..." are your jar files. The setDomainEnv.sh will pick up these and add to CLASSPATH. The above environment variables can be set when you log on or somewhere at the top of setDomainEnv.sh.

References

1. 3 ways to solve java.lang.NoClassDefFoundError in Java J2EE
2. Is JDK “upward” or “backward” compatible?
3. WebLogic's Classloading Framework (Xml and More)
4. System Properties
5. -verbose:class Option
6. Using the new WebLogic Classloader Analysis Tool (CAT)
• Note that I'm not sure if this is still available in newer WLS releases.
7. How to Set Classpath for Java on Windows Unix and Linux
• Main difference between PATH and CLASSPATH is that former is used to locate Java commands while later is used to locate Java class files.
8. java.lang.UnsatisfiedLinkError: Setting Environment Variable (Xml and More)
9. Using the Classloader Analysis Tool (CAT)
10. WebLogic Server (WLS) Support Pattern: Investigating Different Classloading Issues (Doc ID 1572862.1)
11. If you use JPA 2.1 with WLS 12.1.1 or 12.1.2, then you may see this (because JPA 2.1 only supported starting in 12.1.3):
• java.lang.NoClassDefFoundError: javax/persistence/StoredProcedureQuery
12. java.lang.NoClassDefFoundError: sun/io/CharacterEncoding (Xml and More)

java.lang.UnsupportedClassVersionError

When running an application named StoreFrontService, we have run into this exception:

• java.lang.UnsupportedClassVersionError

In this article, we will examine what it is and how to resolve it.

Class Loading Phases

There are three phases in class loading^[1]:

• Loading
• See below
• Linking
• Bytecode verification^[6]
• Checks the classfile semantics, checks the constant pool symbols, and does type checking
• Preparation
• Creates and initializes static fields to standard defaults and allocates method tables
• Initializing
•  Runs the class static initializers, and initializers for static fields

For a given Java class or Java interface, the load class phase takes its name, finds the binary in Java classfile format, defines the Java class, and creates a java.lang.Class object to represent that given Java class or Java interface.

The load class phase can throw a NoClassDefFound error if a binary representation of a Java class or Java interface cannot be found. In addition, the load class phase does format checking on the syntax of the classfile, which can throw a ClassFormatError or UnsupportedClassVersionError.

The Exception

In our expriement, we have seen the following exception being thrown:

####<Mar 21, 2013 8:16:43 AM PDT> <Error> <Class Loaders> <sfperf-x6250-10> <MS_1> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1363879003661> <BEA-2162500> <Class, oracle.fodemo.storefront.store.service.server.serviceinterface.StoreFrontServiceImpl, was compiled with an incompatible version of Java. Make sure that all the classes needed by the application have been compiled with a compatible java version. java.lang.UnsupportedClassVersionError: oracle/fodemo/storefront/store/service/server/serviceinterface/StoreFrontServiceImpl : unsupported classversion 51.0
java.lang.UnsupportedClassVersionError: oracle/fodemo/storefront/store/service/server/serviceinterface/StoreFrontServiceImpl : unsupported classversion 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14

      

This turns out to be that our application StoreFrontService was compiled in JDK 7 (class version: 51)^[7] and we were running it on JRockit bundled with Java 1.6 VM (class version: 50). The compiler is not backwards compatible^[2] because bytecode generated with JDK 7 won't run in Java 1.6 VM (unless compiled with the -target 1.6 flag^[3]). But the JVM is backwards compatible, as it can run older bytecodes.

References

1. Java Performance by Charlie Hunt, Binu John, David Dagastine 
2. Is JDK “upward” or “backward” compatible?
3. Cross-Compilation Options
4. Java SE 6 vs. JRE 1.6 vs. JDK 1.6 - What do these mean?
5. JRockit Version Information
6. Java Virtual Machine Specification
7. Unsupported major.minor version 51.0

java.lang.NoClassDefFoundError: sun/io/CharacterEncoding

sun.io.CharacterEncoding will be removed in JDK 8.  In our Java test using JDK 8, we have found the following message:

• java.lang.NoClassDefFoundError: sun/io/CharacterEncoding

when WebLogic Server (WLS) started.

In this article, we will discuss how to trouble shoot and resolve this issue.

JVM Option -verbose

When it comes to java.lang.NoClassDefFoundError exception, one thing we need to find out this who has thrown that exception.  To trouble shoot class loading issues, the following JVM option comes in handy:

• -verbose

After we add -verbose option, we have found the following messages from the WLS output:

[Loaded oracle.webservices.annotations.PortableWebService from file:/scratch/perfgrp/mt/rup1/fusionapps/oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar]
[Loaded oracle.j2ee.ws.common.util.TestPageUtils from file:/scratch/perfgrp/mt/rup1/fusionapps/oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar]


[Loaded java.lang.UnsatisfiedLinkError from /scratch/perfgrp/JVMs/jdk1.8.0/jre/lib/rt.jar]
[Loaded java.lang.VerifyError from /scratch/perfgrp/JVMs/jdk1.8.0/jre/lib/rt.jar]
[Loaded com.bea.logging.ThrowableWrapper from file:/scratch/perfgrp/mt/rup1/fusionapps/modules/com.bea.core.logging_1.9.0.0.jar]

java.lang.NoClassDefFoundError: sun/io/CharacterEncoding
        at oracle.j2ee.ws.common.util.TestPageUtils.encode(TestPageUtils.java:216)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.createTestPagePath(WebServiceOperation.java:276)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.initialize(WebServiceOperation.java:253)
        at oracle.j2ee.ws.server.management.mbeans.WebServiceOperation.(WebServiceOperation.java:244)
        at oracle.j2ee.ws.server.provider.GenericProviderInterceptorPipeline.registerWebServiceOperationMBean(GenericProviderInterceptorPipeline.java:175)
        Truncated. see log file for complete stack trace

After some investigation, we finally identify that TestPageUtils class from wsclient-rt.jar is the culprit. In JDK 8, the following class:

• sun.io.CharacterEncoding 

will be removed.  It should be rewritten using:

• java.nio.charset.Charset.forName()

For example, here are our changes:

//String encoding = sun.io.CharacterEncoding.aliasName(givenEncoding.toUpperCase());

String encoding = java.nio.charset.Charset.forName(givenEncoding.toUpperCase()).name();

How to Patch wsclient-rt.jar with New Class?

After we got a fix for TestPageUtils class, we loaded the jar with new class.  Note that from the -verbose output, we also know where TestPageUtils was loaded from.  For instance, it was loaded from:

• /.../oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar

Here are the steps that we have taken to apply the patch:

1. cd /.../oracle_common/modules/oracle.webservices_11.1.1
2. mkdir tmpdir
3. cd tmpdir
4. cp ../wsclient-rt.jar .
5. unzip wsclient-rt.jar
6. cp <location of new patch>/TestPageUtils.class oracle/j2ee/ws/common/util
7. jar uf /.../oracle_common/modules/oracle.webservices_11.1.1/wsclient-rt.jar oracle/j2ee/ws/common/util/TestPageUtils.class
8. cd ..
9. rm -rf tmpdir

References

1. The Java Archive Tool (JAR) Examples