SPECjbb 2013: Understanding the Basics

SPEC Benchmark Suites aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation (SPECjbb) to a full system with Java EE, database, disk, and network (SPECjEnterprise).

In this article, we will introduce the basics of SPECjbb 2013 (an older version).  But, descriptions can be equally applied to SPECjbb 2015 with some changes.

Components

SPECjbb 2013 includes three main components (see also: Driver System and Reporter^[4]):

• Controller (or Ctr)
• Function: 
• Directs the execution of the workload
• Configution:
• Only one Ctr per System Under Test (SUT)
• Could be located outside of the SUT
• Heap size of 2GB is suggested 
• Run category^[3] depends on type of Ctr invoked, which set by -m option:
• -m [composite / multicontroller / distcontroller]
• SPECjbb2013-Composite 
• SPECjbb2013-MultiJVM 
• SPECjbb2013-Distributed
• Debugging:
• If OOM is experienced, just re-run the reporter with larger heap using the run_reporter.* scripts. 
• Capture the Ctr standard output can be helpful in debugging many issues.
• Transaction Injector(s) (or TxI)
• Function: 
• Issues requests and services to Backend(s) as directed by the Controller and measures end-to-end response time from issued requests
• Configuration:
• One TxI cannot go across two BEs 
• Heap size of 2GB is suggested.
• Backend(s) (or BE)
• Function:
• Contains business logic code that processes requests and services from TxI, and notifies the TxI that a request has been processed.
• Configuration:
• One or more TxI per BE
• Heap Size: a minimum heap setting of 2GB is needed. 
• Larger heaps may be needed for generational garbage collectors.

Configuration & Deployments

There are many options on configuring and deploying SPECjbb components:

• Single vs. Multiple JVM's
• Benchmark components Ctr, TxI(s) and BE(s) can be configured to run
• Inside a single JVM instance 
• Across multiple JVM instances
• With each component using its own JVM instance deployed to run across single or multiple hosts.
• Group
• Is a logical mapping of BE to TxI(s) 
• Configuration:
• Can have only one BE, but can have one or more TxI(s) if one TxI is not able to fully load the BE. 
• The topology for a Group can be defined using command line options in the run script. 
• The benchmark can be run in single or multiple Group(s) configurations.
• If there are multiple BEs, some % of requests and txns require interaction with other BEs initiating Inter-Java process communication among BEs.
• Configuration of Properties
• SPECjbb2013-Composite and SPECjbb2013-MultiJVM are run from same directory using same property file. 
• SPECjbb2013-Distributed runs across OS images and will require these properties to be set for every launching benchmark component either from property file or command line.
• Driver System(s)
• In addition to SUT, SPECjbb2013-Distributed requires a driver system(s). 
• Ctr and TxI(s) must run on a driver machine(s) using a non-virtualized OS. 
• The SUT only runs BE(s), which can be virtualized environments in this category. 
• The SUT can consist of one or more BEs running on one or more Nodes.

References

1. Rules and Reporting Rules (SPECjbb 2013)
2. User Guide (SPECjbb 2013)
3. Run categories of SPECjbb 2013:
• Based on most common trends in Java deployments, SPECjbb2013 components Controller, TxI(s) and Backend(s), configurations have been mapped into three different run-categories: SPECjbb2013-Composite, SPECjbb2013-Multi-JVM, SPECjbb2013-Distributed.
4. Reporter
• At the end of benchmark run, Controller invokes the reporter unless command line option “-skipReport” was used.
• If needed, the reporter can be manually invoked after the benchmark run.
• Reporter will pick correct template file based on selected run category and report the final run results.

JDK 8: Thread Stack Size Tuning

When you upgrade JDK, you should re-examine all JVM options you have set in your Java applications.  For example, let's look at thread stack size tuning specifically.  As suggested in [1], it states:

• In most applications, 128k happens to be enough for the Java thread stack.

However, after setting that, we have run into the following fatal exception:

The stack size specified is too small, Specify at least 228k
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

In this article, we will discuss thread stack size tuning in JDK 8 (i.e., HotSpot VM).

Default Thread Stack Size

When a new thread is launched, the Java virtual machine creates a new Java stack for the thread. As mentioned earlier, a Java stack stores a thread's state in discrete frames.^[3] The Java virtual machine only performs two operations directly on Java Stacks: it pushes and pops frames.

The default thread stack size varies with JVM, OS and environment variables. To find out what your default ThreadStackSize is on your platform, use:^[1]

java -XX:+PrintFlagsFinal -version

A typical value is 512k. It is generally larger for 64bit JVMs because references are 8 bytes rather than 4 bytes in size (but, you can compress oops or class pointers if you choose).^[2] For example,^[4]

In Java SE 6, the default on Sparc is 512k in the 32-bit VM, and 1024k in the 64-bit VM. On x86 Solaris/Linux it is 320k in the 32-bit VM and 1024k in the 64-bit VM.

On Windows, the default thread stack size is read from the binary (java.exe). As of Java SE 6, this value is 320k in the 32-bit VM and 1024k in the 64-bit VM.

In JDK 8, every time the JVM creates a thread, the OS allocates some native memory to hold that thread’s stack, committing more memory to the process until the thread exits. Thread stacks are fully allocated (i.e., committed, not just reserved) when they are created.

This means that if your application spawns a lot of threads, this can consume a significant amount of memory which could otherwise be used by your application or OS (or it can eventually leads to OutOfMemoryError).

You can reduce your stack size by running with the -Xss option. For example:

java -server -Xss256k

or

java -server -XX:ThreadStackSize=256 

Note that if you have installed a 64-bit VM binary for Linux, you can omit -server option.^[5]

Virtual Memory Map

In JDK 8, HotSpot installation comes with a feature named Native Memory Tracking (default: disabled).  To enable it, use:

-XX:NativeMemoryTracking=[off|detail|summary]

After enabling NMT, you can examine the memory footprint taken by either Thread or Thread Stack using:

jcmd <pid> VM.native_memory [summary | detail | baseline | summary.diff | detail.diff | shutdown] [scale= KB | MB | GB]

 For example, on a 64-bit Linux platform, here is the thread stack size before and after setting -Xss256k:

Before

 Virtual memory map:

[0x0000000040049000 - 0x000000004014a000] reserved and committed 1028KB for Thread Stack from
    [0x00002aec741ca5e4] JavaThread::run()+0x24
    [0x00002aec74083268] java_start(Thread*)+0x108

After

Virtual memory map:

[0x0000000040078000 - 0x00000000400b9000] reserved and committed 260KB for Thread Stack from
    [0x00002b02c69156e4] JavaThread::run()+0x24
    [0x00002b02c67ce338] java_start(Thread*)+0x108

Conclusions

The thread stack is used to push stacks frames in nested method calls. If the nesting is so deep that the thread runs out of space, the thread dies with a StackOverflowError.^[8] If your applications use lots of recursive algorithms or if your applications are built on top of a framework utilizing MVC design pattern such as Oracle ADF, you may want to leave StackThreadSize as defaults.

However, thread stacks are quite large, particularly for a 64-bit JVM.  In [9], Scott Oaks has advised:

• As a general rule, many applications can actually run with a 128 KB stack size in a 32-bit JVM, and a 256 KB stack size in a 64-bit JVM.
• In a 64-bit JVM, there is usually no reason to set this value unless the machine is quite strained for physical memory and the smaller stack size will prevent applications from running out of native memory. 
• On the other hand, using a smaller (e.g., 128 KB) stack size on a 32-bit JVM is often a good idea, as it frees up memory in the process size and allows the JVM to utilize a larger heap.

Finally, the total footprint of the JVM has a significant effect on its performance. So, footprint is one aspect of Java performance that should be commonly monitored.

References

1. What Are the Default HotSpot JVM Values? 
2. JDK 8: UseCompressedClassPointers vs. UseCompressedOops
3. The Java Stack 
4. Frequently Asked Questions About the Java HotSpot VM 
5. HotSpot VM Binaries: 32-Bit vs. 64-Bit 
6. How to Debug Native OutOfMemory in JRockit 
7. HotSpot: Monitoring and Tuning Metaspace in JDK 8
8. Using -Xss to adjust Java default thread stack size to save memory and prevent StackOverflowError
9. Java Performance: The Definitive Guide (Strongly recommended)
10. Java thread stack size on 64-bit linux 
11. Learn More About Performance Improvements in JDK 8 
12. HotSpot Virtual Machine Garbage Collection Tuning Guide
13. Getting Started with the G1 Garbage Collector
14. Other JDK 8 articles on Xml and More
15. Tuning that was great in old JRockit versions might not be so good anymore
    
• Trying to bring over each and every tuning option from a JR configuration to an HS one is probably a bad idea.
• Even when moving between major versions of the same JVM, we usually recommend going back to the default (just pick a collector and heap size) and then redoing any tuning work from scratch (if even necessary).
16. HotSpot VM options (JDK 8)

HotSpot: Sizing System Dictionary in JDK 8

The system dictionary is an internal JVM data structure which holds all the classes loaded by the system.  As described in JDK-7114376,^[1]

The System Dictionary hashtable bucket array size is fixed at 1009. This value is too small for large programs with many names, too large for small programs and just about right for medium sized ones. The default should remain at 1009, but it should be possible to override it on the command line.

Finally, this has happened in JDK 8 and you can set hashtable bucket array size to be larger if you have more classes loaded in your applications by using:^[2]

• -XX:+UnlockExperimentalVMOptions  -XX:PredictedLoadedClassCount=<#>

This can help calls like Class.forName(), which do lookups into this data structure.

In this article, we will look into sizing system dictionary in more details.

Performance of System Dictionary

In 1.4.2, there were some performance enhancements.  One of them is making system dictionary reads lock-free.^[3] From this enhancement, you probably can guess that tuning system dictionary could be important to the JVM's performance.

The current number of buckets in hash table for system dictionary is set to be 1009 by default, which is relatively small for an application which has 49987 classes loaded as shown below:

Loaded   Bytes Unloaded   Bytes       Time
49987 110488.8     2453  7743.8     105.24

From the experience, we know:^[5]

In a good hash table, each bucket has zero or one entries, and sometimes two or three, but rarely more than that.

 Assuming the average ideal length of buckets is three, the hashtable bucket array size then should be:

Ideal hashtable bucket array size = 49987 / 3 = 16662

 

System Dictionary Tuning

Seeing 49987classes loaded in our application at end of its run, we decided to set:

-XX:PredictedLoadedClassCount=16661 (a prime)

Note that PredictedLoadedClassCount is an experimental flag.  So, you also need to set:

-XX:+UnlockExperimentalVMOptions

before it in the command line.

When the JVM allocates memory, the largest chunk is the java heap. The system dictionary is in C heap and it is the loaded class cache. The goal of setting PredictedLoadedClassCount flag is to increase the size of the system dictionary in order to make lookups of loaded classes faster. Before every class being loaded, it requires a checking to see if the class is already loaded.  Larger system dictionary will improve JVM performance during this class loading and resolution phase.

Note that the total number of entries in the hashtable does not change— that is based on the number of loaded classes. What sizing of system dictionary do is to increase the spread of the entries so that the average length of the buckets under a single hash result could be reduced. This would reduce the time it takes to find a given entry.

How to Find Number of Loaded Classes?

There are multiple ways to find out number of loaded classes in an application.  For example, you can use -Xverbose:class to determine what classes are loaded.   Another way to find out is what we have done in this article by setting -XX:+PerfDataSaveToFile and then use "jstat -class" command to decipher the class statistics offline:

$<snipped>/jdk-hs/bin/jstat -class file:////slot/myserver/appmgr/APPTOP/instance/domains/slcaf977.us.oracle.com/CRMDomain/hsperfdata_9872

Loaded   Bytes Unloaded   Bytes       Time
49987 110488.8     2453  7743.8     105.24

Acknowledgement

Some writings here are based on the feedback from Mikael Gerdin and Karen Kinnear. However, the author would assume the full responsibility for the content himself.

References

1. Make system dictionary hashtable bucket array size configurable  (JDK-7114376)
2. Tuning the JVM (at 43:02 mark)
3. Java 2 Platform, Standard Edition (J2SE Platform), version 1.4.2 Performance White Paper 
4. The Class Loader Subsystem 
5. Hash table (Wikipedia) 
6. VM Class Loading
• Actually, the HotSpot VM maintains three main hash tables to track class loading
• SystemDictionary  
• PlaceholderTable 
• LoaderConstraintTable
7. Garbage-First Garbage Collector Tuning

HotSpot: What Does {pd product} Mean?

In [1], we have discussed how to find out default HotSpot JVM values.  In the output, you may have spotted:

{pd product}

What does it mean? In this article, we will examine that in depth.

Platform Dependent

{pd product} means platform-dependent product option.  First of all, if a flag is a product option, maybe you want to pay attention to it and may want to change its value for better performance.  If it's an experimental or diagnostic flag, you may want to ignore it.

If an option is {pd product}, it means that its default value will be dependent on which platform its implementation would be compiled on.  Currently, these platforms are supported, but not limited to:

• ARM
• PPC
• Solaris
• x86

Examples

Two of the {pd product} flags, we would like to discuss here further.  In JDK 7u51,^[2] they have the following default values on x86 platforms:

    uintx ReservedCodeCacheSize     = 50331648        {pd product}
    bool  TieredCompilation         = false           {pd product}

These default values have been changed in later JDK 8 builds.  For example, in JDK 8u20, they have the following default values on the same server:

    uintx ReservedCodeCacheSize     = 251658240        {pd product}
    bool  TieredCompilation         = true             {pd product}

As discussed in [3,4], you may want to tune these flags for better performance.

References

1. What Are the Default HotSpot JVM Values?
2. Java™ SE Development Kit 7, Update 51 (JDK 7u51)
3. Performance Tuning with Hotspot VM Option: -XX:+TieredCompilation (XML and More)
4. A Case Study of Using Tiered Compilation in HotSpot (XML and More)  

JRockit: A Case Study of Thread Local Area (TLA) Tuning

Increasing the Thread Local Area (TLA) size is beneficial for multi-threaded applications where each thread allocates a lot of objects. Increasing the TLA size is also beneficial when the average size of the allocated objects is large, as this allows larger objects to be allocated in the TLAs.

In this article, we will present a case of TLA tuning using an application with the following characteristics:

• A multi-threaded application
• Allocating a lot of objects
• Allocating large objects 

TLA Basics

Without much ado, see [1] for the differences between JRockit's large-object implementations between pre- and post-R28.   The version of JRockit used in this article is R28.2.5.  For this version, TLA sizes can be tuned using the following option:

• -XXtlaSize:min=size,preferred=size,wasteLimit=size

Chaning Default TLA Sizes

For our Linux system, the default TLA settings are:

• min=2k
• preferred=16k
• wasteLimit=2k

To tune for better performance, read [2,4] for advice.  For example, one recommendation offered by Oracle is:

• Setting  -XXtlaSize:wasteLimit to the same value as -XXtlaSize:min.

After trials and errors, we have found the following TLA settings to be better than the defaults:

• -XXtlaSize:min=8k,preferred=512k,wasteLimit=8k

Performance Comparison: Default vs. TLA Tuned

Looking at different KPIs, we have found the application's performance improved by 5.9% in Average Response Time (ART) and 3.32% in 90% Response Time.  Better performance is achieved by reducing pause time % in GC and Total CPU % at the expense of total memory footprint (-1.6%).

Conclusion

As the benchmark results show, you can tune your application's performance by tuning TLA sizes. The performance improvement may vary based on your JRockit versions and system capabilities. But, before you do any fine tuning, read [2, 4] first.

Finally, be warned that options that are specified with -XX are not stable and are not recommended for casual use. These options are subject to change without notice.^[3]

References

1. JRockit: Thread Local Area Size and Large Objects
2. Optimizing Memory Allocation Performance (Section 4.4 of this pdf)
3. -XXtlaSize Parameters
4. Oracle JRockit- The Definitive Guide
5. finding L2 cache size in Linux

How to Create Load Testing Scripts Using OpenScript

This article is one of the Oracle Application Test Suite (OATS)^[1] series published on Xml and More, which includes the following:

• Auto-Correlating Session IDs in Oracle Application Test Suite (OATS)
• Removing "Compatibility setting is not supported" Warning before You Do OpenScript Recording
• How to Configure Scripts to use Databanks in OATS
• ORA-01691: unable to extend lob segment
• How to Restart OATS Services

In this article, we will show:

• How to create load testing scripts using OpenScript^[2]

Introduction to OpenScript

Application Testing Suite (OATS) is comprised of several tightly integrated products.^[1] The script designer —OpenScript—only runs on Windows, but all the runtime components are available for both Linux and Windows. OpenScript is a scripting platform for creating automated test scripts in Java.

You can use OpenScript to create scripts for different testings. For example, OATS supports

• Functional Testing
• Load Testing

In this article, we will show you how to create load testing scripts in OpenScript.

The Platform

Scripting platform is based upon the Eclipse open source development environment. Initial OpenScript product provides access to a limited set of the Eclipse development environment.

The workbench is the base layer of software and code that provide the foundation on which the OpenScript Modules and Application Programming Interfaces (APIs) operate. Each Workbench window contains one or more perspectives. OpenScript Workbench provides the following perspectives:

• Tester perspective
• Developer perspective
• Reset perspective

Workspaces are created in Oracle OpenScript. Workspaces store project-related script files and Results Log files. You can use them to organize your various testing projects. Three levels of management are provided:

• Scripts (lowest)
• Folders
• Repositories (highest)

You can download OATS from [3]. The version used in this demonstration is

Version: 12.3.0.1 Build 376

Cheat Sheet

Like every recording task, you need to rehearse and make sure all glitches are resolved before the final recording. If you have decided the click path, prepare a cheat sheet like below:

[1] Bring_up_FUSE_URL
[2] Login_SALESREPUSER00001_Welcome1
[3] Click_the_Opportunities_Card
[4] Select_Quarter_2_2013
[5] Drilldown_on_Pinnacle_Server
[6] Click_on_Sales_Account_Picker
[7] Search_for_CUSTOMER_101328336
[8] Click_Cancel
[9] Click_Add_Revenue_Item
[10] Select_Type_Item
[11] Click_Product_LOV_and_Search
[12] Search_for_Elite_Pro_DG_452
[13] Select_Product_and_click_Ok
[14] Click_Cancel
[15] Logout

The numbering of step is for human readers and can help the recording. Each row on the list corresponds to a click in your click path and will become the title of step group in OpenScript recording.

OpenScript Preferences

Before recording, there are some preferences you want to set. To set them, you click on View and then OpenScript Preferences. For example, we would like to control the grouping, naming and numbering of step groups by ourselves (see "Cheat Sheet"). So, set your "ADF Load" preferences as below:

Creating a New Project

In this demonstration, we will create an open script for load testing (File > New...). Our web application is CRM FUSE. So, we have selected "Oracle Fusion/ADF" wizard from the New Project (see above).

• Oracle Fusion/ADF
• This option lets you create a new script for load testing of Oracle Application Development Framework (ADF)-based applications and other applications that utilize HTTP and ADF protocols at the protocol level.

At the next step, you are asked to provide a Script Name. So, we set them as follows:

• Create script as a Function Library (unchecked)
• Script Name: FUSE_Saleopty_oct07_wrk

Finally click Finish to create a new script. The resulting script will contain the Initialize, Run, and Finish nodes. The Run node will contain recorded HTTP protocol navigations based upon the defined Step Group preferences and the navigations and ADF protocol for actions performed during recording. You can edit the script tree or Java code to customize the script.^[4]

In the following sections, we will demonstrate how to create:

• First step group
• Remaining step groups

Creating First Step Group

Creating the first step group is a bit different from the rest. So, we describe it separately. Note that we have NOT clicked on the record button yet.

Before we click on the record button (i.e., red circle), create the first message group as shown below:

Open your notepad and copy the first row into the title field as shown below. Then click OK.

Note that we have chosen "No delay" for the first step. But, for other steps, we will specify "Delay 44 secs."

Start Recording

Now click the record button. Your chosen browser (for our demonstration, it's Firefox) will be brought up. Copy your URL:

http://www.mycompany.com:9006/customer/faces/CrmFusionHome

into the address field and hit Enter. This finishes the recording of the first step group.

Next repeat the following subtasks for the remaining groups until it finishes:

1. Creating a new step group in OpenScript
• Right select previous Step Group to bring up context menu and select New > Step Group
2. Copying the next row of click path from Notepad
3. Clicking next step in your Browser

Finally, don't forget to stop the recorder.

Exporting Script

If your runtime environment is Linux, you need to export script created in OpenScript as follows:

File > Export Script...

For example, a new zip file was created in our default repository:

D:\OracleATS\OFT\FUSE_Saleopty_oct07_wrk.zip

You can then copy it to your Linux box:

scp FUSE_Saleopty_Server1_wrk.zip aime1@mylinuxserver:/scratch/aime1/work

References

1. Oracle Application Testing Suite
2. OpenScript for Load Testing Script Troubleshooting (Tutorial)
• Version: 12.3.0.1 Build 376 was used in this article.
• This version requires Firefox 10.0 ESR (Windows download).
3. Oracle Application Testing Suite Downloads
4. Oracle Application Testing Suite 12.x: Oracle Load Testing Overview
5. OATS: Tie All Processes Together — from OpenScript to Scenario (Xml and More)

WebLogic Startup Slowness Caused by Kernel's Random Number Generator

Java Application (i.e., WebLogic Server) could be slow at startup time and it could be caused by the slowness of random number generator used by the application.  You can read [2] for the case that discusses the slowness of WebLogic startup.

In this article, we will examine the following issues:

• /dev/random vs. /dev/urandom
• How to test the performance of a random number generator?
• How to configure it?
• Security considerations

on Linux systems.  Note that this can happen with WLS running on AIX too.

/dev/random vs. /dev/urandom

Without much ado, here is the man output for "urandom":

The character special files /dev/random and /dev/urandom (present since Linux 1.3.30) provide an interface to the kernel's random number generator.  File /dev/random has major device number 1 and minor device number 8.  File /dev/urandom has major device number 1  and  minor  device number 9.

The  random  number  generator  gathers environmental noise from device drivers and other sources into an entropy  pool.   The  generator  also keeps  an  estimate of the number of bits of noise in the entropy pool.  From this entropy pool random numbers are created. 

When read, the /dev/random device will only return random bytes  within the estimated number of bits of noise in the entropy pool.  /dev/random should be suitable for uses that need very high quality randomness such as  one-time  pad  or  key generation.  When the entropy pool is empty, reads from /dev/random will block until additional environmental  noise is gathered. 

A  read  from  the  /dev/urandom device will not block waiting for more entropy.  As a result, if  there  is  not  sufficient  entropy  in  the entropy  pool,  the  returned  values are theoretically vulnerable to a cryptographic attack on the algorithms used by the  driver.   Knowledge of how to do this is not available in the current non-classified literature, but it is theoretically possible that such an attack may  exist.  If this is a concern in your application, use /dev/random instead.

How to Test?

You can use "time" command to measure the performance of each random number generator.  For example, here is the output from the Linux system.

$time head -1 /dev/random                                                  
real    0m9.718s
user    0m0.000s
sys     0m0.001s

$ time head -1 /dev/./urandom

real    0m0.002s

user    0m0.000s

sys     0m0.002s

As you can see that "/dev/urandom" is much faster because it's non-blocking.  However, /dev/random will block until additional environmental noise is gathered and takes longer time to return.

How to Configure?

You can configure which source of seed data for SecureRandom to use at JVM level or at WLS' command-line level.

At the JVM level, you can change the value of securerandom.source property in the file:

• $JAVA_HOME/jre/lib/security/java.security

Here is the description of securerandom.source property:

# Select the source of seed data for SecureRandom. By default an
# attempt is made to use the entropy gathering device specified by
# the securerandom.source property. If an exception occurs when
# accessing the URL then the traditional system/thread activity
# algorithm is used.
#
# On Solaris and Linux systems, if file:/dev/urandom is specified and it
# exists, a special SecureRandom implementation is activated by default.
# This "NativePRNG" reads random bytes directly from /dev/urandom.
#
# On Windows systems, the URLs file:/dev/random and file:/dev/urandom
# enables use of the Microsoft CryptoAPI seed functionality.
#
securerandom.source=file:/dev/urandom

Or, you can specify which source of seed data to use by adding

• -Djava.security.egd=file:/dev/./urandom

to the java command-line that starts WebLogic Server.

Security Considerations

In [1], it warns that if you choose /dev/urandom over /dev/random for better performance, you should be aware of that:

This workaround should not be used in production environments because it uses pseudo-random numbers instead of genuine random numbers.

References

1. Random Number Generator May Be Slow on Machines With Inadequate Entropy
2. Weblogic starts slow
3. Fusion Middleware Performance and Tuning for Oracle WebLogic Server
4. Oracle® Fusion Middleware Tuning Performance of Oracle WebLogic Server 12c (12.2.1)
5. Fusion Middleware Tuning Performance of Oracle WebLogic Server (12.2.1.3.0)

Understanding String Table Size in HotSpot

In JDK-6962930^[2], it requested that string table size be configurable.  The resolved date of that bug was on 04/25/2011 and it's available in JDK 7.  In another JDK bug^[3], it has requested the default size (i.e. 1009) of string table be increased.

In this article, we will examine the following topics:

• What string table is
• How to find the number of interned strings in your applications
• The tradeoff between memory footprint and lookup cost

String Table

In Java, string interning^[1] is a method of storing only one copy of each distinct string value, which must be immutable. Interning strings makes some string processing tasks more time- or space-efficient at the cost of requiring more time when the string is created or interned. The distinct values are stored in a string intern pool, which is the string table in HotSpot.

The size of the string table (i.e., a chained hash table) is configurable in JDK 7.  When the overflow chains become long, performance can degrade.  The current default size of string table is 1009 (or 1009 buckets), which is too small for applications that stress the string table.  Note that the string table itself is allocated in native memory but the strings are java objects.

Increasing the size improves performance (i..e, reducing look-up cost) but increases the StringTable size by 16 bytes on 64-bit systems, 8 bytes on 32-bit systems for every additional entry.  For example, changing the default size to 60013 increases the String Table size by 460K on 32 bit systems.

Finding Number of Interned Strings in the Applications

In HotSpot, it provides a product level option named PrintStringTableStatistics which can be used to print hash table statistics^[4].  For example, using one of our applications (hereafter will be referred as JavaApp), it prints out the following information:

StringTable statistics:
Number of buckets  : 60013
Average bucket size  : 5
Variance of bucket size : 5
Std. dev. of bucket size: 2
Maximum bucket size  : 17

You can find the above output from your manged server's log file in the WebLogic domain.  Note that we have set the following option:

• -XX:StringTableSize=60013

So, there are 60013 buckets in the hash table (or string table).

In JDK, there is also a tool named jmap which can be used to find out number of interned strings in your application.  For example, we have found the following information using:

$ jdk-hs/bin/jmap -heap 18974
Attaching to process ID 18974, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.0-b43

using thread-local object allocation.
Parallel GC with 18 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 2147483648 (2048.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 402653184 (384.0MB)
   MaxPermSize      = 402653184 (384.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
PS Young Generation
<deleted for brevity>

270145 interned Strings occupying 40429904 bytes.

Therefore, we know there are around 260K interned Strings in the table.

Tradeoff Between Memory Footprint and Lookup Cost

Based on curiosity, we have tried to set the string table size to be 277331 (a prime number) to see how JavaApp performs.  Here are our findings:

• Average Response Time: +0.75%
• 90% Response Time: +0.56%

However, the memory footprint has increased:

• Total Memory Footprint: -1.03%

Finally, here is the hash table statistics based on the new size (i.e., 277331):

StringTable statistics:
Number of buckets       :  277331
Average bucket size     :       1
Variance of bucket size :       1
Std. dev. of bucket size:       1
Maximum bucket size     :       8

The conclusion is that increasing string table size from 60013 to 277331 helps JavaApp's performance a little bit at the expense of larger memory footprint.  In this case, the benefit is minimal, keeping string table size to be 60013 is good enough.

References

1. String Interning (Wikipedia)
2. JDK 6962930 : make the string table size configurable
3. JDK 8009928: Increase default value for StringTableSize
4. Java GC tuning for strings
5. All other performance tuning articles on XML and More
6. G1 GC Glossary of Terms

Analyzing the Performance Issue Caused by WebLogic Session Size Too Big

In this article, we will show you lots of interesting stuffs:

1. How to investigate an issue with too many live objects kept in old generation, which cannot be reclaimed in Full GC by HotSpot VM
• For background information regarding Garbage Collectors, read [1-7]
2. How important sesssion-timeout element in web.xml is
3. How to use java tool to generate heap dump
4. How to use Eclipse Memory Analyzer to analyze heap dump (i.e. fod.hprof file)
• Download link: here

The Issues

We have seen slow performance in one of our benchmarks (i.e., FOD).  Here are the symptoms:

• GC took about 50% of the application's CPU time
• Full GC's were not triggered by full Perm Gen, but by full Old Gen
• The old generation is completely full and it isn't clearing up anything but a few soft references during Full GC.

So, the symptoms all point to too many live objects kept in Old Gen and they have caused many Full GC's.  There could be different causes for the above symptoms.  To investigate, you need to create heap dumps and check out what objects HotSpot VM is holding onto.  Below we will show you how to investigate this.

Generating Heap Dump

First, we have used a Java tool jmap to generate a summary of the heap contents:

$./jmap -histo 16345 >/scratch/aime1/tmp/fod.summary

 num     #instances         #bytes  class name
----------------------------------------------
   1:      15777231      822030680  [Ljava.lang.Object;
   2:       3074270      463615192  [C
   3:       4076972      331840256  [Ljava.util.HashMap$Entry;
   4:       9287101      297187232  java.util.HashMap$Entry
   5:       2105437      151591464  org.apache.myfaces.trinidad.bean.util.PropertyHashMap
   6:       4075444       97810656  java.util.HashMap$FrontCache
   7:       1893014       90864672  java.util.HashMap
   8:       2018043       80721720  org.apache.myfaces.trinidad.bean.util.FlaggedPropertyMap
   9:       3017161       72411864  java.lang.String

Unfortunately, it didn't tell us much regarding the biggest java object that have taken ~800M bytes.  But, we did notice that two other property maps also took up a lot of memory space.  So, the next step is to generate a full dump as follows:

$./jmap -dump:live,file=/scratch/aime1/tmp/fod.hprof 16345

To analyze the full heap dump, you need to use Eclipse Memory Analyzer which we have chosen the standalone version.

Analyzing Heap Dump With Memory Analyzer (MAT)

I started the Memory Analyzer with the following command:

$ cd /scratch/sguan/mat/

$ ./MemoryAnalyzer &

When I opened the heap dump file (i.e., fod.hprof), MAT failed with a message related to Java Heap Space.  So, I need to modified the following line in the MemoryAnalyzer.ini file:

-vmargs

-Xmx1024m

by changing -Xmx1024m to -Xmx6240m.  Note that how big you should set for -Xmx option depends on:

• Size of heap dump
• Our original file size is 6.4g and was reduced to 3.6g after loading.  Note that MAT will remove unreachable objects in the loading process.
• Size of physical RAM in your system

The Culprit— WebLogic Session Objects

From the MAT, we have found out that the biggest live object kept in the heap was:

• weblogic.servlet.internal.session.MemorySessionContext @ 0x705e0ff08 

Also, for two previously-mentioned property map objects, they are also related to the session object.  If you trace those objects back to the GC root, you will see that they are pointed to by the session state. They are the reason the session state is so big.

session-timeout Element in web.xml

Session object is used by WebLogic container to track a user's session that uses the StoreFrontModule web application in our FOD benchmark.  Every time a new user comes in, it starts a session. This session object keeps user's data or states in it. If the user leaves, the session is considered idle and the server will retain it in memory for a period of time before reclaiming it.

Java web applications can be configured with a session timeout value which specifies the number of minutes a session can be idle before it is abandoned.  This session timeout element is defined in the Web Application Deployment Descriptor web.xml^[9]. For example, this was the session timeout value we had:

<session-config>
<session-timeout>35</session-timeout>
</session-config>

The problem with our benchmark's slowness is due to this session timeout value being too high.  This has caused the sessions being piled up and taking up lots of memory. Setting it to be a lower value (see [10] on how to patch web.xml in an EAR), our benchmark then performs normally.

References

1. Java Performance by Charlie Hunt and Binu John
2. Understanding Garbage Collection
3. Java HotSpot VM Options
4. GCViewer (a free open source tool)
5. Understanding Garbage Collector Output of Hotspot VM
6. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded
7. HotSpot VM Performance Tuning Tips
8. java.lang.OutOfMemoryError – Weblogic Session size too big
9. Web Application Deployment Descriptor schema
10. WARNING: VERSION_DISPLAY_ENABLED_IN_PRODUCTION_STAGE
11. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston

HotSpot VM Performance Tuning Tips

In some cases, it may be obvious from benchmarking that parts of an application need to be rewritten using more efficient algorithms^[1]. Sometimes it may just be enough to provide a more optimal runtime environment by tuning the JVM parameters.

In this article, we will show you some of the HotSpot VM performance tuning tips.

What to tune?

You can tune HotSpot performance on multiple fronts:

• Code generation^[8,10]
• Memory management

In this article, we will focus more on memory management (or garbage collector).  The goals of tuning garbage collector include:

• To make a garbage collector operate efficiently, by
• Reducing pause time or
• Increasing throughput
• To avoid heap fragmentation
• Different garbage collector uses different compaction to eliminate fragmentation
• To make it scalable for multithreaded applications on multiprocessor systems

In this article, we will cover the following tuning options:

1. Client VM or Server VM
2. 32-bit VM or 64-bit VM
3. GC strategy
4. Heap sizing
5. Further tuning

Client vs. Server VM

The HotSpot Client JVM has been specially tuned to reduce application startup time and memory footprint, making it particularly well suited for client environments. On all platforms, the HotSpot Client JVM is the default.

The Java HotSpot Server VM is similar to the HotSpot Client JVM except that it has been specially tuned to maximize peak operating speed. It is intended for long-running server applications, for which the fastest possible operating speed is generally more important than having the fastest startup time. To invoke the HotSpot Server JVM instead of the default HotSpot Client JVM, use the -server parameter; for example,

• java -server MyApp

In [7], authors have mentioned a third HotSpot VM runtime named tiered.   If you are using Java 6 Update 25, Java 7, or later, you may consider using tiered server runtime as a replacement for the client runtime.   For more details, read [8,10].

32-Bit or 64-Bit VM

The 32-bit JVM is the default for the HotSpot VM. The choice of using a 32-bit or 64-bit JVM is dictated by the memory footprint required by the application along with whether any third-party software used in the application supports 64-bit JVMs and if there are any native components in the Java application. All native components using the Java Native Interface (JNI) in a 64-bit JVM must be compiled in 64-bit mode.

Running 64-bit VM has the following advantages^[7]:

• Larger address space
• Better performance in two fronts
• 64-bit JVMs can make use of additional CPU registers
• Help avoid register spilling
• Register spilling occurs where there is more live state (i.e. variables) in the application than the CPU has registers. 

and one disadvantage:

• Increased width for oops
• Results in fewer oops being available on a CPU cache line and as a result decreases CPU cache efficiency. 
• This negative performance impact can be mitigated by setting:
• -XX:+UseCompressedOops VM command line option

Note that client runtimes are not available in 64-bit HotSpot VMs.   See [11] for more details.

GC Strategy

JVM performance is usually measured by its GC's effectiveness.  Garbage collection (GC) reclaims the heap space previously allocated to objects no longer needed. The process of locating and removing those dead objects can stall your Java application while consuming as much as 25 percent of throughput.

The Java HotSpot virtual machine includes five garbage collectors.^[27] All the collectors are generational.

• Serial Collector
• Both young and old collections are done serially (using a single CPU), in a stop-the-world fashion.
• The old and permanent generations are collected via a mark-sweep-compact collection algorithm. 
• The sweep phase “sweeps” over the generations, identifying garbage. The collector then performs sliding compaction, sliding the live objects towards the beginning of the old generation space (and similarly for the permanent generation), leaving any free space in a single contiguous chunk at the opposite end.
• When to use
• For most applications that are run on client-style machines and that do not have a requirement for low pause times
• How to select
• In the J2SE 5.0 and above, the serial collector is automatically chosen as the default garbage collector on machines that are not server-class machines. On other machines, the serial collector can be explicitly requested by using the -XX:+UseSerialGC command line option.
• Parallel Collector (or throughput collector)
• Young generation collection
• Uses a parallel version of the young generation collection algorithm utilized by the serial collector
• It is still a stop-the-world and copying collector, but performing the young generation collection in parallel, using many CPUs, decreases garbage collection overhead and hence increases application throughput.
• Old generation collection
• Uses the same serial mark-sweep-compact collection algorithm as the serial collector
• When to use
• For applications run on machines with more than one CPU and do not have pause time constraints, since infrequent, but potentially long, old generation collections will still occur. 
• Examples of applications for which the parallel collector is often appropriate include those that do batch processing, billing, payroll, scientific computing, and so on.
• How to select
• In the J2SE 5.0 and above, the parallel collector is automatically chosen as the default garbage collector on server-class machines. On other machines, the parallel collector can be explicitly requested by using the -XX:+UseParallelGC command line option.
• Parallel Compacting Collector
• Young generation collection
• Use the same algorithm as that for young generation collection using the parallel collector.
• Old generation collection
• The old and permanent generations are collected in a stop-the-world, mostly parallel fashion with sliding compaction
• The collector utilizes three phases (see [5] for more details):
• Marking phase
• Summary phase
• Compaction phase
• When to use
• For applications that are run on machines with more than one CPU and applications that have pause time constraints.
• How to select
• If you want the parallel compacting collector to be used, you must select it by specifying the command line option -XX:+UseParallelOldGC.
• Concurrent Mark-Sweep (CMS) Collector^[5,6]
• Young generation collection
• The CMS collector collects the young generation in the same manner as the parallel collector. 
• Old generation collection
• Most of the collection of the old generation using the CMS collector is done concurrently with the execution of the application. 
• The CMS collector is the only collector that is non-compacting. That is, after it frees the space that was occupied by dead objects, it does not move the live objects to one end of the old generation. 
• To minimize risk of fragmentation, CMS is doing statistical analysis of object’s sizes and have separate free lists for objects of different sizes.
• When to use
• For application needs shorter garbage collection pauses and can afford to share processor resources with the garbage collector when the application is running. (Due to its concurrency, the CMS collector takes CPU cycles away from the application during a collection cycle.)
• Typically, applications that have a relatively large set of long-lived data (a large old generation), and that run on machines with two or more processors, tend to benefit from the use of this collector. 
• Compared to the parallel collector, the CMS collector decreases old generation pauses—sometimes dramatically—at the expense of slightly longer young generation pauses, some reduction in throughput, and extra heap size requirements.
• How to select
• If you want the CMS collector to be used, you must explicitly select it by specifying the command line option -XX:+UseConcMarkSweepGC.  
• If you want it to be run in incremental mode, also enable that mode via the  –XX:+CMSIncrementalMode option.
• This feature is useful when applications that need the low pause times provided by the concurrent collector are run on machines with small numbers of processors (e.g., 1 or 2).
• ParNewGC is the parallel young generation collector for use with CMS.  To choose it, you can specify the command line option  ‑XX:+UseParNewGC.
• Generation First (G1) Garbage Collector^[31]
• Summary
• Differs from CMS in the following ways
• Compacting
• Reduce fragmentation and is good for long-running applications 
• Heap is split into regions
• Easy to allocate and resize
• Evacuation pauses 
• For both young and old regions
• Young generation collection
• During a young GC, survivors from the young regions are evacuated to either survivor regions or old regions
• Done with stop-the-world evacuation pauses
• Performed in parallel
• Old generation collection
• Some garbage objects in regions with very high live ratio may be left in the heap and be collected later
• Concurrent marking phase
• Calculates liveness information per region
• Empty regions can be reclaimed immediately
• Identifies best regions for subsequent evacuation pauses
• Remark is done with one stop-the-world puase while initial mark is piggybacked on an evacuation pause
• No corresponding sweeping phase
• Different marking algorithm than CMS
• Old regions are reclaimed by
• Evacuation pauses
• Using compaction
• Where most reclamation is done
• Remark (when totally empty)
• When to use
• The G1 collector is a server-style garbage collector, targeted for multi-processor machines with large memories. It meets garbage collection (GC) pause time goals with high probability, while achieving high throughput. 
• How to select
• If you want the G1 garbage collector to be used, you must explicitly select it by specifying the command line option -XX:+UseG1GC.  

Note that the difference between:

• -XX:+UseParallelOldGC

and

• -XX:+UseParallelGC

is that -XX:+UseParallelOldGC enables both a multithreaded young generation garbage collector and a multithreaded old generation garbage collector, that is, both minor garbage collections and full garbage collections are multithreaded. -XX:+UseParallelGC enables only a multithreaded young generation garbage collector. The old generation garbage collector used with -XX:+UseParallelGC is single threaded. 

Using -XX:+UseParallelOldGC also automatically enables -XX:+UseParallelGC. Hence, if you want to use both a multithreaded young generation garbage collector and a multithreaded old generation garbage collector, you need only specify -XX:+UseParallelOldGC.

Note that the above distinction between -XX:+UseParallelOldGC and -XX:+UseParallelGC are no longer true in JDK 7.  In JDK 7, the following three settings are equivalent:

• Default
• -XX:+UseParallelGC
• -XX:+UseParallelOldGC

They all use multithreaded collectors for both young generation and old generation.

Heap Sizing

If a heap size is small, collection will be fast but the heap will fill up more quickly, thus requiring more frequent collections. Conversely, a large heap will take longer to fill up and thus collections will be less frequent, but they may take longer.

Command line parameters that divide the heap between new and old generations usually cause the greatest performance impact.  If you increase the new generation's size, you often improve the overall throughput; however, you also increase footprint, which may slow down servers with limited memory.

For more details, you can read [12-15].

Further Tuning

HotSpot's default parameters are effective for most small applications that require faster startup and a smaller footprint.  However, more often than not, you will find default settings are not good enough and need further tuning your Java applications.  As shown in [16], there are many VM options exoosed and can be further tuned by brave souls.  I'm not going to discuss such tunings in this article.  But, I'll keep on posting articles on this blogger with VM tunings.  Stay tuned!

References

1. Java Performance Tips
2. Pick up performance with generational garbage collection
3. Java HotSpot™ Virtual Machine Performance Enhancements
4. Oracle JRockit
5. Memory Management in the Java HotSpot™ Virtual Machine
6. Understanding GC pauses in JVM, HotSpot's CMS collector
7. Java Performance by Charlie Hunt and Binu John
8. Performance Tuning with Hotspot VM Option: -XX:+TieredCompilation
9. Java Tuning White Paper
10. A Case Study of Using Tiered Compilation in HotSpot
11. HotSpot VM Binaries: 32-Bit vs. 64-Bit
12. HotSpot Performance Option — SurvivorRatio
13. A Case Study of java.lang.OutOfMemoryError: GC overhead limit exceeded
14. Understanding Garbage Collection
15. Diagnosing Java.lang.OutOfMemoryError
16. What Are the Default HotSpot JVM Values?
17. Understanding Garbage Collector Output of Hotspot VM
18. On Stack Replacement in HotSpot JVM
19. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston
20. Sun Performance and Tuning: Java and the Internet by Adrian Cockroft and Richard Pettit
21. Concurrent Programming in Java: Design Principles and Patterns by Doug Lea
22. Capacity Planning for Web Performance: Metrics, Models, and Methods by Daniel A. Menascé and Virgilio A.F. Almeida
23. Java Performance Tuning (Michael Finocchiaro)
24. Diagnosing Heap Stress in HotSpot
25. Introduction to HotSpot JVM Performance and Tuning
26. Tuning the JVM (video)
• Frequency of Minhor GC dictated by:
• Application object allocation rate
• Size of Eden
• Frequency of object promotion dictated by:
• Frequency of minor GCs (tenuring)
• Size of survivor spaces
• Full GC Frequency dictated by
• Promotion rate
• Size of old generation
27. JEP 173: Retire Some Rarely-Used GC Combinations
28. G1 GC Glossary of Terms
29. Learn More About Performance Improvements in JDK 8 
30. Java SE HotSpot at a Glance
31. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)
32. Tuning that was great in old JRockit versions might not be so good anymore
    
    • Trying to bring over each and every tuning option from a JR configuration to an HS one is probably a bad idea.
    • Even when moving between major versions of the same JVM, we usually recommend going back to the default (just pick a collector and heap size) and then redoing any tuning work from scratch (if even necessary).

Dynamically Sizing JDBC Connection Pool in WebLogic Server

A data source in WebLogic Server has a set of properties that define the initial, minimum, and maximum number of connections in the pool. A data source automatically adds one connection to the pool when all connections are in use. When the pool reaches maxCapacity, the maximum number of connections are opened, and they remain opened unless you enable automatic shrinking on the data source or manually shrink the data source.

In this article, we will discuss the trade-offs between memory footprint and CPU utilization in the task of JDBC connection pool sizing. Before you start, you may want to read this companion article first:

• Monitoring WebLogic JDBC Connection Pool at Runtime

Fixed-Sized vs Dynamically-Sized Pool

Sometimes you would like to set the initial capacity to the same value as the maximum capacity—this way, the connection pool will have all the physical connections ready when the pool is initialized. However, sometimes it's not possible to estimate what your run-time workloads (either average or peak load) would be in advance and it could become wasteful to over-allocate connection instances. Then dynamically-sized pool may be the better approach.

Monitoring JDBC Connection Statistics

As shown above, you can navigate to:

• Services -> Data Sources -> ApplicationDB -> Monitoring -> Statistics

and monitor the connection statistics of a specific data source (i.e., "ApplicationDB").

In our case, ApplicationDB was deployed to multiple servers. As you can see, the active connections on each server is low (i.e., maximum is 6). However, we have set its Initial Capacity to be 20 and all five pools inherit the setting and have a current capacity of 20.

Also, in our case, only SalesServer_1 will ever need over 20 connections concurrently and allocating 20 connections for all pools can be wasteful. So, based on your own situation, you may want to reduce ApplicationDB's initial capacity appropriately.

After you estimate your peak load, you can choose a Maximum Capacity for the data source. In this case, initial and maximum capacity will be different. Then you can configure the way the pool can shrink and grow by using two additional properties:

• Shrink Frequency
• The number of seconds to wait before shrinking a connection pool that has incrementally increased to meet demand.
• When set to 0, shrinking is disabled.
• Minimum Capacity
• The minimum number of physical connections that this connection pool can contain after it is initialized.

You may want to drop some connections from the data source when a peak usage period has ended, freeing up WebLogic Server and DBMS resources. When you shrink a data source, WebLogic Server reduces the number of connections in the pool to the greater of either the Minimum Capacity or the number of connections currently in use.

For best performance, you should always tune pool sizes based on DataSource statistics.

References

1. Monitoring WebLogic JDBC Connection Pool at Runtime
2. Oracle® Fusion Middleware Configuring and Managing JDBC Data Sources for Oracle WebLogic Server 11g Release 1 (10.3.4)
3. Configuring JDBC Data Sources in JDeveloper and Oracle WebLogic Server
4. Monitoring and Tuning Oracle Fusion Applications
5. Why My WebLogic Managed Server is in ADMIN State?
• Read this for a good example of when to set Initial Capacity to be zero.
6. JBO-26061: Error while opening JDBC connection
7. Tuning Data Sources (12.2.1.3.0) 
8. Top Tuning Recommendations for WebLogic Server (12.2.1.3.0)

A Case Study of Using Tiered Compilation in HotSpot

Hotspot has two JITs named c1 (i.e., client JIT) and c2 (i.e., server JIT).^[1] The client JIT starts fast but provides less optimizations. So, it is used for GUI application. The server JIT starts more slowly but provide very good optimizations. The idea of tiered compilation ^[2]is to get the best of both compilers, first JITs the code with c1 and then if the code is really hot to recompile it with c2.

The tiered server runtime is enabled with the following Hotspot VM options:

• -server -XX:+TieredCompilation

In this article, we show a case study of using HotSpot (build 23.0-b18, mixed mode) with TieredCompilation off/on.

Comparison

You can tune JVM's performance by tuning either memory management or code generaion.  Here we have tuned code generator by turning tiered compilation mode on.  With tiered compilation on, more classes are compiled (or compiled more efficiently) and so they execute faster (i.e., +15%).

Using ATG CRM Demo benchmark, we saw the following KPI changes:

	-XX:-TieredCompilation	-XX:+TieredCompilation	% Change
Total Footprint	4431MB	4686MB	-5.4%
Application Server CPU	23%	20%	+15%
Average Response Time	0.234	0.217	+7.8%

Notes

1. Here is our setting for ReservedCodeCacheSize^[2]:
   -TieredCompilation: 128MB
   +TieredCompilation: 256MB
2. When we turned on tiered compilation mode, we also reserve larger code cache for it (i.e., 256 MB vs. 128 MB).  Code cache is allocated out of native memory (vs. heap).  Total footprint shown in the table includes native memory.  So, we see total footprint is larger when tiered compilation mode is turned on.

References

1. HotSpot Glossary of Terms
2. Performance Tuning with Hotspot VM Option: -XX:+TieredCompilation (Xml and More)