Does SPECjbb 2015 Have Less Variability than SPECjbb2013?

What's the advantage of using SPECjbb 2015 over SPECjbb 2013? The answer may be "less variability". In this article, I'm going to examine the credibility of this claim.

Note that the experiments run here are not intended to evaluate the ultimate performance of JVMs (OpenJDK vs. HotSpot) with different configurations. Different JDK 7 versions were used in OpenJDK and HotSpot here. But, the author believes that the differences between builds of OpenJDK and HotSpot should be minor.

To recap, the main purpose of this article is comparing the variability of SPECjbb 2013 and SPECjbb 2015.

OpenJDK vs. HotSpot

To begin with , we want to emphasize that OpenJDK and HotSpot actually share most of the code bases. Henrik Stahl, VP of Product Management in the Java Platform Group at Oracle, has shed some light on it:

Q: What is the difference between the source code found in the OpenJDK repository, and the code you use to build the Oracle JDK?

A: It is very close - our build process for Oracle JDK releases builds on OpenJDK 7 by adding just a couple of pieces, like the deployment code, which includes Oracle's implementation of the Java Plugin and Java WebStart, as well as some closed source third party components like a graphics rasterizer, some open source third party components, like Rhino, and a few bits and pieces here and there, like additional documentation or third party fonts. Moving forward, our intent is to open source all pieces of the Oracle JDK except those that we consider commercial features such as JRockit Mission Control (not yet available in Oracle JDK), and replace encumbered third party components with open source alternatives to achieve closer parity between the code bases.

Raw Data

Common Settings of the Tests

• SPECjbb
• OOTB settings
• -m composite
• JDK 7
• 4GB heap 
• Parallel GC (default)
• OEL 6.6
• Data sampling
• Total Runs: 5 for each test
• Metrics
• Sample Standard Deviation (Std Dev)
• Coefficient of variation (CV)

Different Settings of Tests

• Test 1
• OpenJDK 
• Huge Pages:^[7] 0
• Test 2
• OpenJDK 
• Huge Pages: 2200
• Test 3
• HotSpot
• Huge Pages: 0
• Test 4
• HotSpot
• Huge Pages: 2200

SPECjbb 2013	Test 1			Test 2			Test 3			Test 4
	Mean	Std Dev	CV	Mean	Std Dev	CV	Mean	Std Dev	CV	Mean	Std Dev	CV
max-jOPS	2973.40	76.01	2.56%	3490.60	92.54	2.65%	3124.20	43.22	1.38%	3538	71.98	2.03%
critical-jOPS	748.40	70.02	9.36%	852.20	76.37	8.96%	741.60	36.28	4.89%	828.8	76.54	9.24%

SPECjbb 2015	Test 1			Test 2			Test 3			Test 4
	Mean	Std Dev	CV	Mean	Std Dev	CV	Mean	Std Dev	CV	Mean	Std Dev	CV
max-jOPS	2853.80	55.22	1.93%	3233.40	82.32	2.55%	2893.20	53.89	1.86%	3238.60	92.89	2.87%
critical-jOPS	514	20.21	3.93%	581.80	45.26	7.78%	522.40	18.90	3.62%	571.6	36.67	6.42%

Analysis

Coefficient of variation (CV) is used for the comparison of experiment's variability.  Based on 5 sample points of each test, we have come to the following preliminary conclusions:

• SPECjbb 2013 vs SPECjbb 2015
• On critical-jOPS, SPECjbb 2015 exhibits lower extent of variability in all tests.
• On max-jOPS, SPECjbb 2015 exhibits lower extent of variability in all tests except Test 3&4.
• Enabling Huge Pages or not
• In both SPECjbb 2013 & 2015, enabling huge pages exhibits higher extent of variability (exception: SPECjbb 2013/OpenJDK/critical-jOPS)
• OpenJDK vs HotSpot
• Both JVMs exhibit similar performance in average (i.e., max- or critical- jOPS) .

By no means our results are conclusive.  As more tests are run in the future, we could gain a better picture.  From the preliminary results, it seems to be wise of choosing SPECjbb 2015 over SPECjbb 2013 in future Java SE performance tests.

References

1. Coefficient of variation
2. When to use the sample or population standard deviation
3. SPECjbb2015_release_notes.txt
4. SPECjbb2013 Result File Fields
5. Java 7 Questions & Answers
6. java - Difference between JVM and HotSpot?
7. Huge Pages
• How to test it?
• How to set it?
8. Java Mission Control
• Java Mission Control (JMC) is for Hotspot as JRockit Mission Control (JRMC) is for JRockit.
9. Standard Deviation Calculator

Understanding String Table Size in HotSpot

In JDK-6962930^[2], it requested that string table size be configurable.  The resolved date of that bug was on 04/25/2011 and it's available in JDK 7.  In another JDK bug^[3], it has requested the default size (i.e. 1009) of string table be increased.

In this article, we will examine the following topics:

• What string table is
• How to find the number of interned strings in your applications
• The tradeoff between memory footprint and lookup cost

String Table

In Java, string interning^[1] is a method of storing only one copy of each distinct string value, which must be immutable. Interning strings makes some string processing tasks more time- or space-efficient at the cost of requiring more time when the string is created or interned. The distinct values are stored in a string intern pool, which is the string table in HotSpot.

The size of the string table (i.e., a chained hash table) is configurable in JDK 7.  When the overflow chains become long, performance can degrade.  The current default size of string table is 1009 (or 1009 buckets), which is too small for applications that stress the string table.  Note that the string table itself is allocated in native memory but the strings are java objects.

Increasing the size improves performance (i..e, reducing look-up cost) but increases the StringTable size by 16 bytes on 64-bit systems, 8 bytes on 32-bit systems for every additional entry.  For example, changing the default size to 60013 increases the String Table size by 460K on 32 bit systems.

Finding Number of Interned Strings in the Applications

In HotSpot, it provides a product level option named PrintStringTableStatistics which can be used to print hash table statistics^[4].  For example, using one of our applications (hereafter will be referred as JavaApp), it prints out the following information:

StringTable statistics:
Number of buckets  : 60013
Average bucket size  : 5
Variance of bucket size : 5
Std. dev. of bucket size: 2
Maximum bucket size  : 17

You can find the above output from your manged server's log file in the WebLogic domain.  Note that we have set the following option:

• -XX:StringTableSize=60013

So, there are 60013 buckets in the hash table (or string table).

In JDK, there is also a tool named jmap which can be used to find out number of interned strings in your application.  For example, we have found the following information using:

$ jdk-hs/bin/jmap -heap 18974
Attaching to process ID 18974, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.0-b43

using thread-local object allocation.
Parallel GC with 18 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 2147483648 (2048.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 402653184 (384.0MB)
   MaxPermSize      = 402653184 (384.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
PS Young Generation
<deleted for brevity>

270145 interned Strings occupying 40429904 bytes.

Therefore, we know there are around 260K interned Strings in the table.

Tradeoff Between Memory Footprint and Lookup Cost

Based on curiosity, we have tried to set the string table size to be 277331 (a prime number) to see how JavaApp performs.  Here are our findings:

• Average Response Time: +0.75%
• 90% Response Time: +0.56%

However, the memory footprint has increased:

• Total Memory Footprint: -1.03%

Finally, here is the hash table statistics based on the new size (i.e., 277331):

StringTable statistics:
Number of buckets       :  277331
Average bucket size     :       1
Variance of bucket size :       1
Std. dev. of bucket size:       1
Maximum bucket size     :       8

The conclusion is that increasing string table size from 60013 to 277331 helps JavaApp's performance a little bit at the expense of larger memory footprint.  In this case, the benefit is minimal, keeping string table size to be 60013 is good enough.

References

1. String Interning (Wikipedia)
2. JDK 6962930 : make the string table size configurable
3. JDK 8009928: Increase default value for StringTableSize
4. Java GC tuning for strings
5. All other performance tuning articles on XML and More
6. G1 GC Glossary of Terms