Note that the experiments run here are not intended to evaluate the ultimate performance of JVMs (OpenJDK vs. HotSpot) with different configurations. Different JDK 7 versions were used in OpenJDK and HotSpot here. But, the author believes that the differences between builds of OpenJDK and HotSpot should be minor.
To recap, the main purpose of this article is comparing the variability of SPECjbb 2013 and SPECjbb 2015.
OpenJDK vs. HotSpot
To begin with , we want to emphasize that OpenJDK and HotSpot actually share most of the code bases. Henrik Stahl, VP of Product Management in the Java Platform Group at Oracle, has shed some light on it:
Q: What is the difference between the source code found in the OpenJDK repository, and the code you use to build the Oracle JDK?
A: It is very close - our build process for Oracle JDK releases builds on OpenJDK 7 by adding just a couple of pieces, like the deployment code, which includes Oracle's implementation of the Java Plugin and Java WebStart, as well as some closed source third party components like a graphics rasterizer, some open source third party components, like Rhino, and a few bits and pieces here and there, like additional documentation or third party fonts. Moving forward, our intent is to open source all pieces of the Oracle JDK except those that we consider commercial features such as JRockit Mission Control (not yet available in Oracle JDK), and replace encumbered third party components with open source alternatives to achieve closer parity between the code bases.
Raw Data
Common Settings of the Tests
Coefficient of variation (CV) is used for the comparison of experiment's variability. Based on 5 sample points of each test, we have come to the following preliminary conclusions:
- SPECjbb
- OOTB settings
- -m composite
- JDK 7
- 4GB heap
- Parallel GC (default)
- OEL 6.6
- Data sampling
- Total Runs: 5 for each test
- Metrics
- Sample Standard Deviation (Std Dev)
- Coefficient of variation (CV)
- Test 1
- OpenJDK
- Huge Pages:[7] 0
- Test 2
- OpenJDK
- Huge Pages: 2200
- Test 3
- HotSpot
- Huge Pages: 0
- Test 4
- HotSpot
- Huge Pages: 2200
| SPECjbb 2013 | Test 1 | Test 2 | Test 3 | Test 4 | ||||||||
| Mean | Std Dev | CV | Mean | Std Dev | CV | Mean | Std Dev | CV | Mean | Std Dev | CV | |
| max-jOPS | 2973.40 | 76.01 | 2.56% | 3490.60 | 92.54 | 2.65% | 3124.20 | 43.22 | 1.38% | 3538 | 71.98 | 2.03% |
| critical-jOPS | 748.40 | 70.02 | 9.36% | 852.20 | 76.37 | 8.96% | 741.60 | 36.28 | 4.89% | 828.8 | 76.54 | 9.24% |
| SPECjbb 2015 | Test 1 | Test 2 | Test 3 | Test 4 | ||||||||
| Mean | Std Dev | CV | Mean | Std Dev | CV | Mean | Std Dev | CV | Mean | Std Dev | CV | |
| max-jOPS | 2853.80 | 55.22 | 1.93% | 3233.40 | 82.32 | 2.55% | 2893.20 | 53.89 | 1.86% | 3238.60 | 92.89 | 2.87% |
| critical-jOPS | 514 | 20.21 | 3.93% | 581.80 | 45.26 | 7.78% | 522.40 | 18.90 | 3.62% | 571.6 | 36.67 | 6.42% |
Analysis
Coefficient of variation (CV) is used for the comparison of experiment's variability. Based on 5 sample points of each test, we have come to the following preliminary conclusions:
- SPECjbb 2013 vs SPECjbb 2015
- On critical-jOPS, SPECjbb 2015 exhibits lower extent of variability in all tests.
- On max-jOPS, SPECjbb 2015 exhibits lower extent of variability in all tests except Test 3&4.
- Enabling Huge Pages or not
- In both SPECjbb 2013 & 2015, enabling huge pages exhibits higher extent of variability (exception: SPECjbb 2013/OpenJDK/critical-jOPS)
- OpenJDK vs HotSpot
- Both JVMs exhibit similar performance in average (i.e., max- or critical- jOPS) .
By no means our results are conclusive. As more tests are run in the future, we could gain a better picture. From the preliminary results, it seems to be wise of choosing SPECjbb 2015 over SPECjbb 2013 in future Java SE performance tests.
References
- Coefficient of variation
- When to use the sample or population standard deviation
- SPECjbb2015_release_notes.txt
- SPECjbb2013 Result File Fields
- Java 7 Questions & Answers
- java - Difference between JVM and HotSpot?
- Huge Pages
- Java Mission Control
- Java Mission Control (JMC) is for Hotspot as JRockit Mission Control (JRMC) is for JRockit.
- Standard Deviation Calculator
No comments:
Post a Comment