Saturday, October 24, 2015

Does SPECjbb 2015 Have Less Variability than SPECjbb2013?

What's the advantage of using SPECjbb 2015 over SPECjbb 2013? The answer may be "less variability". In this article, I'm going to examine the credibility of this claim.

Note that the experiments run here are not intended to evaluate the ultimate performance of JVMs (OpenJDK vs. HotSpot) with different configurations. Different JDK 7 versions were used in OpenJDK and HotSpot here. But, the author believes that the differences between builds of OpenJDK and HotSpot should be minor.

To recap, the main purpose of this article is comparing the variability of SPECjbb 2013 and SPECjbb 2015.


OpenJDK vs. HotSpot



To begin with , we want to emphasize that OpenJDK and HotSpot actually share most of the code bases. Henrik Stahl, VP of Product Management in the Java Platform Group at Oracle, has shed some light on it:
Q: What is the difference between the source code found in the OpenJDK repository, and the code you use to build the Oracle JDK?
A: It is very close - our build process for Oracle JDK releases builds on OpenJDK 7 by adding just a couple of pieces, like the deployment code, which includes Oracle's implementation of the Java Plugin and Java WebStart, as well as some closed source third party components like a graphics rasterizer, some open source third party components, like Rhino, and a few bits and pieces here and there, like additional documentation or third party fonts. Moving forward, our intent is to open source all pieces of the Oracle JDK except those that we consider commercial features such as JRockit Mission Control (not yet available in Oracle JDK), and replace encumbered third party components with open source alternatives to achieve closer parity between the code bases.

Raw Data



Common Settings of the Tests
Different Settings of Tests
  • Test 1
    • OpenJDK 
    • Huge Pages:[7] 0
  • Test 2
    • OpenJDK 
    • Huge Pages: 2200
  • Test 3
    • HotSpot
    • Huge Pages: 0
  • Test 4
    • HotSpot
    • Huge Pages: 2200

SPECjbb
2013
Test 1
Test 2
Test 3
Test 4

MeanStd DevCVMeanStd DevCVMeanStd DevCVMean   Std DevCV
  max-jOPS2973.4076.012.56%3490.6092.542.65%3124.2043.221.38%353871.982.03%
  critical-jOPS748.4070.029.36%852.2076.378.96%741.6036.284.89%828.876.549.24%



SPECjbb
2015
Test 1
Test 2
Test 3
Test 4

MeanStd DevCVMeanStd DevCVMeanStd DevCVMeanStd DevCV
  max-jOPS2853.8055.221.93%3233.4082.322.55%2893.2053.891.86%3238.6092.892.87%
  critical-jOPS51420.213.93%581.8045.267.78% 522.40 18.903.62%571.636.676.42%

Analysis


Coefficient of variation (CV) is used for the comparison of experiment's variability.  Based on 5 sample points of each test, we have come to the following preliminary conclusions:
  • SPECjbb 2013 vs SPECjbb 2015
    • On critical-jOPS, SPECjbb 2015 exhibits lower extent of variability in all tests.
    • On max-jOPS, SPECjbb 2015 exhibits lower extent of variability in all tests except Test 3&4.
  • Enabling Huge Pages or not
    • In both SPECjbb 2013 & 2015, enabling huge pages exhibits higher extent of variability (exception: SPECjbb 2013/OpenJDK/critical-jOPS)
  • OpenJDK vs HotSpot
    • Both JVMs exhibit similar performance in average (i.e., max- or critical- jOPS) .
By no means our results are conclusive.  As more tests are run in the future, we could gain a better picture.  From the preliminary results, it seems to be wise of choosing SPECjbb 2015 over SPECjbb 2013 in future Java SE performance tests.

1 comment:

Blogger said...

If you need your ex-girlfriend or ex-boyfriend to come crawling back to you on their knees (no matter why you broke up) you got to watch this video
right away...

(VIDEO) Want your ex CRAWLING back to you...?