Saturday, October 24, 2015

Does SPECjbb 2015 Have Less Variability than SPECjbb2013?

What's the advantage of using SPECjbb 2015 over SPECjbb 2013? The answer may be "less variability". In this article, I'm going to examine the credibility of this claim.

Note that the experiments run here are not intended to evaluate the ultimate performance of JVMs (OpenJDK vs. HotSpot) with different configurations. Different JDK 7 versions were used in OpenJDK and HotSpot here. But, the author believes that the differences between builds of OpenJDK and HotSpot should be minor.

To recap, the main purpose of this article is comparing the variability of SPECjbb 2013 and SPECjbb 2015.


OpenJDK vs. HotSpot



To begin with , we want to emphasize that OpenJDK and HotSpot actually share most of the code bases. Henrik Stahl, VP of Product Management in the Java Platform Group at Oracle, has shed some light on it:
Q: What is the difference between the source code found in the OpenJDK repository, and the code you use to build the Oracle JDK?
A: It is very close - our build process for Oracle JDK releases builds on OpenJDK 7 by adding just a couple of pieces, like the deployment code, which includes Oracle's implementation of the Java Plugin and Java WebStart, as well as some closed source third party components like a graphics rasterizer, some open source third party components, like Rhino, and a few bits and pieces here and there, like additional documentation or third party fonts. Moving forward, our intent is to open source all pieces of the Oracle JDK except those that we consider commercial features such as JRockit Mission Control (not yet available in Oracle JDK), and replace encumbered third party components with open source alternatives to achieve closer parity between the code bases.

Raw Data



Common Settings of the Tests
Different Settings of Tests
  • Test 1
    • OpenJDK 
    • Huge Pages:[7] 0
  • Test 2
    • OpenJDK 
    • Huge Pages: 2200
  • Test 3
    • HotSpot
    • Huge Pages: 0
  • Test 4
    • HotSpot
    • Huge Pages: 2200

SPECjbb
2013
Test 1
Test 2
Test 3
Test 4

MeanStd DevCVMeanStd DevCVMeanStd DevCVMean   Std DevCV
  max-jOPS2973.4076.012.56%3490.6092.542.65%3124.2043.221.38%353871.982.03%
  critical-jOPS748.4070.029.36%852.2076.378.96%741.6036.284.89%828.876.549.24%



SPECjbb
2015
Test 1
Test 2
Test 3
Test 4

MeanStd DevCVMeanStd DevCVMeanStd DevCVMeanStd DevCV
  max-jOPS2853.8055.221.93%3233.4082.322.55%2893.2053.891.86%3238.6092.892.87%
  critical-jOPS51420.213.93%581.8045.267.78% 522.40 18.903.62%571.636.676.42%

Analysis


Coefficient of variation (CV) is used for the comparison of experiment's variability.  Based on 5 sample points of each test, we have come to the following preliminary conclusions:
  • SPECjbb 2013 vs SPECjbb 2015
    • On critical-jOPS, SPECjbb 2015 exhibits lower extent of variability in all tests.
    • On max-jOPS, SPECjbb 2015 exhibits lower extent of variability in all tests except Test 3&4.
  • Enabling Huge Pages or not
    • In both SPECjbb 2013 & 2015, enabling huge pages exhibits higher extent of variability (exception: SPECjbb 2013/OpenJDK/critical-jOPS)
  • OpenJDK vs HotSpot
    • Both JVMs exhibit similar performance in average (i.e., max- or critical- jOPS) .
By no means our results are conclusive.  As more tests are run in the future, we could gain a better picture.  From the preliminary results, it seems to be wise of choosing SPECjbb 2015 over SPECjbb 2013 in future Java SE performance tests.

Sunday, October 18, 2015

SPECjbb 2013: Understanding the Basics


SPEC Benchmark Suites aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation (SPECjbb) to a full system with Java EE, database, disk, and network (SPECjEnterprise).

In this article, we will introduce the basics of SPECjbb 2013 (an older version).  But, descriptions can be equally applied to SPECjbb 2015 with some changes.

Components


SPECjbb 2013 includes three main components (see also: Driver System and Reporter[4]):
  • Controller (or Ctr)
    • Function
      • Directs the execution of the workload
    • Configution:
      • Only one Ctr per System Under Test (SUT)
        • Could be located outside of the SUT
      • Heap size of 2GB is suggested 
      • Run category[3] depends on type of Ctr invoked, which set by -m option:
        • -m [composite / multicontroller / distcontroller]
          • SPECjbb2013-Composite 
          • SPECjbb2013-MultiJVM 
          • SPECjbb2013-Distributed
    • Debugging:
      • If OOM is experienced, just re-run the reporter with larger heap using the run_reporter.* scripts. 
      • Capture the Ctr standard output can be helpful in debugging many issues.
  • Transaction Injector(s) (or TxI)
    • Function
      • Issues requests and services to Backend(s) as directed by the Controller and measures end-to-end response time from issued requests
    • Configuration:
      • One TxI cannot go across two BE
      • Heap size of 2GB is suggested.
  • Backend(s) (or BE)
    • Function:
      • Contains business logic code that processes requests and services from TxI, and notifies the TxI that a request has been processed.
    • Configuration:
      • One or more TxI per BE
      • Heap Size: a minimum heap setting of 2GB is needed. 
        • Larger heaps may be needed for generational garbage collectors.

Configuration & Deployments


There are many options on configuring and deploying SPECjbb components:
  • Single vs. Multiple JVM's
    • Benchmark components Ctr, TxI(s) and BE(s) can be configured to run
      • Inside a single JVM instance 
      • Across multiple JVM instances
        • With each component using its own JVM instance deployed to run across single or multiple hosts.
  • Group
    • Is a logical mapping of BE to TxI(s) 
    • Configuration:
      • Can have only one BE, but can have one or more TxI(s) if one TxI is not able to fully load the BE
      • The topology for a Group can be defined using command line options in the run script. 
      • The benchmark can be run in single or multiple Group(s) configurations.
        • If there are multiple BEs, some % of requests and txns require interaction with other BEs initiating Inter-Java process communication among BEs.
  • Configuration of Properties
    • SPECjbb2013-Composite and SPECjbb2013-MultiJVM are run from same directory using same property file. 
    • SPECjbb2013-Distributed runs across OS images and will require these properties to be set for every launching benchmark component either from property file or command line.
  • Driver System(s)
    • In addition to SUT, SPECjbb2013-Distributed requires a driver system(s). 
      • Ctr and TxI(s) must run on a driver machine(s) using a non-virtualized OS. 
      • The SUT only runs BE(s), which can be virtualized environments in this category. 
      • The SUT can consist of one or more BEs running on one or more Nodes.

References


  1. Rules and Reporting Rules (SPECjbb 2013)
  2. User Guide (SPECjbb 2013)
  3. Run categories of SPECjbb 2013:
    • Based on most common trends in Java deployments, SPECjbb2013 components Controller, TxI(s) and Backend(s), configurations have been mapped into three different run-categories: SPECjbb2013-Composite, SPECjbb2013-Multi-JVM, SPECjbb2013-Distributed.
  4. Reporter
    • At the end of benchmark run, Controller invokes the reporter unless command line option “-skipReport” was used.
      • If needed, the reporter can be manually invoked after the benchmark run.
      • Reporter will pick correct template file based on selected run category and report the final run results.