SPECjbb 2013: Understanding the Basics

SPEC Benchmark Suites aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation (SPECjbb) to a full system with Java EE, database, disk, and network (SPECjEnterprise).

In this article, we will introduce the basics of SPECjbb 2013 (an older version).  But, descriptions can be equally applied to SPECjbb 2015 with some changes.

Components

SPECjbb 2013 includes three main components (see also: Driver System and Reporter^[4]):

• Controller (or Ctr)
• Function: 
• Directs the execution of the workload
• Configution:
• Only one Ctr per System Under Test (SUT)
• Could be located outside of the SUT
• Heap size of 2GB is suggested 
• Run category^[3] depends on type of Ctr invoked, which set by -m option:
• -m [composite / multicontroller / distcontroller]
• SPECjbb2013-Composite 
• SPECjbb2013-MultiJVM 
• SPECjbb2013-Distributed
• Debugging:
• If OOM is experienced, just re-run the reporter with larger heap using the run_reporter.* scripts. 
• Capture the Ctr standard output can be helpful in debugging many issues.
• Transaction Injector(s) (or TxI)
• Function: 
• Issues requests and services to Backend(s) as directed by the Controller and measures end-to-end response time from issued requests
• Configuration:
• One TxI cannot go across two BEs 
• Heap size of 2GB is suggested.
• Backend(s) (or BE)
• Function:
• Contains business logic code that processes requests and services from TxI, and notifies the TxI that a request has been processed.
• Configuration:
• One or more TxI per BE
• Heap Size: a minimum heap setting of 2GB is needed. 
• Larger heaps may be needed for generational garbage collectors.

Configuration & Deployments

There are many options on configuring and deploying SPECjbb components:

• Single vs. Multiple JVM's
• Benchmark components Ctr, TxI(s) and BE(s) can be configured to run
• Inside a single JVM instance 
• Across multiple JVM instances
• With each component using its own JVM instance deployed to run across single or multiple hosts.
• Group
• Is a logical mapping of BE to TxI(s) 
• Configuration:
• Can have only one BE, but can have one or more TxI(s) if one TxI is not able to fully load the BE. 
• The topology for a Group can be defined using command line options in the run script. 
• The benchmark can be run in single or multiple Group(s) configurations.
• If there are multiple BEs, some % of requests and txns require interaction with other BEs initiating Inter-Java process communication among BEs.
• Configuration of Properties
• SPECjbb2013-Composite and SPECjbb2013-MultiJVM are run from same directory using same property file. 
• SPECjbb2013-Distributed runs across OS images and will require these properties to be set for every launching benchmark component either from property file or command line.
• Driver System(s)
• In addition to SUT, SPECjbb2013-Distributed requires a driver system(s). 
• Ctr and TxI(s) must run on a driver machine(s) using a non-virtualized OS. 
• The SUT only runs BE(s), which can be virtualized environments in this category. 
• The SUT can consist of one or more BEs running on one or more Nodes.

References

1. Rules and Reporting Rules (SPECjbb 2013)
2. User Guide (SPECjbb 2013)
3. Run categories of SPECjbb 2013:
• Based on most common trends in Java deployments, SPECjbb2013 components Controller, TxI(s) and Backend(s), configurations have been mapped into three different run-categories: SPECjbb2013-Composite, SPECjbb2013-Multi-JVM, SPECjbb2013-Distributed.
4. Reporter
• At the end of benchmark run, Controller invokes the reporter unless command line option “-skipReport” was used.
• If needed, the reporter can be manually invoked after the benchmark run.
• Reporter will pick correct template file based on selected run category and report the final run results.