Thursday, November 27, 2014

WebLogic: Important Deployment Concepts

WebLogic Server implements the Java EE 5 specification. Java EE 5 includes a deployment specification, JSR-88,[1] that describes a standard API used by deployment tools and application server providers to configure and deploy applications to an application server.

Although the Java EE 5 deployment API provides a simple, standardized way to configure applications and modules for use with a Java EE 5-compliant application server, the specification does not address many deployment features that were available in previous WebLogic Server releases. For this reason, WebLogic Server provides important extensions to the Java EE 5 deployment API specification to support capabilities described in [2].

In this article, we will discuss some important concepts for understanding the deployment process in WebLogic Server.

Deployment Tools

The term application deployment refers to the process of making an application or module available for processing client requests in a WebLogic Server domain.  You have several options to deploy a web application onto WebLogic Server:
  • weblogic.Deployer
    • A packaged deployment tool which provides deployment services for WebLogic Server. 
    • Any deployment operation that can be implemented using the WebLogic Deployment API[19] is implemented, either in part or in full, by weblogic.Deployer.
    • weblogic.Deployer is the recommended deployment tool for the WebLogic Server environment. 
  • WLDeploy
  • WLST[4,5]
    • WLST functionality includes the capabilities of the following WebLogic Server command-line utilities:
      • weblogic.Deployer
      • WLDeploy
  • JSR-88 API for deployment
    • WebLogic Deployment API,[19] which implements and extends the Java EE Deployment API specification (JSR-88)
    • The WebLogic Deployment API[19] classes and interfaces extend and implement the Java EE Deployment API specification (JSR-88) interfaces, which are described in the javax.enterprise.deploy sub-packages.[6]
  • JMX Deployment API[7]
    • Supports all of the common functionality available in the Java EE Deployment API specification (JSR-88). 
    • You can use the JMX API as an alternative to JSR-88 to perform deployment tasks on specified target servers
  • WLS console (through JSR-88 API)[8]
  • FMWC (through JSR-88 API and JMX Deployment API)[9]

Deployment Concepts

To help you understand the deployment process in WebLogic Server, we have listed the most important concepts below:
  • Deployable objects (or artifacts)
    • Which can be deployed on a weblogic domain
      • EJBs
      • Modules
        • Supported module types include:[11,12]
          • JMS, JDBC, Interception, Config, and WLDF[10]
      • Web Services
      • Applications[13]
    • Physical package types:
      •  EARs, standalone J2EE and non-J2EE modules
  • Deployment operations (or tasks)
    • Such as distributing, starting, stopping, deploying, redeploying, undeploying, etc.
      • deploying
        • if successful, an AppDeploymentMBean[14] for the application will be created 
  • Deployment targets
    • Target types available to WebLogic Server include:
      • Cluster
      • JMS Server
      • SAF Agent
      • Server
      • Virtual host 
  • Deployment order
    • An integer value that indicates when this artifact is deployed, relative to other deployable artifacts on a server, during startup.
    • Artifacts with lower values are deployed before those with higher values.
  • Deployment plan
    • You can use either Weblogic Console or weblogic.PlanGenerator to export portions of a WebLogic Server deployment configuration into a deployment plan file.[16,17]
    • Describes the application structure, identifies all deployment descriptors, and exports a subset of the application's configurable properties.  For example, it includes:
      • Resource dependencies
      • Resource declarations
      • Non-resource oriented configurable properties
      • Properties that may be changed in a running application, etc
    • A deployment plan is an XML document used to define an application's deployment configuration for a specific WebLogic Server environment, such as development, test, or production. 
    • A deployment plan resides outside of an application's archive file and contains deployment properties that override an application's existing Java Enterprise Edition and WebLogic Server deployment descriptors. 
    • Use deployment plans to easily change an application's WebLogic Server configuration for a specific environment, without modifying existing deployment descriptors.
  • Deployment configuration
    • Refers to the process of preparing an application or deployable resource for deployment to a WebLogic Server instance.
    • Modification of individual WebLogic Server configuration values based on user inputs and the selected WebLogic Server targets
  • Deployment staging
    • The staging mode of an application or deployment plan can be provided to affect its deployment behavior.
      • An application's deployment plan can be staged independently of the application archive, allowing you to stage a deployment plan when the application is not staged.
      • If you do not specify a staging mode, the deployment plan uses the value specified for application staging as the default.
    • Staging mode includes:
      • STAGE—Application files (or deployment plans) are copied to target servers
      • NO_STAGE—Application files (or deployment plans) are not copied to target servers
      • EXTERNAL_STAGE—Application files (or deployment plans) are copied manually to target servers

"Install Root" Abstraction

The SessionHelper[18] in WebLogic Deployment API[19] views an application and deployment plan artifacts using an "install root" abstraction, which ideally is the actual organization of the application. The install root appears as follows:

install-root (eg myapp) 
-- app 
----- archive (eg myapp.ear) 
-- plan
----- deployment plan (eg plan.xml) 
----- external descriptors (eg META-INF/weblogic-application.xml...) 

Optionally, you may consider using this layout to organize your applications and deployment plans.


  1. JSR 88: JavaTM EE Application Deployment
  2. WebLogic Server Deployment Features
  3. Understanding the WebLogic Deployment API (WebLogic 12.1.2)
  4. WLST FAQs
  5. WLST Command and Variable Reference
  6. Java EE 5 API Summary
  7. The JMX API for Deployment Operations
  8. Administration Console Online Help
  9. Using Fusion Middleware Control to Manage WebLogic Server
  10. The Configuration File in WebLogic Server Domain — config.xml (Xml and More)
  11. Module Types
  12. Example Module Deployment
  13. Oracle® Fusion Middleware Deploying Applications to Oracle WebLogic Server
  14. Interface AppDeploymentMBean
  15. MBean Reference for Oracle WebLogic Server
  16. Create a deployment plan using Administration Console
  17. Oracle WebLogic Server 12c: Creating and Using a Deployment Plan
  18. SessionHelper
  19. WebLogic Deployment API

Sunday, November 23, 2014

G1 GC: Humongous Objects and Humongous Allocations

For Garbage First Garbage Collector (G1 GC), any object that is more than half a region size is considered a Humongous Object.   A humongous object is directly allocated into Humongous Regions.[1]

In this article, we will discuss the following topics:
  • Humongous regions and humongous allocations
  • How humongous allocations impact G1's performance?
  • How to detect humongous allocations?
    • Basic Investigation
    • Advanced Investigation

Humongous Regions and Humongous Allocations

A humongous object is allocated directly in the humongous regions. These humongous regions are a contiguous set of regions. StartsHumongous marks the start of the contiguous set and ContinuesHumongous marks the continuation of the set.

Before allocating any humongous region, the marking threshold is checked, initiating a concurrent cycle, if necessary. Dead humongous objects are freed at the end of the marking cycle during the cleanup phase also during a full garbage collection cycle (but, a new implementation has changed this; see the next section).

Since each individual set of StartsHumongous and ContinuesHumongous regions contains just one humongous object, the space between the end of the humongous object and the end of the last region spanned by the object is unused. For objects that are just slightly larger than a multiple of the heap region size, this unused space can cause the heap to become fragmented.

How Humongous Allocations Impact G1's Performance?

In the old implementation, humongous objects are not released until after a full concurrent marking cycle.[4]  This is far from ideal for many transaction-based enterprise applications that create short-lived (i.e., in the transaction scope) humongous objects such as ResultSet(s) generated from JDBC queries.  This results in the heap filling up relatively quickly and leads to unnecessary marking cycles to reclaim them.

A new implementation since:
java version "1.8.0_40-ea"
Java(TM) SE Runtime Environment (build 1.8.0_40-ea-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b05, mixed mode)
handles humongous regions differently and can reclaim them earlier if they are short-lived (see [4] for details).

For investigating humongous allocations in G1, you should begin with the following minimal set of VM options:
  • -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintReferenceGC (-XX:+UseG1GC)

Basic Humongous Allocation Investigation

For basic humongous allocations investigation, you simply add a new option:
  • -XX:+PrintAdaptiveSizePolicy
This option will print out something like below:
12025.832: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 3660578816 bytes, allocation request: 1048592 bytes, threshold: 3650722120 bytes (85.00 %), source: concurrent humongous allocation]
From the above, we saw a humongous allocation is initiated with a request size of 1048592 bytes. Note that our region size is 1MBytes in this test. So, the request object is considered as humongous (i.e., 1048592 Bytes > half of 1 MBytes).

In the output, you can also find the following entries:
      [Humongous Reclaim: 0.0 ms]
         [Humongous Total: 54]
         [Humongous Candidate: 0]
         [Humongous Reclaimed: 0]
This shows that there are 54 humongous regions found in this GC event.   Also, none of them were reclaimed in this GC event. The reason is that the current algorithm is relatively conservative and it skips humongous objects for reclamation if there are sparse references into them.  In the case that those references belongs to sparse table entries, G1 may improve the reclamation rate by discovering if they are actually not referenced at all after iterating the table.  A performance enhancement is pending for this effort now.[5]

Advanced Humongous Allocation Investigation

To further investigate humongous allocation in more details, you can add:
-XX:+PrintAdaptiveSizePolicy -XX:+UnlockExperimentalVMOptions -XX:+G1ReclaimDeadHumongousObjectsAtYoungGC -XX:G1LogLevel=finest -XX:+G1TraceReclaimDeadHumongousObjectsAtYoungGC
Note that the following VM options are experimental:
  • G1TraceReclaimDeadHumongousObjectsAtYoungGC[4]
  • G1ReclaimDeadHumongousObjectsAtYoungGC[3]
  • G1LogLevel (i.e., [fine|finer|finest])
So, that's why we added the unlock option;
With extra printing options, you will find detailed description of humongous allocations as shown below:
3708.669: [SoftReference, 0 refs, 0.0000340 secs]3708.669: [WeakReference, 1023 refs, 0.0001790 secs]3708.669: [FinalReference, 295 refs, 0.0007020 secs]3708.670: [PhantomReference, 0 refs, 0.0000060 secs]3708.670: [JNI Weak Reference, 0.0000140 secs]Live humongous 1 region 9 size 65538 with remset 1 code roots 0 is marked 0 live-other 0 obj array 0

Live humongous 1 region 10 size 88066 with remset 1 code roots 0 is marked 0 live-other 0 obj array 0
Live humongous 1 region 15 size 262146 with remset 1 code roots 0 is marked 0 live-other 0 obj array 1
Finally, you can also use Java Flight Recorder (JFR) and Java Mission Control (JMC) to get information about humongous objects by showing their allocation sources with stack traces.[7]


  1. Garbage First Garbage Collector Tuning
  2. G1GC: Migration to, Expectations and Advanced Tuning
  3. Thread local allocation buffers (TLAB's)
  4. Early reclamation of large objects in G1
    • Introduced since 8u40 b02 and 8u45
    • Was not back ported to 8u20
  5. Early reclaim of large objects that are referenced by a few objects
  6. G1TraceReclaimDeadHumongousObjectsAtYoungGC
    • JDK-8058801
      • Prints information about live and dead humongous objects
        • (Before bug fix) prints that information when there is at least one humongous candidate.
        • (After bug fix) prints that information even when there is no humongous candidates; it also prints humongous object size in the output
  7. Java Mission Control (JMC) and Java Flight Recorder (JFR)
  8. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)