Oracle Fusion Middleware Diagnostic Framework―How to Diagnosing Problems

Oracle Fusion Middleware includes a Diagnostic Framework, which aids in detecting, diagnosing, and resolving problems. The problems that are targeted in particular are critical errors, such as those caused by

• Code bugs
• Metadata corruption
• Customer data corruption
• Deadlocked threads
• Inconsistent state

In this article, we will cover what Oracle Fusion Middleware Diagnostic Framework is and how it works.

Problem vs Incident

• Problem 
• Is a critical error
• Critical errors manifest as internal errors or other severe errors. Problems are tracked in the Automatic Diagnostic Repository (ADR).
• Has a problem key
• Is a text string that describes the problem
• Includes an error code (in the format XXX-nnnnn) and in some cases, other error-specific values.
• incident 1123 created with problem key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.executeBaseQuery][bi-contentstorage]"
• Incident
• Is a single occurrence of a problem
• When a problem (critical error) occurs multiple times, an incident is created for each occurrence. Incidents are timestamped and tracked in the ADR
• Is identified by a numeric incident ID (see 1123 above), which is unique within the ADR home

Oracle Fusion Middleware Diagnostic Framework 

When a critical error occurs, it is assigned an incident number, and diagnostic data for the error (such as log files) are immediately captured and tagged with this number. The data is then stored in the Automatic Diagnostic Repository (ADR), where it can later be retrieved by incident number and analyzed.  Here is the summary of its features:

• Supports incident detection log filter
• Implements the java.util.logging filter
• Inspects each log message to see if an incident should be created, basing its decision on the diagnostic rules for components and applications.
• Integrated with WebLogic Diagnostics Framework (WLDF)
• All diagnostic data relating to a critical error is captured and stored as an incident in Automatic Diagnostic Repository (ADR)
• Collects diagnostic data, such as
• Thread dumps, DMS metric dumps, and WebLogic Diagnostics Framework (WLDF) server image dumps 
• Provides standardized log formats
• Using the ODL log file format across all Oracle Fusion Middleware components.
• Incident flood control
• Diagnostic Framework applies flood control to incident generation after certain thresholds are reached
• To avoid generating too much diagnostic data, which would consume too much space in the ADR and could possibly slow down your efforts to diagnose and resolve the problem
• Example:
[2019-02-08T23:59:50.082+00:00] [bi_server2] [WARNING] [DFW-40125] [oracle.dfw.incident] [tid: [ACTIVE].ExecuteThread: '62' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 551d9654-1bc1-4b2f-b8d4-cbd3ab71603c-0004765a,0] [partition-name: DOMAIN] [tenant-name: GLOBAL] incident flood controlled with Problem Key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.executeBaseQuery][bi-contentstorage]"

Integration with WLDF

Oracle Fusion Middleware Diagnostics Framework integrates with the following components of WLDF:

• WLDF Watch and Notification
• Watches specific logs and metrics for specified conditions and sends a notification when a condition is met. 
• Oracle Fusion Middleware Diagnostics Framework integrates with the WLDF Watch and Notification component to create incidents.
• There are several types of notifications, including JMX notification and a notification to create a Diagnostic Image. 
• Diagnostic Image Capture
• Gathers the most common sources of the key server state used in diagnosing problems. 
• Packages that state into a single artifact, the Diagnostic Image. 
• With Oracle Fusion Middleware Diagnostics Framework, it writes the artifact to ADR.

Figure 1 shows the interaction when the incident is detected by the incident log detector. It shows the interaction among the incident log detector, the WLDF Diagnostic Image MBean, ADR, and component or application dumps when an incident is detected by the incident log detector.

Figure 1.  Incident Creation Generated by Incident Log Detector

Sample WebLogic Server Log

<Feb 8, 2019 11:59:54,143 PM UTC> <Notice> <Diagnostics> <xxxxxxx020308oacpod-bi-2.svcsbnet308.yyyyyyy2.oraclevcn.com> <bi_server2> <[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <eed7eff4-508d-4c1d-9c2b-b19d8e8936a6-0007d1d0> <1549670394143> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320068> <Watch "UncheckedException" in module "Module-FMWDFW" with severity "Notice" on server "bi_server2" has triggered at Feb 8, 2019 11:59:54 PM UTC. Notification details:
WatchRuleType: Log
WatchRule: (log.severityString == 'Error') and ((log.messageId == 'WL-101020') or (log.messageId == 'WL-101017') or (log.messageId == 'WL-000802') or (log.messageId == 'BEA-101020') or (log.messageId == 'BEA-101017') or (log.messageId == 'BEA-000802'))
WatchData: MESSAGE = [ServletContext@879994790[app:bi-servicelcm-rest module:bi-servicelcm-rest path:null spec-version:3.1]] Root cause of ServletException.
oracle.bi.servicelcm_v2.exceptions.PersistenceBackendException: Unable to create Pod record
at oracle.bi.servicelcm_v2.db.DatabasePodPersistenceManager.getPodImpl(DatabasePodPersistenceManager.java:39) 

... 

<Feb 8, 2019 11:59:58,489 PM UTC> <Emergency> <oracle.dfw.incident> <xxxxxxx020308oacpod-bi-2.svcsbnet308.yyyyyyy2.oraclevcn.com> <bi_server2> <[ACTIVE] ExecuteThread: '68' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <eed7eff4-508d-4c1d-9c2b-b19d8e8936a6-0007d1d1> <1549670398489> <[severity-value: 1] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <incident 1326 created with problem key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.bi.servicelcm_v2.db.SqlHelper.doTransaction][bi-servicelcm-rest]">

DFW-99998 is one of the "Uncaught Exception Problem Keys" and its specific format is:

• DFW-99998 [exception-name][package.class.name][app-name]

For example, the [app-name] in the above example is bi-servicelcm-rest.

For more information about WLDF, see Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

References

1. Diagnosing Problems (Oracle Fusion Middleware Diagnostic Framework)
2. Fusion Middleware Error Messages Reference
3. Understanding WebLogic Incident and the Diagnostic Framework behind It (XML and More)

The Configuration File in WebLogic Server Domain — config.xml

A WebLogic domain^[1] is the basic administrative unit of WebLogic Server. It consists of one or more WebLogic Server instances, and logically related resources and services that are managed collectively as one unit. A WebLogic Server instance can be either an Admin Server or a Managed Server.  Managed Server instances can be grouped into a cluster which work together to provide scalability and high availability for applications.  Each WebLogic domain has one and only one Admin Server. When the Admin Server is used to perform a configuration task, the changes apply only to the domain managed by that Admin Server.

Each domain's configuration is stored in a separate configuration file called config.xml. The config.xml file is a persistent store for the managed objects that WebLogic Server creates and modifies during its executing using the JMX API. The config.xml file specifies the name of the domain and the configuration parameter settings for each server instance, cluster, resource, and service in the domain.

Since each config.xml is assoicated with a specific domain, it is required to be stored in the Root Directory of the Admin Server. For example, you can find it in the following directory:

• .../user_projects/domains/<domain name>/config

in the standalone WLS installation or

• .../system11.1.1.6.38.62.38/DefaultDomain/config

in the Integrated WLS installation^[2]. Note that the root directory of Admin Server can be configured with the -Dweblogic.RootDirectory=path option in the server's startup command.

config.xml

When the Admin Server starts, it loads the config.xml for the domain. When a Managed Server in the same domain starts up, it connects to the domain's Admin Server to obtain configuration and deployment settings.

You should normally use the WLS Admin Console to configure WebLogic Server's manageable objects and services and allow WebLogic Server to maintain the config.xml file. You would only directly update under unusual circumstances. In this article, we will introduce you one such legitimate usage:

• For performance tests, we need to conduct performance analysis with different WLDF diagnostic volume settings:
• Off — No diagnostic data is automatically produced.
• Low — A minimal volume of diagnostic data is automatically produced. This is the default.
• Medium — Additional diagnostic data is automatically generated beyond the amount that is generated for Low.
• High — Additional diagnostic data is automatically generated beyond the amount that is generated for Medium.

WebLogic Diagnostic Framework (WLDF)

The WebLogic Diagnostic Framework (WLDF)^[4] provides features for generating, gathering, analyzing, and persisting diagnostic data from WebLogic Server instances and from applications deployed to server instances.

You can use Admin Console to configure the WLDF diagnostic volume:

1. In the left pane, select Environment > Servers.
2. In the Servers table, click the name of the server instance for which you want to configure the WLDF diagnostic volume.
3. Select Configuration > General.
4. On the Servers: Configuration: General page, select one of the following values in Diagnostic Volume:
   • Off
   • Low
   • Medium
   • High
5. Click Save.

At runtime, WLS maintains three copies of config.xml:

1. ./servers/domain_bak/config_prev/config.xml
2. ./pending/config.xml
3. ./config/config.xml

Each time the Admin Server starts successfully, and each time the configuration is modified, a backup configuration file is created (i.e., the 1st copy). The number of backup copies of config.xml retained by the Admin Server can be configured.

After the Save action, the changes will be propagated to the pending copy (i.e., the 2nd copy). But, not until you activate the changes, will the changes be propagated to the true final copy (i.e., the 3rd copy).

Preparation of Our Performance Tests

Before making any changes to the config.xml file, you should make a copy of it first. To prepare for our different test scenarios, we launch the WLS Admin Console and set WLDF diagnostic volume to different settings. After activating the changes, we make a copy of config.xml and name it accordingly (e.g., config.xml.high, config.xml.off, etc.).

After we have different copies of config.xml with different volume settings, we then run our automation script in such a way:

1. At the beginning of each test, we copy config.xml with appropriate setting (i.e., config.xml.high) to be the config.xml
2. We then launch WLS with the new config.xml for our performance test
• Note that Admin Server needs to  be restarted to pick up new changes.

References

1. WebLogic Server Domains (WebLogic Server 12c)
2. Integrated WebLogic Server (WLS) (Xml and More)
3. Configuring Fusion Middleware Domains (WebLogic Server 12c)
4. Understanding WLDF Configuration
5. Mining WebLogic Diagnostic Data with XSLT
6. WebLogic: How to Create a WLS Domain? (Xml and More)
7. Diagnosing problems (Fusion Middleware Administrator's Guide, 11g Release 1)
• Note that WLDF does not prevent or stop issues from happening.  It only monitors and captures data for defined conditions and the captured info can be used for troubleshooting.
8. WLDF overhead
• The beauty of the WLDF system is twofold:
• The system gathers data at the lowest level, so impact to gathering that data is minimized
• You can collect only the data that you want by defining data harvesters 
9. Configuring WLDF data storage
10. Oracle® Fusion Middleware Configuring and Using the Diagnostics Framework for Oracle WebLogic Server
11. WLDF: Accessing Diagnostic Data With the Data Accessor