Oracle Fusion Middleware Diagnostic Framework―How to Diagnosing Problems

Oracle Fusion Middleware includes a Diagnostic Framework, which aids in detecting, diagnosing, and resolving problems. The problems that are targeted in particular are critical errors, such as those caused by

• Code bugs
• Metadata corruption
• Customer data corruption
• Deadlocked threads
• Inconsistent state

In this article, we will cover what Oracle Fusion Middleware Diagnostic Framework is and how it works.

Problem vs Incident

• Problem 
• Is a critical error
• Critical errors manifest as internal errors or other severe errors. Problems are tracked in the Automatic Diagnostic Repository (ADR).
• Has a problem key
• Is a text string that describes the problem
• Includes an error code (in the format XXX-nnnnn) and in some cases, other error-specific values.
• incident 1123 created with problem key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.executeBaseQuery][bi-contentstorage]"
• Incident
• Is a single occurrence of a problem
• When a problem (critical error) occurs multiple times, an incident is created for each occurrence. Incidents are timestamped and tracked in the ADR
• Is identified by a numeric incident ID (see 1123 above), which is unique within the ADR home

Oracle Fusion Middleware Diagnostic Framework 

When a critical error occurs, it is assigned an incident number, and diagnostic data for the error (such as log files) are immediately captured and tagged with this number. The data is then stored in the Automatic Diagnostic Repository (ADR), where it can later be retrieved by incident number and analyzed.  Here is the summary of its features:

• Supports incident detection log filter
• Implements the java.util.logging filter
• Inspects each log message to see if an incident should be created, basing its decision on the diagnostic rules for components and applications.
• Integrated with WebLogic Diagnostics Framework (WLDF)
• All diagnostic data relating to a critical error is captured and stored as an incident in Automatic Diagnostic Repository (ADR)
• Collects diagnostic data, such as
• Thread dumps, DMS metric dumps, and WebLogic Diagnostics Framework (WLDF) server image dumps 
• Provides standardized log formats
• Using the ODL log file format across all Oracle Fusion Middleware components.
• Incident flood control
• Diagnostic Framework applies flood control to incident generation after certain thresholds are reached
• To avoid generating too much diagnostic data, which would consume too much space in the ADR and could possibly slow down your efforts to diagnose and resolve the problem
• Example:
[2019-02-08T23:59:50.082+00:00] [bi_server2] [WARNING] [DFW-40125] [oracle.dfw.incident] [tid: [ACTIVE].ExecuteThread: '62' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 551d9654-1bc1-4b2f-b8d4-cbd3ab71603c-0004765a,0] [partition-name: DOMAIN] [tenant-name: GLOBAL] incident flood controlled with Problem Key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.executeBaseQuery][bi-contentstorage]"

Integration with WLDF

Oracle Fusion Middleware Diagnostics Framework integrates with the following components of WLDF:

• WLDF Watch and Notification
• Watches specific logs and metrics for specified conditions and sends a notification when a condition is met. 
• Oracle Fusion Middleware Diagnostics Framework integrates with the WLDF Watch and Notification component to create incidents.
• There are several types of notifications, including JMX notification and a notification to create a Diagnostic Image. 
• Diagnostic Image Capture
• Gathers the most common sources of the key server state used in diagnosing problems. 
• Packages that state into a single artifact, the Diagnostic Image. 
• With Oracle Fusion Middleware Diagnostics Framework, it writes the artifact to ADR.

Figure 1 shows the interaction when the incident is detected by the incident log detector. It shows the interaction among the incident log detector, the WLDF Diagnostic Image MBean, ADR, and component or application dumps when an incident is detected by the incident log detector.

Figure 1.  Incident Creation Generated by Incident Log Detector

Sample WebLogic Server Log

<Feb 8, 2019 11:59:54,143 PM UTC> <Notice> <Diagnostics> <xxxxxxx020308oacpod-bi-2.svcsbnet308.yyyyyyy2.oraclevcn.com> <bi_server2> <[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <eed7eff4-508d-4c1d-9c2b-b19d8e8936a6-0007d1d0> <1549670394143> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320068> <Watch "UncheckedException" in module "Module-FMWDFW" with severity "Notice" on server "bi_server2" has triggered at Feb 8, 2019 11:59:54 PM UTC. Notification details:
WatchRuleType: Log
WatchRule: (log.severityString == 'Error') and ((log.messageId == 'WL-101020') or (log.messageId == 'WL-101017') or (log.messageId == 'WL-000802') or (log.messageId == 'BEA-101020') or (log.messageId == 'BEA-101017') or (log.messageId == 'BEA-000802'))
WatchData: MESSAGE = [ServletContext@879994790[app:bi-servicelcm-rest module:bi-servicelcm-rest path:null spec-version:3.1]] Root cause of ServletException.
oracle.bi.servicelcm_v2.exceptions.PersistenceBackendException: Unable to create Pod record
at oracle.bi.servicelcm_v2.db.DatabasePodPersistenceManager.getPodImpl(DatabasePodPersistenceManager.java:39) 

... 

<Feb 8, 2019 11:59:58,489 PM UTC> <Emergency> <oracle.dfw.incident> <xxxxxxx020308oacpod-bi-2.svcsbnet308.yyyyyyy2.oraclevcn.com> <bi_server2> <[ACTIVE] ExecuteThread: '68' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <eed7eff4-508d-4c1d-9c2b-b19d8e8936a6-0007d1d1> <1549670398489> <[severity-value: 1] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <incident 1326 created with problem key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.bi.servicelcm_v2.db.SqlHelper.doTransaction][bi-servicelcm-rest]">

DFW-99998 is one of the "Uncaught Exception Problem Keys" and its specific format is:

• DFW-99998 [exception-name][package.class.name][app-name]

For example, the [app-name] in the above example is bi-servicelcm-rest.

For more information about WLDF, see Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

References

1. Diagnosing Problems (Oracle Fusion Middleware Diagnostic Framework)
2. Fusion Middleware Error Messages Reference
3. Understanding WebLogic Incident and the Diagnostic Framework behind It (XML and More)

Understanding WebLogic Incident and the Diagnostic Framework behind It

From the WebLogic Server log, I have found the following incident:

[INCIDENT_ERROR] [_createIncident] An incident was created. The details are: 

Incident details are: IncidentID=359
ProductID=fusionapps
InstanceID=ApplicationCoreCRMDemoUI
ProductType=ofm
DomainName=atgdomain
ServerName=CRMDemo_server1
. (FND-10102). (FND-10000)
<Dec 8, 2012 9:22:57 PM PST> <Alert> <Diagnostics> 

<BEA-320016> <Creating diagnostic image in /data/ATG/MLRRC4/user_projects/domains/...

   /incident/incdir_359 with a lockout minute period of 1.>

[INCIDENT_ERROR] [JUFormBinding] java.lang.Exception: An application error occurred.  

See the incident log for more information.
  at oracle.apps.fnd.applcore.messages.ExceptionHandlerUtil...

In this article, we will examine what WebLogic Incident and Diagnostic Framework are.

Diagnostic Framework^[1]

Oracle Fusion Middleware includes a Diagnostic Framework (DFW).  DFW is available with all FMW 11g installations that run on WebLogic Server.  It aids in detecting, diagnosing, and resolving problems. The problems that are targeted in particular are critical errors such as those caused by:

• Code bugs
• Metadata corruption
• Customer data corruption
• Deadlocked threads
• Inconsistent state

When a critical error occurs, it is assigned an incident number, and diagnostic data for the error (such as log files) are immediately captured and tagged with this number. The data is then stored in the Automatic Diagnostic Repository (ADR).

The Automatic Diagnostic Repository (ADR) is a file-based repository for storing diagnostics data associated with incidents. It consists of metadata that describes each Problem and Incident, along with the set of diagnostic dump output generated for each incident.

ADR Directory Structure

The ADR root directory is known as ADR base. By default, the ADR base is located in the following directory:

• DOMAIN_HOME/servers/server_name/adr

Within ADR base, there can be multiple ADR homes, where each ADR home is the root directory for all incident data for a particular instance of Oracle WebLogic Server or a Java application.

For example, the following path shows the locations of ADR home for

• Oracle WebLogic Server instance
• ADR_BASE/diag/ofm/domain_name/server_name
• Fusion Application
• ADR_BASE/diag/ofm/fusionapps/app_name

ADRCI

The ADR Command Interpreter (ADRCI) is a utility that enables you to investigate problems, and package and upload first-failure diagnostic data to Oracle Support, all within a command-line environment. ADRCI also enables you to view the names of the dump files in the ADR, and to view the alert log with XML tags stripped, with and without content filtering.

ADRCI is installed in the following directory:

(UNIX) MW_HOME/wlserver_10.3/server/adr
(Windows) MW_HOME\wlserver_10.3\server\adr

$ find -name wlserver_10.3
$ cd ./MLRRC4/wlserver_10.3
$ cd server/adr
$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/export/home/bench/ATG/MLRRC4/wlserver_10.3/server/adr

$ export PATH=${PATH}:/export/home/bench/ATG/MLRRC4/wlserver_10.3/server/adr
$ adrci
ADRCI: Release 11.2.0.1.0 - Production on Tue Dec 11 19:47:41 2012

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

No ADR base is set
adrci> set base MW_HOME/user_projects/domains/atgdomain/servers/CRMDemo_server1/adr
adrci> show homes
ADR Homes:
diag/ofm/fusionapps/ApplicationCoreCRMDemoUI
diag/ofm/atgdomain/CRMDemo_server1

adrci> set homepath diag/ofm/fusionapps/ApplicationCoreCRMDemoUI
adrci> show incident
adrci> show incident -mode DETAIL -p "incident_id=359"

After setting LD_LIBRARY_PATH and PATH environment variables, we invoked adrci utility directly from the command line.  The first thing we did after entering ADRCI is setting ADR base and ADR home.  Within our specified ADR home, we listed all the incidents under it.  Then we listed a specific incident (i.e., 359) that we are interested in.  For further information, read [2,6].

Diagram Credit

• Figure 13-1 ADR Directory Structure for Oracle Fusion Middleware

References

1. Diagnosing Problems
• Describes how to use the Oracle Fusion Middleware Diagnostic Framework to collect and manage information about a problem so that you can resolve it or send it to Oracle Support for resolution.
2. ADRCI: ADR Command Interpreter
   • ADRCI is a command line utility that originated as a Oracle database software utility. Recent versions of Oracle Weblogic also include this utility.
3. Introduction to FMW Diagnostic Framework
4. How to View Java and PL/SQL Incidents
5. Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests
6. Using the ADRCI utility with Oracle Weblogic
7. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston
8. ADRCI-WebLogic
9. Using the ADRCI utility with Oracle Weblogic
10. Oracle Fusion Middleware Diagnostic Framework―How to Diagnosing Problems (XML and More)