Understanding WebLogic Incident and the Diagnostic Framework behind It

From the WebLogic Server log, I have found the following incident:

[INCIDENT_ERROR] [_createIncident] An incident was created. The details are: 

Incident details are: IncidentID=359
ProductID=fusionapps
InstanceID=ApplicationCoreCRMDemoUI
ProductType=ofm
DomainName=atgdomain
ServerName=CRMDemo_server1
. (FND-10102). (FND-10000)
<Dec 8, 2012 9:22:57 PM PST> <Alert> <Diagnostics> 

<BEA-320016> <Creating diagnostic image in /data/ATG/MLRRC4/user_projects/domains/...

   /incident/incdir_359 with a lockout minute period of 1.>

[INCIDENT_ERROR] [JUFormBinding] java.lang.Exception: An application error occurred.  

See the incident log for more information.
  at oracle.apps.fnd.applcore.messages.ExceptionHandlerUtil...

In this article, we will examine what WebLogic Incident and Diagnostic Framework are.

Diagnostic Framework^[1]

Oracle Fusion Middleware includes a Diagnostic Framework (DFW).  DFW is available with all FMW 11g installations that run on WebLogic Server.  It aids in detecting, diagnosing, and resolving problems. The problems that are targeted in particular are critical errors such as those caused by:

• Code bugs
• Metadata corruption
• Customer data corruption
• Deadlocked threads
• Inconsistent state

When a critical error occurs, it is assigned an incident number, and diagnostic data for the error (such as log files) are immediately captured and tagged with this number. The data is then stored in the Automatic Diagnostic Repository (ADR).

The Automatic Diagnostic Repository (ADR) is a file-based repository for storing diagnostics data associated with incidents. It consists of metadata that describes each Problem and Incident, along with the set of diagnostic dump output generated for each incident.

ADR Directory Structure

The ADR root directory is known as ADR base. By default, the ADR base is located in the following directory:

• DOMAIN_HOME/servers/server_name/adr

Within ADR base, there can be multiple ADR homes, where each ADR home is the root directory for all incident data for a particular instance of Oracle WebLogic Server or a Java application.

For example, the following path shows the locations of ADR home for

• Oracle WebLogic Server instance
• ADR_BASE/diag/ofm/domain_name/server_name
• Fusion Application
• ADR_BASE/diag/ofm/fusionapps/app_name

ADRCI

The ADR Command Interpreter (ADRCI) is a utility that enables you to investigate problems, and package and upload first-failure diagnostic data to Oracle Support, all within a command-line environment. ADRCI also enables you to view the names of the dump files in the ADR, and to view the alert log with XML tags stripped, with and without content filtering.

ADRCI is installed in the following directory:

(UNIX) MW_HOME/wlserver_10.3/server/adr
(Windows) MW_HOME\wlserver_10.3\server\adr

$ find -name wlserver_10.3
$ cd ./MLRRC4/wlserver_10.3
$ cd server/adr
$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/export/home/bench/ATG/MLRRC4/wlserver_10.3/server/adr

$ export PATH=${PATH}:/export/home/bench/ATG/MLRRC4/wlserver_10.3/server/adr
$ adrci
ADRCI: Release 11.2.0.1.0 - Production on Tue Dec 11 19:47:41 2012

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

No ADR base is set
adrci> set base MW_HOME/user_projects/domains/atgdomain/servers/CRMDemo_server1/adr
adrci> show homes
ADR Homes:
diag/ofm/fusionapps/ApplicationCoreCRMDemoUI
diag/ofm/atgdomain/CRMDemo_server1

adrci> set homepath diag/ofm/fusionapps/ApplicationCoreCRMDemoUI
adrci> show incident
adrci> show incident -mode DETAIL -p "incident_id=359"

After setting LD_LIBRARY_PATH and PATH environment variables, we invoked adrci utility directly from the command line.  The first thing we did after entering ADRCI is setting ADR base and ADR home.  Within our specified ADR home, we listed all the incidents under it.  Then we listed a specific incident (i.e., 359) that we are interested in.  For further information, read [2,6].

Diagram Credit

• Figure 13-1 ADR Directory Structure for Oracle Fusion Middleware

References

1. Diagnosing Problems
• Describes how to use the Oracle Fusion Middleware Diagnostic Framework to collect and manage information about a problem so that you can resolve it or send it to Oracle Support for resolution.
2. ADRCI: ADR Command Interpreter
   • ADRCI is a command line utility that originated as a Oracle database software utility. Recent versions of Oracle Weblogic also include this utility.
3. Introduction to FMW Diagnostic Framework
4. How to View Java and PL/SQL Incidents
5. Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests
6. Using the ADRCI utility with Oracle Weblogic
7. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston
8. ADRCI-WebLogic
9. Using the ADRCI utility with Oracle Weblogic
10. Oracle Fusion Middleware Diagnostic Framework―How to Diagnosing Problems (XML and More)