Oracle Fusion Middleware Diagnostic Framework―How to Diagnosing Problems

Oracle Fusion Middleware includes a Diagnostic Framework, which aids in detecting, diagnosing, and resolving problems. The problems that are targeted in particular are critical errors, such as those caused by

• Code bugs
• Metadata corruption
• Customer data corruption
• Deadlocked threads
• Inconsistent state

In this article, we will cover what Oracle Fusion Middleware Diagnostic Framework is and how it works.

Problem vs Incident

• Problem 
• Is a critical error
• Critical errors manifest as internal errors or other severe errors. Problems are tracked in the Automatic Diagnostic Repository (ADR).
• Has a problem key
• Is a text string that describes the problem
• Includes an error code (in the format XXX-nnnnn) and in some cases, other error-specific values.
• incident 1123 created with problem key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.executeBaseQuery][bi-contentstorage]"
• Incident
• Is a single occurrence of a problem
• When a problem (critical error) occurs multiple times, an incident is created for each occurrence. Incidents are timestamped and tracked in the ADR
• Is identified by a numeric incident ID (see 1123 above), which is unique within the ADR home

Oracle Fusion Middleware Diagnostic Framework 

When a critical error occurs, it is assigned an incident number, and diagnostic data for the error (such as log files) are immediately captured and tagged with this number. The data is then stored in the Automatic Diagnostic Repository (ADR), where it can later be retrieved by incident number and analyzed.  Here is the summary of its features:

• Supports incident detection log filter
• Implements the java.util.logging filter
• Inspects each log message to see if an incident should be created, basing its decision on the diagnostic rules for components and applications.
• Integrated with WebLogic Diagnostics Framework (WLDF)
• All diagnostic data relating to a critical error is captured and stored as an incident in Automatic Diagnostic Repository (ADR)
• Collects diagnostic data, such as
• Thread dumps, DMS metric dumps, and WebLogic Diagnostics Framework (WLDF) server image dumps 
• Provides standardized log formats
• Using the ODL log file format across all Oracle Fusion Middleware components.
• Incident flood control
• Diagnostic Framework applies flood control to incident generation after certain thresholds are reached
• To avoid generating too much diagnostic data, which would consume too much space in the ADR and could possibly slow down your efforts to diagnose and resolve the problem
• Example:
[2019-02-08T23:59:50.082+00:00] [bi_server2] [WARNING] [DFW-40125] [oracle.dfw.incident] [tid: [ACTIVE].ExecuteThread: '62' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 551d9654-1bc1-4b2f-b8d4-cbd3ab71603c-0004765a,0] [partition-name: DOMAIN] [tenant-name: GLOBAL] incident flood controlled with Problem Key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.executeBaseQuery][bi-contentstorage]"

Integration with WLDF

Oracle Fusion Middleware Diagnostics Framework integrates with the following components of WLDF:

• WLDF Watch and Notification
• Watches specific logs and metrics for specified conditions and sends a notification when a condition is met. 
• Oracle Fusion Middleware Diagnostics Framework integrates with the WLDF Watch and Notification component to create incidents.
• There are several types of notifications, including JMX notification and a notification to create a Diagnostic Image. 
• Diagnostic Image Capture
• Gathers the most common sources of the key server state used in diagnosing problems. 
• Packages that state into a single artifact, the Diagnostic Image. 
• With Oracle Fusion Middleware Diagnostics Framework, it writes the artifact to ADR.

Figure 1 shows the interaction when the incident is detected by the incident log detector. It shows the interaction among the incident log detector, the WLDF Diagnostic Image MBean, ADR, and component or application dumps when an incident is detected by the incident log detector.

Figure 1.  Incident Creation Generated by Incident Log Detector

Sample WebLogic Server Log

<Feb 8, 2019 11:59:54,143 PM UTC> <Notice> <Diagnostics> <xxxxxxx020308oacpod-bi-2.svcsbnet308.yyyyyyy2.oraclevcn.com> <bi_server2> <[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <eed7eff4-508d-4c1d-9c2b-b19d8e8936a6-0007d1d0> <1549670394143> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320068> <Watch "UncheckedException" in module "Module-FMWDFW" with severity "Notice" on server "bi_server2" has triggered at Feb 8, 2019 11:59:54 PM UTC. Notification details:
WatchRuleType: Log
WatchRule: (log.severityString == 'Error') and ((log.messageId == 'WL-101020') or (log.messageId == 'WL-101017') or (log.messageId == 'WL-000802') or (log.messageId == 'BEA-101020') or (log.messageId == 'BEA-101017') or (log.messageId == 'BEA-000802'))
WatchData: MESSAGE = [ServletContext@879994790[app:bi-servicelcm-rest module:bi-servicelcm-rest path:null spec-version:3.1]] Root cause of ServletException.
oracle.bi.servicelcm_v2.exceptions.PersistenceBackendException: Unable to create Pod record
at oracle.bi.servicelcm_v2.db.DatabasePodPersistenceManager.getPodImpl(DatabasePodPersistenceManager.java:39) 

... 

<Feb 8, 2019 11:59:58,489 PM UTC> <Emergency> <oracle.dfw.incident> <xxxxxxx020308oacpod-bi-2.svcsbnet308.yyyyyyy2.oraclevcn.com> <bi_server2> <[ACTIVE] ExecuteThread: '68' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <eed7eff4-508d-4c1d-9c2b-b19d8e8936a6-0007d1d1> <1549670398489> <[severity-value: 1] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <incident 1326 created with problem key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.bi.servicelcm_v2.db.SqlHelper.doTransaction][bi-servicelcm-rest]">

DFW-99998 is one of the "Uncaught Exception Problem Keys" and its specific format is:

• DFW-99998 [exception-name][package.class.name][app-name]

For example, the [app-name] in the above example is bi-servicelcm-rest.

For more information about WLDF, see Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

References

1. Diagnosing Problems (Oracle Fusion Middleware Diagnostic Framework)
2. Fusion Middleware Error Messages Reference
3. Understanding WebLogic Incident and the Diagnostic Framework behind It (XML and More)

Configuring Diagnostic Framework (DFW) Settings

A while ago, I've posted an article titled

Understanding WebLogic Incident and the Diagnostic Framework behind It^[1]

This article is a follow-up of that one.  In this article, we will discuss how to configure Diagnostic Framework Settings.

Diagnostic Framework (DFW)

A quick recap what Diagnostic Framework (DFW) is.  Oracle Fusion Middleware includes a Diagnostic Framework (DFW). DFW is available with all FMW 11g installations that run on WebLogic Server. It aids in detecting, diagnosing, and resolving problems, which are targeted in particular critical errors.

There are two ways that you can modify DFW settings:

1. Modifying the configuration file named dfw_config.xml
2. Making updates via Fusion Middleware Control (FMW Console)^[2]

In this article, we will show how to modify the following setting:

• maxTotalIncidentSize 

which configures the maximum total disk space allocated to incidents.

Configuration File

DFW's configuration file is named dfw_config.xml.  There is one dfw_config.xml file for each server.  For example, there is one for  CRMCommonServer_1:

• config/fmwconfig/servers/CRMCommonServer_1/dfw_config.xml

in the CRMDomain.

Here is the sample contents of dfw_config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<diagnosticsConfiguration xmlns="<snipped>" 
  xmlns:xs="http://www.w3.org/2001/XMLSchema-instance">
  <!-- maxTotalIncidentSize configures the maximum total disk space
       allocated to incidents, in megabytes. -->
  <incidentCleanup maxTotalIncidentSize="500"/>
  <incidentCreation
    incidentCreationEnabled="true"
    logDetectionEnabled="true"
    uncaughtExceptionDetectionEnabled="true"
    floodControlEnabled="true"
    floodControlIncidentCount="5"
    floodControlIncidentTimePeriod="60"
    reservedMemoryKB="512"/>
  <threadDump useExternalCommands="true"/>
  <dumpSampling enabled="true">
    <dumpSample
      sampleName="JVMThreadDump"
      diagnosticDumpName="jvm.threads"
      samplingInterval="60"
      rotationCount="10"
      dumpedImplicitly="true"
      toAppend="true">
      <dumpArgument name="timing" value="true"/>
      <dumpArgument name="context" value="true"/>
    </dumpSample>
    <dumpSample
      sampleName="JavaClassHistogram"
      diagnosticDumpName="jvm.classhistogram"
      samplingInterval="1800"
      rotationCount="5"
      dumpedImplicitly="false"
      toAppend="true">
    </dumpSample>
  </dumpSampling>
</diagnosticsConfiguration> 

In our case, we would like to update

• maxTotalIncidentSize

to 150 MB from 500 MB.

FMW Console

To start Oracle Enterprise Manager 11g, I have used the following URL:

• http://myserver.oracle.com:9001/em

The following diagrams show how to configure

• maxTotalIncidentSize 

using the Fusion Middleware Control System MBean Browser:

1. From the target navigation pane, expand the farm, then WebLogic Domain.
2. Select the domain.
3. From the WebLogic Domain menu, choose System MBean Browser.
4. The System MBean Browser page is displayed.
5. Expand Application Defined Beans, then oracle.dfw, then domain.domain_name, then dfw.jmx.DiagnosticsConfigMBean.
6. Select one of the DiagnosticConfig entries. There is one DiagnosticConfig entry for each server.
7. In the Application Defined MBean pane, expand Show MBean Information to see the server name.

References

1. Understanding WebLogic Incident and the Diagnostic Framework behind It (XML and More)
2. Configuring Diagnostic Framework Settings
3. Understanding the Diagnostic Framework in FMW 12c

Oracle Fusion Middleware 12c (12.1.2.0.0) Released

Oracle Fusion Middleware Team just announced the release of Oracle Fusion Middleware 12c (12.1.2.0.0). Media is being made available for download on the Oracle Technology Network (OTN), and the Oracle Software Delivery Cloud (OSDC). This includes the following products:

• Oracle WebLogic Server 12c (12.1.2.0.0)
• Oracle Coherence 12c (12.1.2.0.0)
• Oracle TopLink 12c (12.1.2.0.0)
• Oracle Fusion Middleware Infrastructure 12c (12.1.2.0.0)
• Oracle HTTP Server 12c (12.1.2.0.0)
• Oracle Virtual Assembly Builder 12c (12.1.2.0.0)
• Oracle JDeveloper 12c (12.1.2.0.0)

Here is the related information for these releases:

• Product Download (OTN, OSDC)
• Bill of Material
• Certification (OTN, My Oracle Support - MOS)
• Languages Supported
• Standards Supported
• Collateral
• Training
• Documentation
• Pricing
• Useful Support Note

You can also visit the Oracle Fusion Middleware page on OTN to find more information about the products.

Cloning Issue—What If Host Name(s) Are Stored in the Database

This article is a follow-up to the previous article:

• Simplify Cloning by Using Hosts File^[1]

As pointed out in that article, there are limitations and issues with cloning—either an application or a database. This article describes one of the issues (see also [2, 3, 4]): when host names are stored in the database tables.

Cloning

In [1], we have outlined the cloning tasks step by step.  In this article, we will discuss one of the cloning issues:

• If host name(s) are stored in the database tables

To do cloning (see [3]), we need to duplicate a software installation from a source to a destination.  As described in [1], you can use /etc/hosts file to simplify the cloning task by redirecting host name resolution on Linux. However, if the host name is stored in the database table and not yet listed in the hosts file, your application can still fail.

In the following section, we will describe one use case that showcases this issue.

ADF Task Flow

A user would like to change the host name of a Redhat Linux which hosts SOASuite and WebCenter.  After following the approach as described in [1] using hosts file.  He found out that his workflow still wasn't working correctly.

When he opened a task, the browser tried to connect to:

• http://localhost.localdomain:7001/workflow/SubmitTravelRequestUIView/faces/adf.task-flow

and he has found that localhost.localdomain still pointed to the previous host.  Now the server doesn't know how to resolve this name and asks browsers to connect to that host.

After some diggings, he has finally identified that there is a setting in the WFTASKDISPLAY table of SOAINFRA schema, which still keeps the previous host name.  Presumably, when a task was deployed, previous host name was stored there and caused the malfunction of his ADF task flow.

To be honest, this is a case reported by another user.  So, I might have described it wrong.  However, it does point out a potential problem—if you have renamed a server and some other servers may not recognize that old name anymore because the old host name is stored in the database.  So, you need to change hosts file on those machines too to redirect the mapping of the old name to the new IP address.

References

1. Simplify Cloning by Using Hosts File
2. ORA-00313: open failed for members of log group 1 of thread 1
3. Using rsync to Clone Local and Remote Systems
4. ORA-01031: insufficient privileges
5. Migrating Oracle B2B from Test to Production (T2P) (Chap 10 of the Book "Getting Started with Oracle SOA B2B Integration: A Hands-On Tutorial")
• This section provides a real-world scenario to replicate (clone) the test environment to production for Oracle SOA.
• Oracle Fusion Middleware provides a series of scripts for this task.
6. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)
• For your Oracle production systems, follow official recommendations as shown in this article.

How to Download WebLogic Upgrade Installer?

In our old Oracle Fusion Middleware Installation, we have a 10.3.5.0 WebLogic Server. See [1] for how to find the version of WebLogic Server.  Because we are testing a new feature in HotSpot VM which requires 10.3.6.0 WebLogic Server to work with, we need to upgrade WebLogic Server from 10.3.5.0 to 10.3.6.0.

After some research, we have found [2] which provides the upgrade instructions and decided to give it a try.  In this article, we report the steps that we have used to download the WebLogic Upgrade Installer.

Types of Installers

In total, there are four types of WebLogic Server installers available:

• OS-specific Package installer
• Generic Package installer
• Upgrade installer
• Development-only and supplemental installers

You can read [4] for the detailed descriptions of these various types of installers.  For our purpose, what we need is an Upgrade installer.

Upgrade installers, as their names suggest, allow you to upgrade an existing WebLogic Server installation to a later patch release.  Note that if you have an existing WebLogic Server 10.3.0, 10.3.1, 10.3.2, or 10.3.3 installation that includes Workshop for WebLogic, and you want to use an Upgrade installer to upgrade that installation to WebLogic Server 10.3.6, you must uninstall Workshop for WebLogic before running the Upgrade installer.  See [5] for more information.

How to Download?

Our steps have followed the instructions in [3], however, with minor modifications.  To download an Upgrade installer:

1. Enter the My Oracle Support URL (https://support.oracle.com/) in a Web browser.
2. Click Sign In and enter your My Oracle Support username and password.
3. Select the Patches and Updates tab.
4. In the Patch Search pane, click Product or Family (Advanced Search).
5. From the Product Is: drop-down list, select Oracle WebLogic Server.
6. From the Release Is: drop-down list, click the arrow next to the Oracle WebLogic Server folder, select the release you want to download, and click Close. (To upgrade to WebLogic Server 10.3.6, select WLS 10.3.6.0 from the list.)
7. From the Platform drop-down list, select your platform and click Close. You can select multiple platforms. Selected platforms are indicated by a check mark.
8. Click + next to Platform drop-down list. A new filter row appears and + changes to -.
9. From the left operand drop-down list, select Description.  From the operator drop-down list, select contains.  In the right operand field, specify INSTALLER.
10. Click Search. The search results list displays all available Upgrade installers for the selected release for all of the selected platforms.
11. Click the Download button on the right to begin the download.
12. Click Save.
13. Browse to the directory where you want to save the installer, and click Save again to start the file download. A compressed file downloads for the selected platform.
14. After the download completes, extract the compressed file, which contains only the appropriate installer executable for the selected platform.

Note that the typical description for an upgrade installer patch includes the following text:

• WEBLOGIC SERVER 11GR1(10.3.4) UPGRADE INSTALLER

For example, here is the description of the patch that we have found:

• PLACEHOLDER BUG FOR WEBLOGIC SERVER 11GR1 (10.3.6) UPGRADE INSTALLER (Patch)

Finally, if you run into download issue (for example, the download sits at 99% and never completes), try using different browser (for example, Chrome instead of IE).

References

1. How to find Oracle WebLogic Server Version?
2. Oracle® Fusion Middleware Installation Guide for Oracle WebLogic Server 11g Release 1 (10.3.6)
3. Downloading an Upgrade Installer From My Oracle Support
4. Types of Installers
5. Uninstalling the Software
6. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston
7. Oracle® Fusion Middleware Patching Guide 11g Release 1 (11.1.1.7.0)
• Patching involves copying a small collection of files over an existing installation.
• Upgrade involves moving from a previous major version to a new major version. For example, an upgrade would be required to move from Oracle Application Server 10g to Oracle Fusion Middleware 11g.
8. Oracle Fusion Middleware 12c —Install, Patch, and Upgrade
9. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More) 

umount: /dev/sdd: device is busy

This article is a follow-up on a previous article [1].  When we tried to unmount one of the file system, it failed with the following messages:

# umount /dev/sdd
umount: /dev/sdd: device is busy
umount: /dev/sdd: device is busy

In the following sections, we will summarize what we have found.

What Was Keeping the Device Busy?

On Linux platforms, you can use fuser command to find what processes have been keeping your device busy. The fuser command lists the process numbers of local processes that use the local or remote files specified by the File parameter. For block special devices, the command lists the processes that use any file on that device.  For example, here is what we have found with fuser command:
# fuser -m /dev/sdd
/dev/sdd:            24381c 24393ce 24449ce

In the listing, each process number is followed by a letter indicating how the process uses the file^[2]:

Item	Description
c	Uses the file as the current directory.
e	Uses the file as a program's executable object.
r	Uses the file as the root directory.
s	Uses the file as a shared library (or other loadable object).

What Application It Is Given a Process Number?

To find which application it is given its process number, you can use:

#  ps auxw|grep  24381
oracle   24381  0.0  0.0  63868  1160 ?        S    Dec19   0:00 /bin/sh -f /home/oracle/atg/IDM_11gR1.PS4.RC4/Oracle_IDM1/bin/emctl asstart agent

# ps auxw|grep  24393
oracle   24393  0.0  0.0  88552 10904 ?        S    Dec19   0:01 /home/oracle/atg/IDM_11gR1.PS4.RC4/Oracle_IDM1/perl/bin/perl /home/oracle/atg/IDM_11gR1.PS4.RC4/Oracle_IDM1/bin/emwd.pl agent /home/oracle/atg/IDM_11gR1.PS4.RC4/Instances/asinst_1/EMAGENT/EMAGENT/sysman/log/emagent.nohup

# ps auxw|grep  24449
oracle   24449  0.0  0.3 318624 38424 ?        Sl   Dec19   0:09 /home/oracle/atg/IDM_11gR1.PS4.RC4/Oracle_IDM1/bin/emagent

In our case, the above orphaned processes still hang around after the following command:

• $MW_HOME/asinst_1/bin/opmnctl  stopall

Normally, "opmnctl stopall"^[5] should be able to stop opmn and all managed processes.  However, there are some mis-configured host names in our Management Agent property file:

• $MW_HOME/asinst_1/EMAGENT/EMAGENT/sysman/config/emd.properties

which have caused orphaned processes unresponsive.

After stopping these orphaned processes by force, we are able to unmount /dev/sdd file system and have it checked by e2fsck command.

References

1. EXT3-fs warning: checktime reached, running e2fsck is recommended
2. fuser command
3. How to umount when the device is busy
4. Oracle® Database Administrator's Reference 11g Release 1 (11.1) for Linux and UNIX-Based Operating Systems
5. opmnctl Command Quick Reference

How to Configure Logging Using Weblogic Scripting Tool

Weblogic Scripting Tool (WLST) is a command-line tool that runs on the same machine as the Weblogic server and allows the user to browse the configuration and state of the server through a tree of mbeans (managememnt beans).  It is based on the Java scripting interpreter, Jython.

Using WLST, you can configure a server instance’s logging and message output.  To determine which log attributes can be configured, see LogMBean and LogFileMBean in the WebLogic Server MBean Reference^[1].

In this article, we first show you how to configure log attributes from WebLogic Server Administration Console and then show you how to set attributes of LogMBean using WLST.

Modifying Attribute from WLS Console

To bring WLS Administration Console up, you type the following address into your browser's address field:

• http://<myserver>:7001/console

and log in with your credentials (say, "weblogic/weblogic1").  To modify "Rotation file size" of the log attribute, you do:

• Click "Lock & Edit"
• Select and Click:
• Environment > Servers > CRMDemo_server1 > Logging

You modify "Rotation file size" to be a different value and then activate your change.  

Next to the "Rotation file size" field, you can click on "More Info..." to see its detailed description.

As you can see, "Rotation file size" field is linked to the following MBean Attibute:

• LogMBean.FileMinSize

Modifying Attribute from WLST

Instead of modifying log attributes from the console, you can also achieve it by using WLST.

$cd $MW_HOME/wlserver_10.3/common/bin

$./wlst.sh

wls:/offline> connect("weblogic","weblogic1", "t3://localhost:7001")  

wls:/atgdomain/serverConfig> cd("Servers/CRMDemo_server1/Log/CRMDemo_server1")

wls:/atgdomain/serverConfig/Servers/CRMDemo_server1/Log/CRMDemo_server1> ls()
dr--   DomainLogBroadcastFilter
dr--   LogFileFilter
-r--   FileMinSize                                  5000
-r--   FileName                                     logs/CRMDemo_server1.log
 .
 .
 .
-r-x   unSet                                        Void : String(propertyName)

As shown above, this is what happened:

1. We connected to the server
2. We set Current Management Object (CMO) to the server log config
3. We listed all LogMBean attributes
4. Our log attribute FileMinSize was shown on the list

To modify "FileMinSize" log attribute, you can create a script (i.e., setFileMinSize.py) such as:

# Connect to the server
connect("weblogic","weblogic1","t3://localhost:7001")
edit()
startEdit()

# set CMO to the server log config
cd("Servers/CRMDemo_server1/Log/CRMDemo_server1")
ls()

# change LogMBean attributes
set("FileMinSize", 400)

# list the current directory to confirm the new attribute values
ls()

# save and activate the changes
save()
activate(block="true")

# all done...
exit()

References

1. WebLogic Server MBean Reference
2. Getting Runtime Information
3. Oracle® Fusion Middleware WebLogic Scripting Tool Command Reference 11g Release 1 (10.3.3)
4. My "enable debug logging" in OAM WLST script
5. Configuring WebLogic Logging Services
6. weblogic.Server Command-Line Reference
7. How to List Current Message Levels of All Loggers Using WLST?

Understanding WebLogic Incident and the Diagnostic Framework behind It

From the WebLogic Server log, I have found the following incident:

[INCIDENT_ERROR] [_createIncident] An incident was created. The details are: 

Incident details are: IncidentID=359
ProductID=fusionapps
InstanceID=ApplicationCoreCRMDemoUI
ProductType=ofm
DomainName=atgdomain
ServerName=CRMDemo_server1
. (FND-10102). (FND-10000)
<Dec 8, 2012 9:22:57 PM PST> <Alert> <Diagnostics> 

<BEA-320016> <Creating diagnostic image in /data/ATG/MLRRC4/user_projects/domains/...

   /incident/incdir_359 with a lockout minute period of 1.>

[INCIDENT_ERROR] [JUFormBinding] java.lang.Exception: An application error occurred.  

See the incident log for more information.
  at oracle.apps.fnd.applcore.messages.ExceptionHandlerUtil...

In this article, we will examine what WebLogic Incident and Diagnostic Framework are.

Diagnostic Framework^[1]

Oracle Fusion Middleware includes a Diagnostic Framework (DFW).  DFW is available with all FMW 11g installations that run on WebLogic Server.  It aids in detecting, diagnosing, and resolving problems. The problems that are targeted in particular are critical errors such as those caused by:

• Code bugs
• Metadata corruption
• Customer data corruption
• Deadlocked threads
• Inconsistent state

When a critical error occurs, it is assigned an incident number, and diagnostic data for the error (such as log files) are immediately captured and tagged with this number. The data is then stored in the Automatic Diagnostic Repository (ADR).

The Automatic Diagnostic Repository (ADR) is a file-based repository for storing diagnostics data associated with incidents. It consists of metadata that describes each Problem and Incident, along with the set of diagnostic dump output generated for each incident.

ADR Directory Structure

The ADR root directory is known as ADR base. By default, the ADR base is located in the following directory:

• DOMAIN_HOME/servers/server_name/adr

Within ADR base, there can be multiple ADR homes, where each ADR home is the root directory for all incident data for a particular instance of Oracle WebLogic Server or a Java application.

For example, the following path shows the locations of ADR home for

• Oracle WebLogic Server instance
• ADR_BASE/diag/ofm/domain_name/server_name
• Fusion Application
• ADR_BASE/diag/ofm/fusionapps/app_name

ADRCI

The ADR Command Interpreter (ADRCI) is a utility that enables you to investigate problems, and package and upload first-failure diagnostic data to Oracle Support, all within a command-line environment. ADRCI also enables you to view the names of the dump files in the ADR, and to view the alert log with XML tags stripped, with and without content filtering.

ADRCI is installed in the following directory:

(UNIX) MW_HOME/wlserver_10.3/server/adr
(Windows) MW_HOME\wlserver_10.3\server\adr

$ find -name wlserver_10.3
$ cd ./MLRRC4/wlserver_10.3
$ cd server/adr
$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/export/home/bench/ATG/MLRRC4/wlserver_10.3/server/adr

$ export PATH=${PATH}:/export/home/bench/ATG/MLRRC4/wlserver_10.3/server/adr
$ adrci
ADRCI: Release 11.2.0.1.0 - Production on Tue Dec 11 19:47:41 2012

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

No ADR base is set
adrci> set base MW_HOME/user_projects/domains/atgdomain/servers/CRMDemo_server1/adr
adrci> show homes
ADR Homes:
diag/ofm/fusionapps/ApplicationCoreCRMDemoUI
diag/ofm/atgdomain/CRMDemo_server1

adrci> set homepath diag/ofm/fusionapps/ApplicationCoreCRMDemoUI
adrci> show incident
adrci> show incident -mode DETAIL -p "incident_id=359"

After setting LD_LIBRARY_PATH and PATH environment variables, we invoked adrci utility directly from the command line.  The first thing we did after entering ADRCI is setting ADR base and ADR home.  Within our specified ADR home, we listed all the incidents under it.  Then we listed a specific incident (i.e., 359) that we are interested in.  For further information, read [2,6].

Diagram Credit

• Figure 13-1 ADR Directory Structure for Oracle Fusion Middleware

References

1. Diagnosing Problems
• Describes how to use the Oracle Fusion Middleware Diagnostic Framework to collect and manage information about a problem so that you can resolve it or send it to Oracle Support for resolution.
2. ADRCI: ADR Command Interpreter
   • ADRCI is a command line utility that originated as a Oracle database software utility. Recent versions of Oracle Weblogic also include this utility.
3. Introduction to FMW Diagnostic Framework
4. How to View Java and PL/SQL Incidents
5. Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests
6. Using the ADRCI utility with Oracle Weblogic
7. Professional Oracle WebLogic Server by Robert Patrick, Gregory Nyberg, and Philip Aston
8. ADRCI-WebLogic
9. Using the ADRCI utility with Oracle Weblogic
10. Oracle Fusion Middleware Diagnostic Framework―How to Diagnosing Problems (XML and More)