Cross Column

Showing posts with label Troubleshooting. Show all posts
Showing posts with label Troubleshooting. Show all posts

Wednesday, February 11, 2026

🧭 How I Diagnosed a Cursor‑Unresponsive Freeze in Microsoft Edge

While rapidly switching between tabs and Copilot in Microsoft Edge, the cursor suddenly froze.
Photo Credit: Gemini Nano Banana

The Shift from Black Screens to Frozen Cursors

After resolving a black-screen conflict between Microsoft and Lenovo updates, a new, distinct issue emerged. While rapidly switching between tabs and Copilot in Microsoft Edge, the cursor suddenly froze. Unlike the previous total system lockups, the display remained active and the browser appeared to continue rendering content in the background.

Symptom Analysis: The GPU Timeout

This behavior points toward a GPU driver timeout (TDR hang) or a stall in the graphics power-management layer. Because Windows relies on the GPU to render the cursor, the pointer is often the first casualty when the graphics stack stalls, even if the rest of the system hasn't fully crashed. This specific failure likely resulted from the friction between Edge’s GPU-accelerated rendering and Lenovo’s background power-management services. A forced restart was required to recover, prompting a deeper dive into the mechanics of this partial freeze.

If you prefer the short version, you can read only the Key Takeaways and jump right to the Final Analysis below. Otherwise, the full article walks through the diagnostic process step by step and explains the technical details behind the cursor‑unresponsive freeze.


Key Takeaways

  • Evolving Failure Modes: After resolving a driver mismatch between Microsoft and Lenovo, the system transitioned from black-screen crashes to partial freezes (unresponsive cursor with an active display).
  • The TDR Hang: The frozen pointer indicates a GPU driver timeout (TDR stall). Because Windows utilizes the GPU to render the cursor, the pointer freezes first when the graphics stack or power-management layer stalls.
  • Trigger Conditions: The stall occurred during rapid tab-switching in Edge, a task that heavily stresses GPU acceleration and power-state transitions.
  • Hidden Lenovo Services: Despite an attempted removal, Lenovo Vantage and its background services remained active. Task Manager confirmed multiple backend processes were still managing thermal, power, and ACPI states.
  • Driver Conflict: Recent January 2026 updates to Lenovo’s ACPI and GPU components likely created friction with Windows power settings, leading to the kernel-level stalls identified in Reliability Monitor.
  • Persistence of Bloatware: Uninstalling Vantage does not remove all low-level drivers (Power Manager, ITS, Active Protection), which continue to exert control over system stability and GPU workloads.
  • Final Diagnosis: The freeze was a GPU/ACPI power-management stall triggered by the conflict between Lenovo’s background monitoring tools and intensive browser-based multitasking.

🧭 Step‑by‑Step Summary of the Diagnosing Process

With Copilot’s guidance, I walked through each step of the investigation to understand why the cursor became unresponsive while Edge was still alive. Here’s how the analysis unfolded through the conversation between me and the AI:

Step 1️⃣  — Identifying the Freeze Pattern

  • I reported a system freeze while using Edge.
  • Event Viewer showed Kernel‑Power 41 with no meaningful events before it.
  • Reliability Monitor confirmed “Windows was not properly shut down.”
  • This pattern indicated a deep kernel‑level stall, not a normal crash.

Conclusion: The freeze was caused by a low‑level driver or ACPI/power subsystem hang.

Step 2️⃣— Narrowing Down the Likely Subsystem

  • I described switching between Edge tabs (Google, Copilot, Gmail).
  • Cursor froze first — a classic sign of a GPU driver stall.
  • AMD‑based ThinkPads are known for GPU + ACPI instability.

Conclusion: The freeze was likely triggered by GPU or power‑management drivers.

Step 3️⃣ — Investigating Recent Driver Changes

I showed a list of Lenovo driver updates from January 2026.

These updates often include:

  • ACPI/power‑management components
  • GPU‑related modules
  • Embedded controller interactions

Conclusion: A recent Lenovo‑pushed driver update likely destabilized the system, an issue that was resolved in the previous article.

Step 4️⃣ — Checking Whether Lenovo Vantage Was Still Installed

  • I initially believed Lenovo Vantage had already been uninstalled. 
  • However, I soon discovered that wasn’t the case—both Lenovo Vantage and Lenovo Vantage Service were still listed in the Apps menu.

Conclusion: The Lenovo ecosystem was still active and influencing drivers.

Open Task Manager → go to the Processes tab → look for any Lenovo‑related processes
Open Task Manager → go to the Processes tab → look for any Lenovo‑related processes


Step 5️⃣ — Inspecting Running Processes

Task Manager revealed:

  • Lenovo.Modern.ImController (32 bit)
  • Lenovo Power Manager Host (32 bit)
  • Lenovo Power Management Service
  • Lenovo ITS Service
  • Lenovo Active Protection System

These are the core Lenovo backend components that manage:

  • ACPI control
  • GPU power-state management
  • thermal behavior
  • driver coordination

When active, this backend can silently push driver updates, override Windows power settings, alter GPU power states, load unstable ACPI modules, interfere with sleep and wake transitions, and even trigger freezes during GPU‑accelerated tasks such as rapid tab switching in Edge. In short, Lenovo’s backend was still fully active—and fully capable of causing the cursor‑unresponsive freeze I experienced.

Conclusion: Lenovo’s backend was fully active and could cause freezes.

Step 6️⃣ — Removing Lenovo Vantage

I uninstalled:

  • Lenovo Vantage
  • Lenovo Vantage Service

After reboot, Task Manager showed all Vantage components gone.

Conclusion: The Lenovo control stack was successfully removed.

Step 7️⃣ — Identifying Remaining Lenovo System Services

I still saw:

  • Lenovo Vantage (2)
  • Lenovo Power Manager Host
  • Lenovo ITS Service
  • Lenovo Active Protection System

These are separate Lenovo system drivers, not part of the Vantage app.

Their roles:

  • Power Manager → ACPI + battery + power states
  • ITS → thermal control
  • Active Protection → HDD shock protection (irrelevant for SSD)

Conclusion: These remaining services can still influence system stability and can be disabled if you want a fully “pure Windows” environment.


πŸ“Œ Final Diagnosis


The cursor-unresponsive freeze was caused by a GPU/ACPI power-management stall (likely a TDR hang), specifically triggered during rapid tab switching in Edge—an activity that demands high-frequency power-state transitions and stresses the GPU’s rendering pipelines.

This instability appears to be part of a broader trend; notably, system performance has become increasingly volatile since the Lenovo warranty expired one year ago. The current failure is almost certainly the result of a "perfect storm" involving:

  • Lenovo’s January driver updates, which introduced regressions in power and GPU handling.
  • Active interference from Lenovo Vantage and its persistent backend services.
  • AMD GPU hardware acceleration demands during heavy multitasking.

Following an analysis via Microsoft Copilot, the primary recommendation is the complete removal of Vantage and its associated background services to eliminate these proprietary conflicts and restore system reliability.

Friday, March 5, 2021

RDG—Oracle Analytics Cloud Remote Data Gateway

You can connect to remote on-premise data sources from Oracle Analytics Cloud. This enables you to deploy Oracle Analytics Cloud with large on-premise data sets without migrating the data to the cloud. Users can analyze the data in data visualizations, and in reporting dashboards and analyses.
Oracle Analytics Cloud Data Gateway (which supersedes Remote Data Connector) enables you to query on-premises data sources from Oracle Analytics Cloud.

Note that a new feature named private access channel (PAC) was also introduced for the Oracle Analytics Cloud (OAC) Gen 2 instances since 5.9.  Here are the high-level descriptions of of it:[14]

Private Access Channel compliments and provides an alternative to Remote Data Gateway (RDG) for the private data sources it supports. Both PAC and RDG may be used within the same instance of OAC. There may be sources that PAC supports and RDG does not and vice-versa.

Figure 1.  Data Gateway Architecture (Source: [11]) . The callouts are
    1. Oracle Analytics Cloud issues and queues queries using Data Gateway queue.
    2. The agent looks for queries to process via long-polling.
    3. The agent executes the queries.
    4. The agent sends the query results to Oracle Analytics Cloud.

Data Gateway Architecture


Figure 1 shows a high-level diagram of the architecture for Data Gateway. 

Oracle Analytics Cloud communicates through a firewall with an on-premise database using HTTPS, with an agent installed in the on-premise environment that channels database queries between Oracle Analytics Cloud and the on-premise database. 
On-premise Environment
You install Data Gateway in a subnet that gives visibility to both Oracle Analytics Cloud and the target data sources. Your network needs to allow outgoing (egress) traffic from the node where Data Gateway is installed to the public internet on port 443 so that Data Gateway can communicate with Oracle Analytics Cloud. In addition, the network needs to allow outgoing (egress) traffic from the Data Gateway agent to the data source. 
For example, you might test the network by opening a browser on the node where Data Gateway is installed and connect to Oracle Analytics Cloud. You might also test the connection from the same node to the data source using a generic JDBC tool.
Data Gateway regularly communicates with Oracle Analytics Cloud to see whether Oracle Analytics Cloud has queries that need processing, a process known as long-polling. Data Gateway makes a long-running TLS-encrypted HTTP request to Oracle Analytics Cloud and waits until Oracle Analytics Cloud has a query to process. If there're no queries from Oracle Analytics Cloud after two minutes, Data Gateway terminates and re-issues the request in order to avoid the request being identified and terminated as an idle or stale connection by the network.
The HTTPS communication between Data Gateway and Oracle Analytics Cloud leverages the SSL certificate of your Oracle Analytics Cloud service instance. The same certificate is used to encrypt your browser connections to Oracle Analytics Cloud.

Data Gateway vs Remote Data Connector


Data Gateway replaces the Remote Data Connector utility that was used in earlier releases. Although you can still use Remote Data Connector, Oracle recommend that you use Data Gateway. Data Gateway has many advantages over Remote Data Connector:

  • You can deploy multiple Data Gateway agents to query the same database. This provides failover and a highly-available architecture.
  • In Remote Data Connector, the communication is initiated by Oracle Analytics Cloud. In Data Gateway, the communication is initiated by Data Gateway. As a result:
    • You don't need to install anything in a network demilitarized zone (DMZ).
    • You don't have to open ports for ingress traffic in the on-premises firewall.
    • You don't have to register your Oracle Analytics Cloud as a 'safe' domain.
Note that you cannot use Data Gateway and Remote Data Connector at the same time with your Oracle Analytics Cloud instance.

Server Deployment vs Personal Deployment

Data Gateway supports two kinds of deployment:

  • Server Deployment
    • You can use Oracle Universal Installer to install on Linux
    • You can have multiple Data Gateway agents on each node (physical or virtual)
    • You can edit Data Gateway properties in 
      • <Installed location>/jetty/obiee_rdc_agent.properties
  • Personal Deployment
    • You can copy the binary files onto MacOS or Windows
    • You can edit Data Gateway properties in 
      • <Installed location>\obiee_rdc_agent.properties

High Availability


High Availability is supported for Data Gateway and Oracle recommends at least two Data Gateways per Oracle Analytics Cloud instance.  

How HA is supported?

  • On the Oracle Analytics Cloud side, high availability is provided natively.
  • On the Data Gateway side, you set up high availability by deploying two Data Gateways for each Oracle Analytics Cloud instance.
    • However, all of these agents must be capable of servicing all remote queries (that is, you cannot configure one agent to service queries for one data source only, and another agent to service queries for a different data source).

Troubleshoot Data Gateway

Here's how to troubleshoot common setup issues with Data Gateway.

Issue reportedDo this
All issues

Check that you are on the correct version of Data Gateway that matches your version of Oracle Analytics Cloud.

To check the version of the Data Gateway agent, on a server deployment, refer to the version in the file <installdirectory>/inventory/registry.xml.

Agent state change failed with error: Agent name or Oracle Analytics Cloud URL not specified or the Key pair not generatedClick Save, then Enable. If the problem persists, restart the application. If necessary, check your network.

Invalid Oracle Analytics Cloud URL (Data Gateway can't communicate with Oracle Analytics Cloud)

  • Check that you've enabled and configured Data Gateway in Oracle Analytics Cloud Console.
  • Make sure you can reach the Oracle Analytics Cloud URL from the environment where Data Gateway is running. For example, on Linux you might use a traceroute command, such as sudo traceroute -T -p 443 <Fully qualified domain name of your Oracle Analytics Cloud instance>.
  • Make sure there's nothing else blocking communication through the firewall.
  • If you're using a proxy:
    1. Stop the Data Gateway agent.
    2. On the machine where you've installed Data Gateway, configure the Data Gateway properties:

      In a server deployment, edit this file:

      <Installed location>/jetty/obiee_rdc_agent.properties

      In a personal deployment, edit this file:

      <Installed location>\obiee_rdc_agent.properties

    3. Update the following properties with the details of your internet proxy host:
      • proxyUserName
      • proxyPassword
      • proxyHost (for example, www-proxy.us.oracle.com)
      • proxyPort (for example, 80)
    4. Start the Data Gateway agent.
Test fails in Console page Remote Data Connectivity

If the test fails, this means that the Data Gateway agent can't authenticate. Possible reasons include:

  • The agent key hasn't been copied to the Remote Data Connectivity page in Oracle Analytics Cloud Console.
  • The agent key has been regenerated in the agent, but the new key hasn't been copied to the Remote Data Connectivity page in Oracle Analytics Cloud Console.
  • There's no suitable network route from the agent to Oracle Analytics Cloud.

Frequently Asked Questions about Data Gateway


Please check RDG Documentation on Data Gateway Frequently Asked Questions :
  • What operating systems does Data Gateway support?
  • What is the Data Gateway Architecture?
  • What is the difference between Remote Data Connector and Data Gateway?
  • Where do I install Data gateway?
  • Can I deploy multiple Data Gateway agents?
  • How do I configure High Availability for Data Gateway?
  • Why is Data Gateway traffic egress only?
  • How does Data Gateway manage SSL certificates?
  • How do I size Data Gateway?
  • Where is Data Gateway running? Do I install it on a virtual machine (VM)?
  • Can I use Data Gateway and Remote Data Connector with my Oracle Analytics Cloud instance at the same time?
  • How is Data Gateway network traffic secured?
  • Can Data Gateway limit queries that affect performance or security?
    • Data Gateway doesn't limit the query row size. The query row size limit is determined by the number of Oracle Compute Units (OCPUs) that your Oracle Analytics Cloud service has.
  • What is the timeout setting for Data Gateway?

Deployment of Remote Data Gateway


Read the below articles for the step-by-step instructions on the installation and deployment of remote data gateway (RDG) for Oracle Analytics Cloud:
Installing Oracle Analytics Cloud Remote Data Gateway
Preparing Linux for Oracle Remote Data Gateway
Deploying Oracle Analytics Cloud Remote Data Gateway in a Private Subnet
Deploying Oracle Analytics Cloud Remote Data Gateway in a Public Subnet
Deploying Oracle Analytics Cloud Remote Data Gateway On Premise
Deploying Oracle Analytics Cloud Remote Data Gateway using Local Peering Gateways

If you have subscribed to My Oracle Support, you can also check the below articles for more information:
  • NOTE:2590606.1 - OAC: How To Increase Logging For Remote Data Gateway Agent in OAC
  • NOTE:2574387.1 - Guidance For Using Remote Data Gateway (RDG) on Environments Upgraded From Oracle Analytics Cloud 105.2 and Lower
  • NOTE:2631494.1 - OAC: Test From Remote Data Gateway(RDG) Agent Config page fails with "Invalid OAC URL" Error
  • NOTES:2653783.1  - OAC RDG 5.5 : RDG Agent Configuration With Proxy Authentication Errors Out While Connecting To OAC
  • NOTES:2815638.1 - All Dashboard Reports Using RPD Failed With Error: Invalid Oracle URL specified

References

  1. Troubleshooting Data Gateway
  2. Installing Oracle Analytics Cloud Remote Data Gateway
  3. Completing the Deployment of Remote Data Gateway for Oracle Analytics Cloud
  4. OAC: Test From Remote Data Gateway(RDG) Agent Config page fails with "Invalid OAC URL" Error (Doc ID 2631494.1)
  5. OAC RDG 5.5 : RDG Agent Configuration With Proxy Authentication Errors Out While Connecting To OAC (Doc ID 2653783.1)
  6. Configuring Oracle Analytics Cloud
  7. Configure and Register Data Gateway for Data Visualization
  8. Give Data Sources Access to Analytics Cloud Instances
  9. Cache Management in Oracle Analytics Cloud 5.4 Onwards
  10. Best Practices from Oracle Development's A‑Team
  11. Connect to On-premise Data Sources
  12. Connecting to Private Data Sources using Oracle Analytics Cloud Private Access Channel
    • Private Access Channel compliments and provides an alternative to Remote Data Gateway (RDG) for the private data sources it supports. Both PAC and RDG may be used within the same instance of OAC.
  13. Preparing Network Gateways for Private Oracle Analytics Cloud Data Sources
  14. Connect to Private Data Sources Through a Private Access Channel
  15. Oracle Analytics Managing Security for Oracle Analytics Server, 5.9.0 (pdf)
  16. Troubleshooting Private Access Channel Connectivity for Oracle Analytics Cloud
  17. Oracle Analytics Cloud supported Data Sources
  18. Connecting to Oracle Analytics Cloud Private Endpoint with VPN / FastConnect
  19. Manage Service Access and Security
  20. Prerequisites for a Private Access Channel (PAC)

Sunday, August 2, 2020

How ABRT Avoid Storing Duplicated Crashes — Deduplication

Processes crash for a multitude of reasons and it’s often difficult to understand the root causes that contribute to such crashes.  The Automatic Bug Reporting Tool, commonly abbreviated as ABRT, could offer help for forensic investigation.

ABRT


ABRT consists of the abrtd daemon and a number of system services and utilities to process, analyze, and report detected problems. 

The daemon runs silently in the background most of the time, and springs into action when an application crashes or a kernel oops is detected. The daemon then collects the relevant problem data such as a core file if there is one, the crashing application's command-line parameters, and other data of forensic utility.

Why ABRT?


Earlier when applications crashed, core dumps were generated, but not limited, which could quickly fill up the disk.

A solution is to use ABRT.  For example, it can
  • Rotate cores within a size limit by deleting the oldest[11]
  • Avoid storing duplicate crashes by deduplication[9]

Elements Collected by ABRT


In the below table, it shows a shortened list of elements collected by ABRT and their descriptions. For a full list see [4].  These elements are stored in the form of files in a single directory per detected problem (such a directory is called 'dump directory').
 
core_backtraceMachine readable backtrace with no private data
coredumpCoredump of the crashing process
countNumber of times this problem occured
crash_functionFunction which crashed
dmesgCopy of dmesg
docker_inspectOutput of docker inspect $(container_id)
dso_listList of dynamic libraries loaded at the time of crash
duphashHash of the crash's backtrace
environDump of process environment variable along with their values
event_logMessages produced by ABRT tools during processing the detected problem
executableExecutable path of the component which caused the problem.
global_pidValue of %P as passed by kernel to the core_pattern helper (see man core for more details)
hostnameHostname of the affected machine
kernelKernel version string
kernel_logResults of vmcore crash analysis performed by retrace-server
kernel_tainted_longTainted kernel description
kernel_tainted_shortKernel tainted flags (For more information about tainted flags see [1])
last_occurrenceTime of the last occurence (unixtime)
 

Deduplication


When ABRT catches new crash, it compares it to the rest of the stored problems to avoid storing duplicate crashes:
  1. It first checks if there is core_bactrace or uuid item in the problem directory it is processing
  2. If there is a core_backtrace
    • It iterates over all other dump directories and computes similarity to their core backtraces (if any). If one of them is similar enough to be considered duplicate, event processing is stopped and only notify-dup event is fired.
  3. Or if there is an uuid item (and no core backtrace)
    • Simple comparison of uuid hashes is used for duplicate detection.
You can read abrt-action-analyze-backtrace for more information.[6]


count & last_occurrence


After the forensic investigation, you can use:
  • abrt-cli rm <path to the problem directory>
to remove the specified problem data directory with all its contents.

[abrt]# abrt-cli rm ccpp-2019-08-21-13:59:02-31929
PrivateReports is disabled. Run abrt-cli-root to see all problems detected by ABRT.
rm 'ccpp-2019-08-21-13:59:02-31929'

However, note that ABRT performs a detection of duplicate problems by comparing new problems with all locally saved problems. 

For a repeating crash, ABRT requires you to act upon it only once. But, if you delete the crash dump of that problem, the next time this specific problem occurs, ABRT will treat it as a new crash: ABRT will alert you about it, prompt you to fill in a description, and report it. To avoid having ABRT notifying you about a recurring problem, do not delete its problem data.

If you didn't remove a specific problem data directory, here is what would happen when ABRT catches a new crash :
  • ABRT compares it to the rest of locally stored problems
  • If it's a new problem, a new problem directory will be created
  • Otherwise, ABRT will update the recurring problem by:
    • Incrementing "count" by one
    • Updating "last_occurrence" with a new epoch
[ccpp-2019-08-21-13:59:02-31929]# ls -lrt
total 868572
-rw-r-----. 1 abrt  abrt         3 Aug 21  2019 uid
-rw-r-----. 1 abrt  abrt        10 Aug 21  2019 time
-rw-r-----. 1 abrt  abrt        32 Aug 21  2019 os_release
-rw-r-----. 1 abrt  abrt        30 Aug 21  2019 kernel
-rw-r-----. 1 abrt  abrt        24 Aug 21  2019 hostname
-rw-r-----. 1 abrt   abrt         6 Aug 21  2019 architecture
-rw-r-----. 1 abrt  abrt     70033 Aug 21  2019 maps
-rw-r-----. 1 abrt  abrt      1323 Aug 21  2019 limits
-rw-r-----. 1 abrt  abrt        88 Aug 21  2019 cgroup
-rw-r-----. 1 abrt  abrt         4 Aug 21  2019 type
-rw-r-----. 1 abrt  abrt        90 Aug 21  2019 reason
-rw-r-----. 1 abrt  abrt        39 Aug 21  2019 pwd
-rw-r-----. 1 abrt  abrt         5 Aug 21  2019 pid
-rw-r-----. 1 abrt  abrt      2072 Aug 21  2019 open_fds
-rw-r-----. 1 abrt  abrt        48 Aug 21  2019 executable
-rw-r-----. 1 abrt  abrt     14722 Aug 21  2019 environ
-rw-r-----. 1 abrt  abrt        48 Aug 21  2019 cmdline
-rw-r-----. 1 abrt  abrt         4 Aug 21  2019 analyzer
-rw-r-----. 1 abrt  abrt         5 Aug 21  2019 abrt_version
-rw-r-----. 1 abrt  abrt 886996992 Aug 21  2019 coredump
-rw-r-----. 1 abrt  abrt         7 Aug 21  2019 username
-rw-r-----. 1 abrt  abrt   1846076 Aug 21  2019 sosreport.tar.xz
-rw-r-----. 1 abrt  abrt         0 Aug 21  2019 event_log
-rw-r-----. 1 abrt  abrt        93 Aug 21  2019 machineid
-rw-r-----. 1 abrt  abrt    378414 Aug 21  2019 core_backtrace
-rw-r-----. 1 abrt  abrt        40 Aug 21  2019 uuid
-rw-r-----. 1 abrt  abrt      1424 Aug 21  2019 dso_list
-rw-r-----. 1 abrt  abrt       199 Aug 21  2019 var_log_messages
-rw-r-----. 1 abrt  abrt         2 Jul 25 08:00 count
-rw-r-----. 1 abrt  abrt        10 Jul 25 08:00 last_occurrence

[ccpp-2019-08-21-13:59:02-31929]# cat count
2

[ccpp-2019-08-21-13:59:02-31929]# cat last_occurrence
1595664006

[ccpp-2019-08-21-13:59:02-31929]# date -u -d @1595664006
Sat Jul 25 08:00:06 UTC 2020

[ccpp-2019-08-21-13:59:02-31929]# cat reason
Process /u01/app/xxx/server/bin/yyy was killed by signal 11 (SIGSEGV)

ABRT Configuration Files


Standard ABRT installation currently provides the following ABRT specific configuration files:
  • /etc/abrt/abrt.conf — allows you to modify the behavior of the abrtd service.
  • /etc/abrt/abrt-action-save-package-data.conf — allows you to modify the behavior of the abrt-action-save-package-data program.
  • /etc/abrt/plugins/CCpp.conf — allows you to modify the behavior of ABRT's core catching hook.
For example, the default location where problem data directories are created and in which problem core dumps and all other problem data are stored is:
/var/spool/abrt

[~]# cd /var/spool/abrt

[abrt]# ls -lrt
total 32
-rw-------. 1 root   root   23 Mar  8 05:18 last-via-server
-rw-------. 1 root   root   48 Jul 25 08:00 last-ccpp
drwxr-x---. 2 abrt   abrt 4096 Jul 28 15:22 ccpp-2019-08-21-13:59:02-31929

Read [11] for all the details of ABRT configuration files.

References

  1. ABRT Documentation (Release 2.14)
  2. How to properly delete a report problem in ABRT
  3. AUTOMATIC BUG REPORTING TOOL (ABRT)
  4. Elements collected by ABRT
  5. Basic ABRT components
  6. abrt-action-analyze-backtrace
    • Analyzes C/C++ backtrace, generates duplication hash, backtrace rating, and identifies crash function in problem directory DIR
    • Then it saves this data as new elements global_uuid, rating, crash_function in this problem directory
  7. abrt-backtrace
  8. ABRT FAQ
  9. ABRT Design
  10. backtrace_rating (Red Hat doc)
    • Numerical representation of quality of backtrace based on ratio of unrecognized frames among all backtrace frames
  11. ABRT SPECIFIC CONFIGURATION

Friday, February 15, 2019

Oracle Fusion Middleware Diagnostic Framework―How to Diagnosing Problems

Oracle Fusion Middleware includes a Diagnostic Framework, which aids in detecting, diagnosing, and resolving problems. The problems that are targeted in particular are critical errors, such as those caused by
  • Code bugs
  • Metadata corruption
  • Customer data corruption
  • Deadlocked threads
  • Inconsistent state
In this article, we will cover what Oracle Fusion Middleware Diagnostic Framework is and how it works.

Problem vs Incident

  • Problem 
    • Is a critical error
    • Has a problem key
      • Is a text string that describes the problem
      • Includes an error code (in the format XXX-nnnnn) and in some cases, other error-specific values.
        • incident 1123 created with problem key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.executeBaseQuery][bi-contentstorage]"
  • Incident
    • Is a single occurrence of a problem
      • When a problem (critical error) occurs multiple times, an incident is created for each occurrence. Incidents are timestamped and tracked in the ADR
    • Is identified by a numeric incident ID (see 1123 above), which is unique within the ADR home

Oracle Fusion Middleware Diagnostic Framework 


When a critical error occurs, it is assigned an incident number, and diagnostic data for the error (such as log files) are immediately captured and tagged with this number. The data is then stored in the Automatic Diagnostic Repository (ADR), where it can later be retrieved by incident number and analyzed.  Here is the summary of its features:
  • Supports incident detection log filter
    • Implements the java.util.logging filter
    • Inspects each log message to see if an incident should be created, basing its decision on the diagnostic rules for components and applications.
  • Integrated with WebLogic Diagnostics Framework (WLDF)
  • All diagnostic data relating to a critical error is captured and stored as an incident in Automatic Diagnostic Repository (ADR)
    • Collects diagnostic data, such as
  • Provides standardized log formats
    • Using the ODL log file format across all Oracle Fusion Middleware components.
  • Incident flood control
    • Diagnostic Framework applies flood control to incident generation after certain thresholds are reached
      • To avoid generating too much diagnostic data, which would consume too much space in the ADR and could possibly slow down your efforts to diagnose and resolve the problem
    • Example:
    • [2019-02-08T23:59:50.082+00:00] [bi_server2] [WARNING] [DFW-40125] [oracle.dfw.incident] [tid: [ACTIVE].ExecuteThread: '62' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 551d9654-1bc1-4b2f-b8d4-cbd3ab71603c-0004765a,0] [partition-name: DOMAIN] [tenant-name: GLOBAL] incident flood controlled with Problem Key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager.executeBaseQuery][bi-contentstorage]"

Integration with WLDF


Oracle Fusion Middleware Diagnostics Framework integrates with the following components of WLDF:
  • WLDF Watch and Notification
    • Watches specific logs and metrics for specified conditions and sends a notification when a condition is met. 
      • Oracle Fusion Middleware Diagnostics Framework integrates with the WLDF Watch and Notification component to create incidents.
    • There are several types of notifications, including JMX notification and a notification to create a Diagnostic Image. 
  • Diagnostic Image Capture
    • Gathers the most common sources of the key server state used in diagnosing problems. 
      • Packages that state into a single artifact, the Diagnostic Image
      • With Oracle Fusion Middleware Diagnostics Framework, it writes the artifact to ADR.
Figure 1 shows the interaction when the incident is detected by the incident log detector. It shows the interaction among the incident log detector, the WLDF Diagnostic Image MBean, ADR, and component or application dumps when an incident is detected by the incident log detector.
Figure 1.  Incident Creation Generated by Incident Log Detector
Sample WebLogic Server Log

<Feb 8, 2019 11:59:54,143 PM UTC> <Notice> <Diagnostics> <xxxxxxx020308oacpod-bi-2.svcsbnet308.yyyyyyy2.oraclevcn.com> <bi_server2> <[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <eed7eff4-508d-4c1d-9c2b-b19d8e8936a6-0007d1d0> <1549670394143> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320068> <Watch "UncheckedException" in module "Module-FMWDFW" with severity "Notice" on server "bi_server2" has triggered at Feb 8, 2019 11:59:54 PM UTC. Notification details:
WatchRuleType: Log
WatchRule: (log.severityString == 'Error') and ((log.messageId == 'WL-101020') or (log.messageId == 'WL-101017') or (log.messageId == 'WL-000802') or (log.messageId == 'BEA-101020') or (log.messageId == 'BEA-101017') or (log.messageId == 'BEA-000802'))
WatchData: MESSAGE = [ServletContext@879994790[app:bi-servicelcm-rest module:bi-servicelcm-rest path:null spec-version:3.1]] Root cause of ServletException.
oracle.bi.servicelcm_v2.exceptions.PersistenceBackendException: Unable to create Pod record
at oracle.bi.servicelcm_v2.db.DatabasePodPersistenceManager.getPodImpl(DatabasePodPersistenceManager.java:39) 
... 
<Feb 8, 2019 11:59:58,489 PM UTC> <Emergency> <oracle.dfw.incident> <xxxxxxx020308oacpod-bi-2.svcsbnet308.yyyyyyy2.oraclevcn.com> <bi_server2> <[ACTIVE] ExecuteThread: '68' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <eed7eff4-508d-4c1d-9c2b-b19d8e8936a6-0007d1d1> <1549670398489> <[severity-value: 1] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <incident 1326 created with problem key "DFW-99998 [weblogic.jdbc.extensions.PoolDisabledSQLException][oracle.bi.servicelcm_v2.db.SqlHelper.doTransaction][bi-servicelcm-rest]">
DFW-99998 is one of the "Uncaught Exception Problem Keys" and its specific format is:
  • DFW-99998 [exception-name][package.class.name][app-name]
For example, the [app-name] in the above example is bi-servicelcm-rest.

Monday, May 14, 2018

NTP Time Synchronization―All Things Considered

In this article, we will cover the following topics:
  1. Importance of time synchronization
  2. Needs for NTP time synchronization
  3. How to synchronize time
  4. How to verify synchronized time

Importance of Time Synchronization


Time synchronization plays an important role in cloud computing:
As single systems have been replaced by multiple, loosely-coupled systems, this need has evolved into a requirement for both accurate and consistent clocks among these systems. The need for millisecond or microsecond timestamp accuracy derives from the need to execute transactions in a correct sequence, particularly if many transactions occur almost simultaneously. 
You can read [1] for more details.

Needs for NTP Time Synchronization


Each of your network of servers has its own clock.  Computer clocks are notorious for drifting. They are typically based on inexpensive oscillator circuits or battery backed quartz crystals that can easily drift seconds and minutes per day, accumulating significant errors over time. That is why most enterprise networks today rely on network time servers that acquire time from the Global Navigation Satellite System (GNSS) and distribute it to clients over a network through the Network Time Protocol (NTP).

Timing accuracy from NTP servers depends upon the accuracy of the source and the precision with which all operations and applications are synchronized.

NTP Time Synchronization via utpupdate


ntpdate can be run manually as necessary to set the host clock, or it can be run from the host startup script to set the clock at boot time.

This is useful in some cases to set the clock initially before starting the NTP daemon ntpd. It is also possible to run ntpdate from a cron script. However, it is important to note that ntpdate with contrived cron scripts is no substitute for the NTP daemon, which uses sophisticated algorithms to maximize accuracy and reliability while minimizing resource use. Finally, since ntpdate does not discipline the host clock frequency as does ntpd, the accuracy using ntpdate is limited.

# /usr/sbin/ntpdate -u 10.252.148.42
14 May 09:57:05 ntpdate[2890]: step time server 10.252.148.42 offset 30.442100 sec

NTP Time Synchronization via NTP Daemon


Depending on your Linux systems, for example, below commands are used to update NTP server configuration file and restart ntpd in a local Oracle Linux system.


$ sudo bash
Password:

bash-3.2# vi /etc/ntp.conf
bash-3.2# /etc/init.d/ntpd restart
Shutting down ntpd:                                        [  OK  ]
ntpd: Synchronizing with time server:                      [  OK  ]
Starting ntpd:                                             [  OK  ]

Verification of Synchronized Time


There are multiple ways of verifying the results of NTP time synchronization:
  • ntpstat
    • ntpstat will report the synchronisation state of the NTP daemon running on the local machine.  
    • If the local system is found to be  synchronised to  a  reference  time source, ntpstat will report the approximate time accuracy.
  • date -R
    •  -R, --rfc-2822
      • output date and time in RFC 2822 format
  • /usr/sbin/ntpq -p
    • The  ntpq utility program is used to monitor NTP daemon ntpd operations and determine performance.
    • -p 
      • Print a list of the peers known to the server as well as a summary of their state. This is equivalent to the  peers  interactive command.

ntpstat

bash-3.2# ntpstat
unsynchronised
  time server re-starting
   polling server every 64 s


bash-3.2# ntpstat
synchronised to NTP server (10.252.148.42) at stratum 4
   time correct to within 31 ms
   polling server every 64 s

date

bash-4.1# date -R
Mon, 14 May 2018 08:17:33 -0700

/usr/sbin/ntpq -p

bash-4.1# /usr/sbin/ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ucf-c2z3-opc-di 10.68.0.42       3 u   20   64  377    0.971    0.105   0.057
 LOCAL(0)        .LOCL.          10 l    -   64    0    0.000    0.000   0.000


VariableDescription
remotehost name or IP number of peer
refidassociation ID or kiss code
stpeer status word
tu: unicast, b: broadcast, l: local
whensec/min/hr since last received packet
pollpoll interval (log2 s)
reachreach shift register (octal)
delayroundtrip delay
offsetoffset
jitterjitter


References

  1. The Importance of Network Time Synchronization for Enterprise Solutions Whitepaper
  2. Network Time Protocol (Version 3) Specification, Implementation and Analysis
  3. ntpq - standard NTP query program
  4. NTP Time Synchronization

© Travel for Life Guide. All Rights Reserved.

Analytical Insights on Health, Culture, and Security.