How ABRT Avoid Storing Duplicated Crashes — Deduplication

Processes crash for a multitude of reasons and it’s often difficult to understand the root causes that contribute to such crashes.  The Automatic Bug Reporting Tool, commonly abbreviated as ABRT, could offer help for forensic investigation.

ABRT

ABRT consists of the abrtd daemon and a number of system services and utilities to process, analyze, and report detected problems. 

The daemon runs silently in the background most of the time, and springs into action when an application crashes or a kernel oops is detected. The daemon then collects the relevant problem data such as a core file if there is one, the crashing application's command-line parameters, and other data of forensic utility.

Why ABRT?

Earlier when applications crashed, core dumps were generated, but not limited, which could quickly fill up the disk.

A solution is to use ABRT.  For example, it can

• Rotate cores within a size limit by deleting the oldest^[11]
• Avoid storing duplicate crashes by deduplication^[9]

Elements Collected by ABRT

In the below table, it shows a shortened list of elements collected by ABRT and their descriptions. For a full list see [4].  These elements are stored in the form of files in a single directory per detected problem (such a directory is called 'dump directory').

 

core_backtrace	Machine readable backtrace with no private data
coredump	Coredump of the crashing process
count	Number of times this problem occured
crash_function	Function which crashed
dmesg	Copy of dmesg
docker_inspect	Output of docker inspect $(container_id)
dso_list	List of dynamic libraries loaded at the time of crash
duphash	Hash of the crash's backtrace
environ	Dump of process environment variable along with their values
event_log	Messages produced by ABRT tools during processing the detected problem
executable	Executable path of the component which caused the problem.
global_pid	Value of %P as passed by kernel to the core_pattern helper (see man core for more details)
hostname	Hostname of the affected machine
kernel	Kernel version string
kernel_log	Results of vmcore crash analysis performed by retrace-server
kernel_tainted_long	Tainted kernel description
kernel_tainted_short	Kernel tainted flags (For more information about tainted flags see [1])
last_occurrence	Time of the last occurence (unixtime)

 

Deduplication

When ABRT catches new crash, it compares it to the rest of the stored problems to avoid storing duplicate crashes:

1. It first checks if there is core_bactrace or uuid item in the problem directory it is processing
2. If there is a core_backtrace
• It iterates over all other dump directories and computes similarity to their core backtraces (if any). If one of them is similar enough to be considered duplicate, event processing is stopped and only notify-dup event is fired.
3. Or if there is an uuid item (and no core backtrace)
• Simple comparison of uuid hashes is used for duplicate detection.

You can read abrt-action-analyze-backtrace for more information.^[6]

count & last_occurrence

After the forensic investigation, you can use:

• abrt-cli rm <path to the problem directory>

to remove the specified problem data directory with all its contents.

[abrt]# abrt-cli rm ccpp-2019-08-21-13:59:02-31929

PrivateReports is disabled. Run abrt-cli-root to see all problems detected by ABRT.

rm 'ccpp-2019-08-21-13:59:02-31929'

However, note that ABRT performs a detection of duplicate problems by comparing new problems with all locally saved problems. 

For a repeating crash, ABRT requires you to act upon it only once. But, if you delete the crash dump of that problem, the next time this specific problem occurs, ABRT will treat it as a new crash: ABRT will alert you about it, prompt you to fill in a description, and report it. To avoid having ABRT notifying you about a recurring problem, do not delete its problem data.

If you didn't remove a specific problem data directory, here is what would happen when ABRT catches a new crash :

• ABRT compares it to the rest of locally stored problems
• If it's a new problem, a new problem directory will be created
• Otherwise, ABRT will update the recurring problem by:
• Incrementing "count" by one
• Updating "last_occurrence" with a new epoch

[ccpp-2019-08-21-13:59:02-31929]# ls -lrt

total 868572

-rw-r-----. 1 abrt  abrt         3 Aug 21  2019 uid

-rw-r-----. 1 abrt  abrt        10 Aug 21  2019 time

-rw-r-----. 1 abrt  abrt        32 Aug 21  2019 os_release

-rw-r-----. 1 abrt  abrt        30 Aug 21  2019 kernel

-rw-r-----. 1 abrt  abrt        24 Aug 21  2019 hostname

-rw-r-----. 1 abrt   abrt         6 Aug 21  2019 architecture

-rw-r-----. 1 abrt  abrt     70033 Aug 21  2019 maps

-rw-r-----. 1 abrt  abrt      1323 Aug 21  2019 limits

-rw-r-----. 1 abrt  abrt        88 Aug 21  2019 cgroup

-rw-r-----. 1 abrt  abrt         4 Aug 21  2019 type

-rw-r-----. 1 abrt  abrt        90 Aug 21  2019 reason

-rw-r-----. 1 abrt  abrt        39 Aug 21  2019 pwd

-rw-r-----. 1 abrt  abrt         5 Aug 21  2019 pid

-rw-r-----. 1 abrt  abrt      2072 Aug 21  2019 open_fds

-rw-r-----. 1 abrt  abrt        48 Aug 21  2019 executable

-rw-r-----. 1 abrt  abrt     14722 Aug 21  2019 environ

-rw-r-----. 1 abrt  abrt        48 Aug 21  2019 cmdline

-rw-r-----. 1 abrt  abrt         4 Aug 21  2019 analyzer

-rw-r-----. 1 abrt  abrt         5 Aug 21  2019 abrt_version

-rw-r-----. 1 abrt  abrt 886996992 Aug 21  2019 coredump

-rw-r-----. 1 abrt  abrt         7 Aug 21  2019 username

-rw-r-----. 1 abrt  abrt   1846076 Aug 21  2019 sosreport.tar.xz

-rw-r-----. 1 abrt  abrt         0 Aug 21  2019 event_log

-rw-r-----. 1 abrt  abrt        93 Aug 21  2019 machineid

-rw-r-----. 1 abrt  abrt    378414 Aug 21  2019 core_backtrace

-rw-r-----. 1 abrt  abrt        40 Aug 21  2019 uuid

-rw-r-----. 1 abrt  abrt      1424 Aug 21  2019 dso_list

-rw-r-----. 1 abrt  abrt       199 Aug 21  2019 var_log_messages

-rw-r-----. 1 abrt  abrt         2 Jul 25 08:00 count

-rw-r-----. 1 abrt  abrt        10 Jul 25 08:00 last_occurrence

[ccpp-2019-08-21-13:59:02-31929]# cat count

2

[ccpp-2019-08-21-13:59:02-31929]# cat last_occurrence

1595664006

[ccpp-2019-08-21-13:59:02-31929]# date -u -d @1595664006

Sat Jul 25 08:00:06 UTC 2020

[ccpp-2019-08-21-13:59:02-31929]# cat reason

Process /u01/app/xxx/server/bin/yyy was killed by signal 11 (SIGSEGV)

ABRT Configuration Files

Standard ABRT installation currently provides the following ABRT specific configuration files:

• /etc/abrt/abrt.conf — allows you to modify the behavior of the abrtd service.
• /etc/abrt/abrt-action-save-package-data.conf — allows you to modify the behavior of the abrt-action-save-package-data program.
• /etc/abrt/plugins/CCpp.conf — allows you to modify the behavior of ABRT's core catching hook.

For example, the default location where problem data directories are created and in which problem core dumps and all other problem data are stored is:

/var/spool/abrt

[~]# cd /var/spool/abrt

[abrt]# ls -lrt

total 32

-rw-------. 1 root   root   23 Mar  8 05:18 last-via-server

-rw-------. 1 root   root   48 Jul 25 08:00 last-ccpp

drwxr-x---. 2 abrt   abrt 4096 Jul 28 15:22 ccpp-2019-08-21-13:59:02-31929

Read [11] for all the details of ABRT configuration files.

References

1. ABRT Documentation (Release 2.14)
2. How to properly delete a report problem in ABRT
3. AUTOMATIC BUG REPORTING TOOL (ABRT)
4. Elements collected by ABRT
5. Basic ABRT components
6. abrt-action-analyze-backtrace
• Analyzes C/C++ backtrace, generates duplication hash, backtrace rating, and identifies crash function in problem directory DIR
• Then it saves this data as new elements global_uuid, rating, crash_function in this problem directory
7. abrt-backtrace
8. ABRT FAQ
9. ABRT Design
10. backtrace_rating (Red Hat doc)
• Numerical representation of quality of backtrace based on ratio of unrecognized frames among all backtrace frames
11. ABRT SPECIFIC CONFIGURATION