ABRT
ABRT consists of the abrtd daemon and a number of system services and utilities to process, analyze, and report detected problems.
The daemon runs silently in the background most of the time, and springs into action when an application crashes or a kernel oops is detected. The daemon then collects the relevant problem data such as a core file if there is one, the crashing application's command-line parameters, and other data of forensic utility.
Why ABRT?
Earlier when applications crashed, core dumps were generated, but not limited, which could quickly fill up the disk.
A solution is to use ABRT. For example, it can
- Rotate cores within a size limit by deleting the oldest[11]
- Avoid storing duplicate crashes by deduplication[9]
Elements Collected by ABRT
In the below table, it shows a shortened list of elements collected by ABRT and their descriptions. For a full list see [4]. These elements are stored in the form of files in a single directory per detected problem (such a directory is called 'dump directory').
| core_backtrace | Machine readable backtrace with no private data |
| coredump | Coredump of the crashing process |
| count | Number of times this problem occured |
| crash_function | Function which crashed |
| dmesg | Copy of dmesg |
| docker_inspect | Output of docker inspect $(container_id) |
| dso_list | List of dynamic libraries loaded at the time of crash |
| duphash | Hash of the crash's backtrace |
| environ | Dump of process environment variable along with their values |
| event_log | Messages produced by ABRT tools during processing the detected problem |
| executable | Executable path of the component which caused the problem. |
| global_pid | Value of %P as passed by kernel to the core_pattern helper (see man core for more details) |
| hostname | Hostname of the affected machine |
| kernel | Kernel version string |
| kernel_log | Results of vmcore crash analysis performed by retrace-server |
| kernel_tainted_long | Tainted kernel description |
| kernel_tainted_short | Kernel tainted flags (For more information about tainted flags see [1]) |
| last_occurrence | Time of the last occurence (unixtime) |
Deduplication
When ABRT catches new crash, it compares it to the rest of the stored problems to avoid storing duplicate crashes:
- It first checks if there is core_bactrace or uuid item in the problem directory it is processing
- If there is a core_backtrace
- It iterates over all other dump directories and computes similarity to their core backtraces (if any). If one of them is similar enough to be considered duplicate, event processing is stopped and only notify-dup event is fired.
- Or if there is an uuid item (and no core backtrace)
- Simple comparison of uuid hashes is used for duplicate detection.
You can read abrt-action-analyze-backtrace for more information.[6]
count & last_occurrence
After the forensic investigation, you can use:
- abrt-cli rm <path to the problem directory>
to remove the specified problem data directory with all its contents.
[abrt]# abrt-cli rm ccpp-2019-08-21-13:59:02-31929
PrivateReports is disabled. Run abrt-cli-root to see all problems detected by ABRT.
rm 'ccpp-2019-08-21-13:59:02-31929'
However, note that ABRT performs a detection of duplicate problems by comparing new problems with all locally saved problems.
For a repeating crash, ABRT requires you to act upon it only once. But, if you delete the crash dump of that problem, the next time this specific problem occurs, ABRT will treat it as a new crash: ABRT will alert you about it, prompt you to fill in a description, and report it. To avoid having ABRT notifying you about a recurring problem, do not delete its problem data.
If you didn't remove a specific problem data directory, here is what would happen when ABRT catches a new crash :
- ABRT compares it to the rest of locally stored problems
- If it's a new problem, a new problem directory will be created
- Otherwise, ABRT will update the recurring problem by:
- Incrementing "count" by one
- Updating "last_occurrence" with a new epoch
[ccpp-2019-08-21-13:59:02-31929]# ls -lrt
total 868572
-rw-r-----. 1 abrt abrt 3 Aug 21 2019 uid
-rw-r-----. 1 abrt abrt 10 Aug 21 2019 time
-rw-r-----. 1 abrt abrt 32 Aug 21 2019 os_release
-rw-r-----. 1 abrt abrt 30 Aug 21 2019 kernel
-rw-r-----. 1 abrt abrt 24 Aug 21 2019 hostname
-rw-r-----. 1 abrt abrt 6 Aug 21 2019 architecture
-rw-r-----. 1 abrt abrt 70033 Aug 21 2019 maps
-rw-r-----. 1 abrt abrt 1323 Aug 21 2019 limits
-rw-r-----. 1 abrt abrt 88 Aug 21 2019 cgroup
-rw-r-----. 1 abrt abrt 4 Aug 21 2019 type
-rw-r-----. 1 abrt abrt 90 Aug 21 2019 reason
-rw-r-----. 1 abrt abrt 39 Aug 21 2019 pwd
-rw-r-----. 1 abrt abrt 5 Aug 21 2019 pid
-rw-r-----. 1 abrt abrt 2072 Aug 21 2019 open_fds
-rw-r-----. 1 abrt abrt 48 Aug 21 2019 executable
-rw-r-----. 1 abrt abrt 14722 Aug 21 2019 environ
-rw-r-----. 1 abrt abrt 48 Aug 21 2019 cmdline
-rw-r-----. 1 abrt abrt 4 Aug 21 2019 analyzer
-rw-r-----. 1 abrt abrt 5 Aug 21 2019 abrt_version
-rw-r-----. 1 abrt abrt 886996992 Aug 21 2019 coredump
-rw-r-----. 1 abrt abrt 7 Aug 21 2019 username
-rw-r-----. 1 abrt abrt 1846076 Aug 21 2019 sosreport.tar.xz
-rw-r-----. 1 abrt abrt 0 Aug 21 2019 event_log
-rw-r-----. 1 abrt abrt 93 Aug 21 2019 machineid
-rw-r-----. 1 abrt abrt 378414 Aug 21 2019 core_backtrace
-rw-r-----. 1 abrt abrt 40 Aug 21 2019 uuid
-rw-r-----. 1 abrt abrt 1424 Aug 21 2019 dso_list
-rw-r-----. 1 abrt abrt 199 Aug 21 2019 var_log_messages
-rw-r-----. 1 abrt abrt 2 Jul 25 08:00 count
-rw-r-----. 1 abrt abrt 10 Jul 25 08:00 last_occurrence
[ccpp-2019-08-21-13:59:02-31929]# cat count
2
[ccpp-2019-08-21-13:59:02-31929]# cat last_occurrence
1595664006
[ccpp-2019-08-21-13:59:02-31929]# date -u -d @1595664006
Sat Jul 25 08:00:06 UTC 2020
[ccpp-2019-08-21-13:59:02-31929]# cat reason
Process /u01/app/xxx/server/bin/yyy was killed by signal 11 (SIGSEGV)
ABRT Configuration Files
- /etc/abrt/abrt.conf — allows you to modify the behavior of the abrtd service.
- /etc/abrt/abrt-action-save-package-data.conf — allows you to modify the behavior of the abrt-action-save-package-data program.
- /etc/abrt/plugins/CCpp.conf — allows you to modify the behavior of ABRT's core catching hook.
For example, the default location where problem data directories are created and in which problem core dumps and all other problem data are stored is:
/var/spool/abrt
[~]# cd /var/spool/abrt
[abrt]# ls -lrt
total 32
-rw-------. 1 root root 23 Mar 8 05:18 last-via-server
-rw-------. 1 root root 48 Jul 25 08:00 last-ccpp
drwxr-x---. 2 abrt abrt 4096 Jul 28 15:22 ccpp-2019-08-21-13:59:02-31929
Read [11] for all the details of ABRT configuration files.
References
- ABRT Documentation (Release 2.14)
- How to properly delete a report problem in ABRT
- AUTOMATIC BUG REPORTING TOOL (ABRT)
- Elements collected by ABRT
- Basic ABRT components
- abrt-action-analyze-backtrace
- Analyzes C/C++ backtrace, generates duplication hash, backtrace rating, and identifies crash function in problem directory DIR
- Then it saves this data as new elements global_uuid, rating, crash_function in this problem directory
- abrt-backtrace
- ABRT FAQ
- ABRT Design
- backtrace_rating (Red Hat doc)
- Numerical representation of quality of backtrace based on ratio of unrecognized frames among all backtrace frames
- ABRT SPECIFIC CONFIGURATION
