Debugging Core Dump Files on Linux - A Detailed Guide

Core dumps play a key role in debugging programs that exit abnormally. They preserve the state of a program at failure, and with them, programmers can view and identify the causes of failures. In this article, a walkthrough is taken through a step-by-step exercise of enabling, creating, and checking out core dumps in Linux, and touches on high-end tools and techniques for debugging sophisticated failures, enabling quick diagnoses and resolution.

1. Enabling Core Dumps in Linux

Check and Set ulimit

1. Check for current value for core dumps:

   Shell

   ulimit -c

   
   

   Output:

   Shell

   0

   
   

   A value of 0 informs us that core dumps have been disabled..

2. Enable core dumps temporarily:

   Shell

   ulimit -c unlimited

   
   

   Check again:

   Shell

   ulimit -c

   
   

   Output:

   Shell

   unlimited

   
   

3. To make it persistent, add the following in /etc/security/limits.conf:

   Shell

   * soft core unlimited
   *    hard    core    unlimited

   
   

4. Configure location for core dumps:

   Shell

   sudo sysctl -w kernel.core_pattern=/var/dumps/core.%e.%p

   
   

   • %e: Program name

   • %p: Process ID

5. Make it persistent: Add the following in  /etc/sysctl.conf:

   Shell

   kernel.core_pattern=/var/dumps/core.%e.%p

   
   

   Reload configuration:

   Shell

   sudo sysctl -p

   
   

2. Generate Core Dumps for Testing

C Program to cause Segfault

Create a testing program:

#include <stdio.h>
int main() {
 int *ptr = NULL; // Null pointer
 *ptr = 1; // Segmentation fault
 return 0;
}

Make a build of it:

gcc -g -o crash_test crash_test.c

Execute the program:

./crash_test

Output:

Segmentation fault (core dumped)

Check for location of the core dump:

ls /var/dumps

Example:

core.crash_test.12345

3. Analyze with GDB

Load the Core Dump

gdb ./crash_test /var/dumps/core.crash_test.12345

Basic Analysis

Get Backtrace

bt

Output:

#0 0x0000000000401132 in main () at crash_test.c:4
4       *ptr = 1;

The backtrace identifies that the crash happened at line 4.

Examine Variables

info locals

Output:

ptr = (int *) 0x0

The variable ptr is a null pointer, and a segmentation fault is confirmed.

Disassemble the Code

disassemble main

Output:

Dump of assembler code for function main:
 0x000000000040112a <+0>: mov %rsp,%rbp
 0x000000000040112d <+3>: movl $0x0,-0x4(%rbp)
   0x0000000000401134 <+10>: movl   $0x1,(%rax)

Displays actual assembly instruction at location of crash.

Check Registers

info registers

Output:

rax 0x0 0
rbx 0x0 0
rcx            0x0      0

Register rax is 0, showing the null pointer dereference.

4. Debugging Multithreaded Applications

Check Threads

info threads

Output:

 Id Target Id Frame
* 1    Thread 0x7f64c56 (LWP 12345) "crash_test" main () at crash_test.c:4

Switch to a Specific Thread

thread 1

Get Backtrace for All Threads

thread apply all bt

5. Using Advanced Tools

Valgrind – Analysis of Memory Issue

valgrind --tool=memcheck ./crash_test

Output:

Invalid write of size 4
 at 0x401132: main (crash_test.c:4)
 Address 0x0 is not stack'd, malloc'd or (recently) free'd

Confirms an invalid access in memory.

ELFutils for Symbol Inspection

eu-readelf -a /var/dumps/core.crash_test.12345

Output:

Program Headers:
  LOAD 0x000000 0x004000 0x004000 0x1234 bytes

Displays sections and symbol information in the core file.

Crash Utility for Kernel Dumps

sudo crash /usr/lib/debug/vmlinux /var/crash/core

Use for kernel-space core dumps.

Generate Core Dumps for Running Processes

gcore <PID>

6. Debugging Specific Issues

Segmentation Faults

info frame
x/16xw $sp

Check near stack pointer in memory.

Heap Corruption

Check for heap corruption with Valgrind or AddressSanitizer:

gcc -fsanitize=address -g -o crash_test crash_test.c
./crash_test

Shared Libraries Mismatch

info shared
ldd ./crash_test

Check shared libraries loaded in a proper manner.

7. Best Practices for Core Dump Debugging

• Save Symbols Independently: Deploy stripped binaries for production and store debug symbols in a secure manner.
• Automate Dump: Employ systemd-coredump for efficient dump management.
• Analyze Logs: Enable full application logs for tracking run-time faults.
• Redact Sensitive Information: Remove sensitive information in shared core dumps.
• Test in Debugging: Employ debug build with full symbols for in-depth debugging.

8. Conclusion

Debugging a core dump in Linux is a logical progression and utilizes tools such as GDB, Valgrind, and Crash Utility. By careful examination of backtraces, state of memory, and register values, developers can pinpoint root causes and remedy them in no time. Best practices will yield more efficient diagnostics and rapid resolution for production-critical environments.