The essentials
Quick reference
One focused task per row. Jump to the related section for complete, working examples.
| Use | Syntax | Examples |
|---|---|---|
| List recent core dumps | coredumpctl list --since today --no-pager | View examples |
| Inspect one crash | coredumpctl info 4242 --no-pager | View examples |
| Filter by executable | coredumpctl list COREDUMP_EXE=/usr/local/bin/app \
--no-pager | View examples |
| Extract a core safely | umask 077; coredumpctl dump 4242 --output=app.core | View examples |
| Open latest matching dump | coredumpctl debug /usr/local/bin/app | View examples |
| Produce a batch backtrace | gdb --nx --batch --quiet -ex 'thread apply all bt full' \
/usr/local/bin/app app.core | View examples |
| Inspect core resource limit | ulimit -c | View examples |
| Inspect kernel core handler | sysctl kernel.core_pattern | View examples |
| Trace a new command | strace -f -tt -T -s 256 -o trace.log -- \
/usr/local/bin/app --check | View examples |
| Trace file and network calls | strace -f -e trace=%file,%network -o trace.log -- \
/usr/local/bin/app --check | View examples |
| Attach to a process | sudo strace -f -p 4242 -e trace=%file -o trace.log | View examples |
| Summarize system calls | strace -f -c -- /usr/local/bin/app --check | View examples |
| Read a kernel stack | sudo cat /proc/4242/stack | View examples |
| Inspect process status | cat /proc/4242/status | View examples |
| Record mapped objects | cat /proc/4242/maps | View examples |
| Read an ELF build ID | readelf -n /usr/local/bin/app | sed -n '/Build ID/p' | View examples |
Start with the least invasive evidence: service status, journal records, process identity, and a narrow system-call trace. Core images can contain credentials, customer data, encryption keys, and complete process memory, so restrict access and retention. Tool availability, ptrace policy, core handlers, and package names vary by distribution; reproduce on a non-production host whenever possible.
Step by step
Detailed examples
Triage crash metadata before handling process memory
coredumpctl queries journal metadata recorded by systemd-coredump. Match an executable path, boot, time range, or journal field rather than assuming a reused PID is unique. A listed dump can be absent, truncated, inaccessible, or already expired; journal retention and external core-file retention are independent. Unprivileged users normally see only permitted records.
coredumpctl list --since '2026-08-12 00:00:00' --no-pager
coredumpctl info COREDUMP_EXE=/usr/local/bin/app --no-pager
journalctl _EXE=/usr/local/bin/app --since today --no-pagerProtect dumps and match the exact executable and symbols
A core is a snapshot of process address space and can expose secrets. Extract it only to an access-controlled, encrypted location approved by policy, never attach it to an unrestricted ticket, and securely retire it according to retention requirements. Use the exact unstripped binary, shared libraries, and debug symbols from the crashed build; package names for debuginfo vary by distribution and containers may require their own root filesystem. Disable initialization files when opening an artifact that is not fully trusted.
umask 077
coredumpctl dump 4242 --output=app.core
readelf -n /usr/local/bin/app | sed -n '/Build ID/p'
gdb --nx --batch --quiet -ex 'set pagination off' -ex 'thread apply all bt full' /usr/local/bin/app app.coreUnderstand limits, handlers, and systemd service policy
Core creation depends on the signal, dumpable state, resource limits, filesystem access, kernel configuration, and core_pattern. A leading pipe in core_pattern invokes a handler instead of writing a local file. For systemd services, LimitCORE= controls the service limit, while coredump.conf controls processing and storage. Changing global capture policy requires privilege, can consume substantial disk, and affects future crashes only; distribution defaults differ.
ulimit -Sc
ulimit -Hc
sysctl kernel.core_pattern fs.suid_dumpable
systemctl show app.service -p LimitCORE
systemd-analyze cat-config systemd/coredump.confTrace a reproducible command with a narrow syscall set
strace observes the kernel interface, which is ideal for permission, path, socket, and missing-file failures but not application-level intent. Trace a short reproduction, follow children when needed, and filter classes to limit overhead. Trace files can reveal arguments, paths, payload fragments, environment-derived data, and network endpoints; create them with restrictive permissions and redact before sharing.
umask 077
strace -f -tt -T -s 256 -e trace=%file,%network -o trace.log -- /usr/local/bin/app --check
grep -E 'ENOENT|EACCES|ECONNREFUSED' trace.logAttach to production only with a rollback and latency budget
Attaching requires permission governed by credentials, capabilities, Yama ptrace_scope, namespaces, and security policy. strace stops tasks briefly on syscall events and high-rate workloads can suffer severe latency. Prefer a summary or narrow filter, coordinate with the service owner, set a short observation window externally, and stop strace to detach. Never weaken system-wide ptrace policy merely to bypass an unexplained denial.
sudo strace -f -p 4242 -e trace=%file -s 128 -o trace.log
# Press Ctrl-C in the strace terminal to detach; confirm the service remains healthy.
systemctl status app.service --no-pagerUse procfs and logs for lower-impact live evidence
procfs exposes current state without retaining all syscall arguments. Capture status, limits, descriptors, maps, and task state, but remember that a process can exit or change between reads. Reading another user's process usually needs privilege and may still be restricted. File descriptors and environment data can disclose secrets, so avoid copying /proc/PID/environ into routine diagnostics.
pid=4242
cat /proc/$pid/status
cat /proc/$pid/limits
ls -l /proc/$pid/fd
cat /proc/$pid/mapsCorrelate addresses with immutable build artifacts
Optimized code, stripped symbols, inlining, and mismatched libraries make plausible backtraces misleading. Record build IDs and package/container image identity at incident time, then load matching symbols in a controlled analysis environment. Rebuilding from the same source is not necessarily binary-identical. gdb can execute initialization scripts, so analyze untrusted artifacts with safe initialization settings and isolation.
readelf -n /usr/local/bin/app | sed -n '/Build ID/p'
sha256sum /usr/local/bin/app
cat /proc/4242/maps
command -v rpm >/dev/null && rpm -qf /usr/local/bin/app
command -v dpkg-query >/dev/null && dpkg-query -S /usr/local/bin/appSources and further reading
References
Authoritative documentation used to verify and expand this cheat sheet.
Help us improve
Found a typo or missing example?
Tell us what would make this cheat sheet clearer, more complete, or more useful.