The essentials

Quick reference

One focused task per row. Jump to the related section for complete, working examples.

UseSyntaxExamples
Measure one commandperf stat -- sleep 10View examples
Measure one processsudo perf stat --pid 1234 -- sleep 10View examples
List available perf eventsperf listView examples
List dynamic probessudo perf probe --listView examples
Record process call graphssudo perf record --call-graph dwarf --pid 1234 -- sleep \ 30View examples
Open a perf reportperf report --stdio --input perf.dataView examples
Profile one process livesudo perf top --pid 1234View examples
Trace one process syscallssudo timeout 10 perf trace --pid 1234View examples
Locate tracefsfindmnt -t tracefs -o TARGET,SOURCE,OPTIONSView examples
List ftrace tracerssudo cat /sys/kernel/tracing/available_tracersView examples
List tracepointssudo cat /sys/kernel/tracing/available_eventsView examples
Stream ftrace outputsudo cat /sys/kernel/tracing/trace_pipeView examples
Clear the trace buffersudo sh -c ': > /sys/kernel/tracing/trace'View examples
Disable global tracingsudo sh -c 'echo 0 > /sys/kernel/tracing/tracing_on'View examples
Probe BPF supportsudo bpftool feature probe kernelView examples
List loaded BPF programssudo bpftool prog showView examples
List BPF mapssudo bpftool map showView examples
List matching probessudo bpftrace --list 'tracepoint:sched:sched_process_*'View examples
Count process executionssudo timeout 30 bpftrace -e \ 'tracepoint:sched:sched_process_exec { @[comm] = count(); }'View examples
Show bpftrace environmentbpftrace --infoView examples
Check kernel lockdowncat /sys/kernel/security/lockdownView examples
Read perf restrictionssysctl kernel.perf_event_paranoid kernel.kptr_restrictView examples
Inspect profile artifactstat --format='%a %U:%G %s %n' perf.dataView examples

Kernel observability tools can reveal why a system is slow, but they also consume CPU and memory, expose arguments and stack data, and can destabilize production when probes are broad or recursive. Start with counters, scope by PID, cgroup, CPU, event, and duration, estimate cardinality, and arrange automatic cleanup. Kernel version, configuration, lockdown, BTF, symbols, privileges, and container boundaries determine what is available.

Step by step

Detailed examples

01

Begin with low-overhead counters

perf stat answers whether a workload is CPU-bound, stalled, switching, faulting, or migrating before sampling stacks. Scope to a command, PID, cgroup, or CPU and choose a short interval. Multiplexing occurs when requested events exceed hardware counters, reducing precision; review time-enabled and time-running data.

Measure a bounded process window
perf stat -- sleep 10
# Attach only with workload-owner approval:
# sudo perf stat --pid 1234 -- sleep 10
# Record the perf and kernel versions with results:
perf --version
uname -r
Back to quick reference ↑
02

Discover events on the target kernel

PMU events vary by CPU, tracepoints vary by kernel and modules, and probe symbols depend on debugging information and optimization. Never assume a copied event exists or means the same thing. Dynamic probes are shared kernel state; inventory existing probes before adding or deleting any.

Inspect available sources
perf list
sudo cat /sys/kernel/tracing/available_events
sudo perf probe --list
bpftrace --info
Back to quick reference ↑
03

Bound sample frequency, call graphs, scope, and duration

High-frequency sampling and DWARF stack unwinding can materially slow workloads and generate large perf.data files. Start at a modest rate, one PID or cgroup, short duration, and a storage quota. Frame pointers are cheaper when binaries preserve them; DWARF can recover more user stacks at higher cost.

Review a scoped recording
# Confirm disk headroom and approval first.
# sudo perf record --frequency 99 --call-graph dwarf --pid 1234 -- sleep 30
# perf report --stdio --input perf.data
df -h .
ulimit -a
Back to quick reference ↑
04

Use private trace instances instead of global controls

The top-level tracefs controls and buffer are shared, so changing current_tracer, events, filters, or tracing_on can disrupt another investigation. Trace instances provide separation where supported. Set filters before enabling, use bounded readers, and understand that trace_pipe consumes records while trace is a snapshot.

Inspect ftrace without altering state
findmnt -t tracefs -o TARGET,SOURCE,OPTIONS
sudo cat /sys/kernel/tracing/current_tracer
sudo cat /sys/kernel/tracing/tracing_on
sudo cat /sys/kernel/tracing/available_tracers
sudo ls /sys/kernel/tracing/instances
Back to quick reference ↑
05

Inventory kernel, BTF, programs, maps, and pins

Loaded BPF objects can be owned by networking, security, observability, and orchestration agents. Do not detach or delete by ID merely because an object is unfamiliar. BTF and helper availability affect portability; map memory and pinned objects can outlive the launching process depending on references.

Capture read-only BPF inventory
bpftrace --info
sudo bpftool feature probe kernel
sudo bpftool prog show
sudo bpftool map show
sudo bpftool link show
sudo find /sys/fs/bpf -maxdepth 3 -print
Back to quick reference ↑
06

Control probe fan-out and map cardinality

A one-liner can attach to thousands of probes or create a map key per PID, stack, path, address, or user, consuming unbounded memory. List matches first, constrain predicates, aggregate low-cardinality fields, attach for a fixed duration, and handle SIGTERM cleanup. Reading string arguments may fail or expose secrets.

Run a bounded low-cardinality count
sudo bpftrace --list 'tracepoint:sched:sched_process_exec'
# Approved 30-second aggregate:
# sudo timeout 30 bpftrace -e 'tracepoint:sched:sched_process_exec { @[comm] = count(); }'
Back to quick reference ↑
07

Do not weaken host-wide security controls for convenience

perf_event_paranoid, kptr_restrict, lockdown, LSM policy, CAP_PERFMON, CAP_BPF, and CAP_SYS_ADMIN influence access. Lowering sysctls or granting broad capabilities expands observation or kernel attack surface for every affected process. Prefer a tightly controlled tracing service or short privileged session and restore policy afterward.

Audit tracing access controls
sysctl kernel.perf_event_paranoid kernel.kptr_restrict
cat /sys/kernel/security/lockdown 2>/dev/null
grep -E '^Cap(Eff|Bnd|Amb):' /proc/self/status
mount | grep -E 'tracefs|bpf'
Back to quick reference ↑
08

Plan cleanup before attaching probes

Interrupted tools usually detach temporary probes, but crashes, pinned objects, dynamic probes, trace instances, or manually enabled events can remain. Record the pre-change state and object owner, use a timeout, and remove only artifacts created by the session. Clearing global buffers destroys other operators' evidence.

Compare state after a session
sudo perf probe --list
sudo bpftool prog show
sudo bpftool link show
sudo find /sys/fs/bpf -maxdepth 3 -print
sudo cat /sys/kernel/tracing/tracing_on
sudo cat /sys/kernel/tracing/set_event
Back to quick reference ↑
09

Protect symbols, stacks, paths, and trace records

Profiles can reveal proprietary symbol names, code layout, command names, filenames, user identifiers, network endpoints, and arguments. perf.data and map dumps are production data. Restrict file permissions, minimize captured fields, use approved retention, and share redacted reports rather than raw traces.

Review artifact exposure
stat perf.data 2>/dev/null
perf buildid-list --input perf.data 2>/dev/null
# Store raw profiles in an access-controlled incident directory.
# Generate a filtered report for wider sharing.
Back to quick reference ↑

Keep exploring

More Linux cheat sheets

Linux · Containers, Virtualization, and Performance16 quick references

cgroup v2 Resource Control Cheat Sheet

Identify cgroup v2, apply CPU, memory, I/O, and task controls with systemd, inspect pressure and limit events, and understand safe hierarchy and delegation rules.

Open cheat sheet
Linux · Containers, Virtualization, and Performance20 quick references

Containers with Podman Cheat Sheet

Run and operate rootless Podman containers with secure defaults, durable storage, health checks, Quadlet services, secrets, updates, and recoverable backups.

Open cheat sheet

Sources and further reading

References

Authoritative documentation used to verify and expand this cheat sheet.

  1. Linux Kernel Projectperf: Linux Profiling with Performance Countersperf.wiki.kernel.org
  2. Linux Kernel Projectperf-stat(1) Manual Pageman7.org
  3. Linux Kernel Projectftrace: Function Tracerkernel.org
  4. Linux Kernel ProjectBPF Documentationkernel.org
  5. Linux Kernel Projectbpftool Documentationkernel.org
  6. bpftrace Projectbpftrace Reference Guidebpftrace.org

Help us improve

Found a typo or missing example?

Tell us what would make this cheat sheet clearer, more complete, or more useful.

Share feedback