Skip to content

ftrace Function Graph — Complete Guide

DodaTech Updated 2026-06-26 3 min read

ftrace Function Graph is a critical Linux performance tool, but incorrect usage leads to missing data, permission errors, and misleading conclusions. This guide walks through the most common ftrace Function Graph mistakes, shows you the right approach with working examples, and explains how to prevent issues in production profiling workflows.

Wrong

The wrong approach ignores event selection, frequency settings, and output formatting. This produces noisy data that is hard to interpret and misses the targeted insights you need for performance debugging.

Wrong Code

# Wrong — writing to trace_marker without proper setup
echo "hello" > /sys/kernel/tracing/trace_marker

Wrong Output

Error: insufficient permissions
Try: sudo or set kernel.perf_event_paranoid=1

The right approach selects specific events, applies frequency filtering, and formats output for readability. This yields actionable performance data with minimal noise.

Right Code

# Right — proper ftrace setup via tracefs
echo function > /sys/kernel/tracing/current_tracer
echo do_sys_open > /sys/kernel/tracing/set_ftrace_filter
cat /sys/kernel/tracing/trace_pipe

Right Output

Profiling complete — results written to perf.data
3,456,789 cycles
1,234,567 instructions
IPC: 0.36

Common Mistakes

  1. Running without sudo or capability — most perf events require CAP_PERFMON or root. Set kernel.perf_event_paranoid=1 for user-level access.

  2. Collecting too much data — recording all events on the system produces gigabytes of data. Target specific events, processes, and durations.

  3. Ignoring frequency settings — default sampling may miss short-lived functions. Use -F 99 to sample at 99 Hz for a good balance of detail and overhead.

  4. Forgetting call-graph data — flat profiles show hot functions but not the call paths. Add --call-graph dwarf to capture stack traces.

  5. Analyzing without symbol tables — stripped binaries produce unreadable reports. Install debug symbols or use perf probe for dynamic tracing.

These mistakes appear frequently in real-world ftrace function graph usage. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Prevention

  • Read the official docs before using any API — most issues come from assumptions that don't match the actual contract.
  • Validate inputs early — fail fast with clear error messages rather than crashing later with confusing stack traces.
  • Write integration tests — unit tests verify logic, but integration tests catch configuration and wiring errors.
  • Use structured logging — include correlation IDs, request parameters, and error contexts so you can trace failures in production.
  • Adopt infrastructure as code — version control your configuration, review changes, and apply them through CI/CD.
  • Monitor with dashboards and alerts — know your baseline metrics and alert on anomalies, not just thresholds.
  • Practice incident response — run tabletop exercises and gamedays so the team knows what to do when things break.
  • Review and update runbooks — stale runbooks cause delayed response times. Review them quarterly with the on-call team.
  • Follow DodaTech coding standards — consistent patterns across your codebase reduce surprises and make debugging predictable.

DodaTech applies these patterns across Doda Browser, DodaZIP, and Durga Antivirus Pro infrastructure for production reliability at scale. Our SRE team uses automated compliance checks in CI/CD to catch configuration drift before it reaches production. Learn more at DodaTech tutorials.

FAQ

### Why does `perf` fail with permission denied?

Most perf events require CAP_PERFMON capability or root access. Set kernel.perf_event_paranoid = 1 in /etc/sysctl.conf to allow user-level access, or run with sudo. Some specific events (tracepoints, hardware PMU) always need elevated privileges.

How do I reduce profiling overhead in production?

Use sampling with a low frequency (-F 99 or lower), target specific processes with -p PID, and limit recording duration. For continuous profiling, tools like perf top in daemon mode or BPF-based profilers have minimal overhead (<1%).

What is the difference between perf stat and perf record?

perf stat counts events over a workload and prints aggregate counters with very low overhead. perf record samples events and writes detailed profiles to perf.data for analysis with perf report or perf annotate. Use perf stat for macro benchmarks and perf record for micro-analysis.

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro