Enable frame pointers on Linux for the perf profiler

[adamchainz]

Python supports tracing with Linux's perf, and "for best results":

Python should be compiled with CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"

The first of these options was previously discussed in #239 which stalled due to Python deprecating dttace support, but perf support is here to stay.

[jjhelmus]

Enabling frame pointers typically has a small effect on performance. There is a trade off, performance vs robust profiling information.

It would be good to run some performance benchmarks to see how large of effect adding frame pointers has.

[geofft]

So the linked article does give a way to make use of profiling on a build without frame pointers, but the requirement is that Python's JIT needs to dynamically generate fake debug information for perf to capture, and that generation has overhead.

As I understand it, perf needs either frame pointers or debuginfo to meaningfully figure out the call stack (see also Brendan Gregg's article). If you don't have debuginfo at all for some reason (which I guess is common for distro-packaged software, but distros usually split those off into separate debug packages instead of just throwing them away?), then frame pointers are absolutely necessary. If you have debuginfo, then perf record needs to snapshot the whole stack and perf script/perf report will need to evaluate the DWARF debuginfo to turn that stack into a list of function calls. If you have frame pointers, then perf record can walk the frame pointers directly. So this means that the benefit of frame pointers compared to debuginfo is a smaller perf.data file and a faster analysis step (after capturing the data), but strictly speaking there's nothing you can do with frame pointers that you can't also do with just debuginfo. So in theory, a binary with debuginfo and without frame pointers (which is what we do currently) gets you better runtime performance with no loss of functionality.

However, when there's a JIT (like Python has) then it's impossible to generate debuginfo ahead of time for the JITted code, whereas I guess you effectively generate the code to use frame pointers ahead of time (because the way Python's JIT works is it assembles code out of "JIT stencils" that are compiled as part of compiling Python itself). So that imposes a runtime overhead to generate DWARF debuginfo for perf to use, and it looks like that overhead is significant. Using the last version of pystone.py that was in the CPython repo (which is not a very good benchmark, but it suffices) on an AWS t3.large:

$ hyperfine 'uvx python3.14 '{,-Xperf_jit}' pystone.py 1000000'
Benchmark 1: uvx python3.14  pystone.py 1000000
  Time (mean ± σ):      2.080 s ±  0.058 s    [User: 2.059 s, System: 0.021 s]
  Range (min … max):    2.030 s …  2.202 s    10 runs
 
Benchmark 2: uvx python3.14 -Xperf_jit pystone.py 1000000
  Time (mean ± σ):      2.960 s ±  0.120 s    [User: 2.941 s, System: 0.017 s]
  Range (min … max):    2.850 s …  3.220 s    10 runs
 
Summary
  uvx python3.14  pystone.py 1000000 ran
    1.42 ± 0.07 times faster than uvx python3.14 -Xperf_jit pystone.py 1000000

If this were a smaller overhead, I would very much like to say that we should avoid imposing the overhead of frame pointers on everyone, and people who are interested in benchmarking can take the overhead when they're benchmarking. But I'm worried a 42% performance impact is too much for that to be a workable answer. @adamchainz, I'm curious what you think about this.

So I think I'm okay with turning on frame pointers for now, and trying to figure out if there's something else we can do later (e.g., can we turn on frame pointers just for the JIT stencils and get perf to look at debuginfo for the rest of it? can we distribute frame-pointer-enabled builds for a couple of configurations without regretting what that does to our CI matrix? etc.).

[adamchainz]

the requirement is that Python's JIT needs to dynamically generate fake debug information for perf to capture

To be clear, that isn't “Python’s JIT” (the one that makes it go faster) but “Python’s perf JIT” (the one that exposes the stack info to perf), as implemented in python/cpython#112254 .

But yes, it does have overhead, and thank you for measuring that.

---

If this were a smaller overhead, I would very much like to say that we should avoid imposing the overhead of frame pointers on everyone, and people who are interested in benchmarking can take the overhead when they're benchmarking. But I'm worried a 42% performance impact is too much for that to be a workable answer. @adamchainz, I'm curious what you think about this.

Honestly, I think it's acceptable to avoid any overhaed for the majority of usage.

I don't have a strong opinion here - I opened the issue to take the discussion out of a private Slack channel and into the open.

[jjhelmus]

In February, I ran the pyperformance suite against python-build-standalone built with and without frame pointers. The results showed an overall ~2% performance loss when frame pointers were added. Individual benchmarks varied from 0% to 4%.

If this loss in performance is a reasonable trade off for better monitoring depends on the use case.

I can't find the results to share, only the summary I wrote in my notes. I will re-run the tests this week to get fresh results.

[jjhelmus]

@pablogsal may be interested in this discussion. The topic is if python-build-standalone, the CPython that uv can install, should include frame pointers for better monitoring and profile results.

[pablogsal]

I want to clarify something that seems to be causing confusion here: the 42% overhead measured by @geofft is not the cost of frame pointers: it's the cost of -Xperf_jit, which are two completely separate mechanisms.

Frame pointers (-fno-omit-frame-pointer) are a compile-time option that keeps the frame pointer register populated at all times, allowing tools like perf to walk the call stack cheaply without any runtime overhead beyond the register usage itself. This is what this issue is about, and @jjhelmus's pyperformance results (~2% overhead) are the relevant data point here.

*ñ-Xperf_jit is an entirely different feature: it makes Python's interpreter dynamically emit DWARF debug information at runtime so that perf can symbolicate the interpreter's eval loop and JIT-compiled code. That runtime DWARF emission is what causes the 42% overhead, and it only applies when you explicitly pass -Xperf_jit. Nobody pays that cost unless they opt into it. And yes, a ~42% overhead would obviously be too much to enable by default, but again, that's not what's being proposed here. If anything frame pointers avoids the need for that 42%.

Beyond perf, frame pointers are genuinely beneficial for a wide range of tooling: system profilers (perf, pprof, py-spy), debuggers, crash reporters, eBPF-based tracers, continuous profilers like Datadog or Pyroscope all of them work significantly better or become outright more reliable with frame pointers present. The benefits go well beyond just profiling.

This is also why major Linux distributions are moving to enable frame pointers by default across their entire package ecosystem. Fedora has been shipping with frame pointers enabled in recent Fedora, and Ubuntu. The consensus in the broader Linux world is that the ~1-2% overhead is a worthwhile trade-off for the observability and debuggability gains.

The real trade-off for this issue is therefore: ~2% performance overhead in exchange for frame pointers always being availabl no flags needed, no runtime cost, and the full ecosystem of profiling and debugging tools just works out of the box.

[pablogsal]

Just to be clear: I think is very much worth it and we should absolutely do it

[Fidget-Spinner]

In February, I ran the pyperformance suite against python-build-standalone built with and without frame pointers. The results showed an overall ~2% performance loss when frame pointers were added. Individual benchmarks varied from 0% to 4%.

That's actually incredibly good. It used to be more back in the day. Like 0% to 9% slowdown. Pablo discussed this with me before and we suspect the tail calling interpreter might've paid off some of the frame pointer cost/solved some of the original issues in the linked thread. If that's actually the case, then the cost of FP might be less on 3.14+ where TC is enabled. This may complicate things, where the cost of FP is a lot less on 3.14+ than it is on 3.13-, so you might want to fastmark it. If you want a quick and dirty benchmark, I now recommend Sam Gross's fastmark, which runs a subset of pyperformance very quickly (< 2 mins on my system).

If you can share the actual individual benchmarks, it's possible to comment if the slowdown is on a microbenchmark or macro one. 4% slowdown on say, nbody, is unnoticeable in real-world workloads, but 4% on the FastAPI benchmark is probably something worth thinking about.

[geofft]

Yes, sorry my writeup was unclear and thanks for the clarification @pablogsal - what I mean is that we have two options:

1. We can turn on frame pointers, which imposes a couple of percentage point slowdown always, for all users, including those who are uninterested in profiling.
2. We can ask users who want to use perf (or similar tools) to use -Xperf_jit, which imposes a more significant slowdown on an opt-in basis.

As I understand it, either option provides the same functionality, i.e. equally accurate call stacks, so the differences are performance and user experience. Frame pointers are on by default, but -Xperf_jit has to be manually specified.

If the cost of -Xperf_jit was small, then I'd lean towards the second one. If it were as small as a couple percentage points, then it seems to me obviously better to have this slowdown be sometimes instead of all the time, and IMO any users using perf tools can also figure out how to add this option (and we can maybe add hints around this). But if we're actually looking at a 42% opt-in slowdown, then I think it's debatable whether that's too much, and whether a couple-percent slowdown all the time is a better option. (On the other hand, that 42% number is from pystones, which is a microbenchmark, and not a great one at that, so this may be an overestimate. I haven't yet run the pyperformance suite with -Xperf_jit.)

... All this said, I think we're finding that we need frame pointers for correctness in #991 (comment) so that glibc backtrace() works, and correctness is more important than performance. So that might be the argument for frame pointers, at least in the medium term.

[pablogsal]

I think the discussion so far is framing this a bit too narrowly around perf and call stack quality. The benefits are much broader than that.

Frame pointers are what make the entire ecosystem of low-overhead system observability tools actually work. perf, bpftrace, eBPF-based tracers, continuous profilers like Datadog, Pyroscope, Polar Signals, py-spy, crash reporters, debuggers - all of them work significantly better or become outright more reliable with frame pointers present. Without frame pointers, every single one of these tools either falls back to DWARF unwinding (which requires copying the entire raw stack from kernelspace to userspace on every sample, generating huge data files, and doing expensive post-processing) or just gives you broken/incomplete stacks. So it is not just that the program pays a cost without frame pointers, the profiler itself becomes a worse and more invasive tool.

There is also a compounding effect that I think is easy to underestimate. When frame pointers are absent and you fall back to DWARF unwinding, the overhead of the profiler starts to perturb the workload you are trying to measure. You end up in a situation where profiling is expensive enough that people avoid doing it in production, which means performance regressions go undetected for longer, which means the actual cost of not having frame pointers is much larger than the 1-2% you see in benchmarks.

This is exactly why the broader Linux ecosystem has been moving so decisively in this direction. Fedora enabled frame pointers by default in Fedora 38 after years of debate: https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer. Ubuntu did the same for all 64-bit packages in 24.04: https://ubuntu.com/blog/ubuntu-performance-engineering-with-frame-pointers-by-default. Google, Meta and Netflix to my knowledge have been running their entire production fleets with frame pointers for years because the observability gains are that valuable. Brendan Gregg covers this history well here: https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html.

The consensus from everyone who has operated at scale is that the ~1-2% overhead is not even close to the threshold where it becomes a real concern, and the wins from being able to continuously profile and actually understand what your software is doing in production are orders of magnitude more valuable.

[pablogsal]

I think this is a no brainer :)

[indygreg]

Frame pointers should be enabled on 100% of x86-64 / aarch64 binaries in 2026. Full stop. Benefits greatly outweigh the marginal performance decrease. It is an oversight on my part that I didn't enable them at the inception of this project.

https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html has more details and will hopefully assuage any concerns anyone has.

IMO we shouldn't stop at enabling frame pointers in PBS: we should advocate CPython enable them by default not only in the core interpreter but also for compiled C extensions. The lack of frame pointers in Python extensions externalizes a ton of hardship on the entire Python ecosystem and undermines profiling efforts. Companies expend non-trivial effort to recompile C extensions with frame pointers to facilitate better observability. I'd encourage the Astral folks to maximize frame pointer existence in the tooling they control and advocate for frame pointer adoption in the broader Python ecosystem.

We should also keep our eye on sframes. But that tech is still a bit new and it is probably too early to advocate for broader ecosystem adoption. But PBS could show leadership by adopting them sooner and I encourage exploring sframes as a separate effort.

[pablogsal]

IMO we shouldn't stop at enabling frame pointers in PBS: we should advocate CPython enable them by default not only in the core interpreter but also for compiled C extensions.

As much I enthusiastically agree this sadly will require a PEP. See also python/cpython#96174