You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Extends the #847 profiling contract: the shipped binary keeps its symbol table
so function-level flame graphs work in the field. This asks for one more
property with the same rationale: build the release binary with -C force-frame-pointers=yes, so the frame-pointer half of the profiling
ecosystem works on the shipped image too.
What we measured against the shipped binary
While validating an external CPU profiler against the release image (as
shipped, no rebuild), we needed an independent reference for the call chains.
Both of perf's modes failed on the shipped binary:
perf record --call-graph fp: unusable — the release profile does not
emit frame pointers, so the kernel's rbp walk has nothing to follow.
perf record --call-graph dwarf: libdw fails to evaluate this binary's
call-frame information. Measured on a 6.18 KVM guest under a bench/pgo-training load: only 22 of 618 sampled chains ever crossed
the libc-to-aisix boundary; everything else stopped there. The CFI data
itself is fine — readelf --debug-dump=frames-interp parses all 36,928
FDEs cleanly — it is elfutils' evaluator that gives up on the fat-LTO
output. Upstream's bug, but it ships in every distro's perf.
So today, no stock profiling tool can walk the shipped binary's stacks. To get
a working reference at all we had to rebuild the pinned release commit with RUSTFLAGS="-C force-frame-pointers=yes". On that build, perf --call-graph fp walked 99.9% of samples to the thread entry with user-chain depth p90
of 44 — essentially perfect. One flag is the difference between "no external
tool works" and "everything fp-based works".
Cost
Industry measurements put force-frame-pointers at under 1% on x86-64
(16 GPRs; losing rbp as a scratch register stopped mattering a decade
ago). Fedora 38+, Ubuntu 24.04+ and Arch now build their entire
distributions with frame pointers on this reasoning.
Fat LTO, codegen-units = 1, and PGO are unaffected — frame pointers are a
per-function prologue property, orthogonal to inlining and layout.
What it buys
perf record -g (fp is its default), bcc/bpftrace stack helpers, and every
other rbp-walking tool work on the release image as shipped — support
and customers can profile without a rebuild.
Profiling validation stops rebuilding the world: the flag makes the shipped
artifact its own reference target.
Honest limits
It does not fix libdw's fat-LTO problem; --call-graph dwarf stays broken
until elfutils does.
rbp walkers still lose the frames below any call into a library built
without frame pointers (today: glibc in the base image). That blind spot
belongs to the base image's libc, not to this flag; it shrinks as distros
ship fp-built libc.
Implementation sketch
The release binary's RUSTFLAGS are set in the Dockerfile's PGO phases
(-Cprofile-generate=... and -Cprofile-use=...) and unset in the PGO=off branch. Append -C force-frame-pointers=yes to all three — keeping
both PGO phases' flags identical so the trained layout matches the shipped
shape — or hoist it into one ARG/ENV so it cannot drift between branches.
The #847 CI check already pins the symbol table because "a future strip step,
RUSTFLAGS change, or base-image swap would break flame graphs silently".
Frame pointers deserve the same pin once added: assert a sampled function
prologue starts with push %rbp; mov %rsp,%rbp (e.g. objdump -d on main)
in the same job, so the property cannot regress silently either.
Extends the #847 profiling contract: the shipped binary keeps its symbol table
so function-level flame graphs work in the field. This asks for one more
property with the same rationale: build the release binary with
-C force-frame-pointers=yes, so the frame-pointer half of the profilingecosystem works on the shipped image too.
What we measured against the shipped binary
While validating an external CPU profiler against the release image (as
shipped, no rebuild), we needed an independent reference for the call chains.
Both of
perf's modes failed on the shipped binary:perf record --call-graph fp: unusable — the release profile does notemit frame pointers, so the kernel's rbp walk has nothing to follow.
perf record --call-graph dwarf: libdw fails to evaluate this binary'scall-frame information. Measured on a 6.18 KVM guest under a
bench/pgo-trainingload: only 22 of 618 sampled chains ever crossedthe libc-to-aisix boundary; everything else stopped there. The CFI data
itself is fine —
readelf --debug-dump=frames-interpparses all 36,928FDEs cleanly — it is elfutils' evaluator that gives up on the fat-LTO
output. Upstream's bug, but it ships in every distro's
perf.So today, no stock profiling tool can walk the shipped binary's stacks. To get
a working reference at all we had to rebuild the pinned release commit with
RUSTFLAGS="-C force-frame-pointers=yes". On that build,perf --call-graph fpwalked 99.9% of samples to the thread entry with user-chain depth p90of 44 — essentially perfect. One flag is the difference between "no external
tool works" and "everything fp-based works".
Cost
force-frame-pointersat under 1% on x86-64(16 GPRs; losing
rbpas a scratch register stopped mattering a decadeago). Fedora 38+, Ubuntu 24.04+ and Arch now build their entire
distributions with frame pointers on this reasoning.
symbol table Establish the on-CPU profiling workflow: adopt cargo-flamegraph and add a dedicated profiling build profile #847 already ships (~10 MiB).
codegen-units = 1, and PGO are unaffected — frame pointers are aper-function prologue property, orthogonal to inlining and layout.
What it buys
perf record -g(fp is its default), bcc/bpftrace stack helpers, and everyother rbp-walking tool work on the release image as shipped — support
and customers can profile without a rebuild.
artifact its own reference target.
Honest limits
--call-graph dwarfstays brokenuntil elfutils does.
without frame pointers (today: glibc in the base image). That blind spot
belongs to the base image's libc, not to this flag; it shrinks as distros
ship fp-built libc.
Implementation sketch
The release binary's
RUSTFLAGSare set in the Dockerfile's PGO phases(
-Cprofile-generate=...and-Cprofile-use=...) and unset in thePGO=offbranch. Append-C force-frame-pointers=yesto all three — keepingboth PGO phases' flags identical so the trained layout matches the shipped
shape — or hoist it into one
ARG/ENVso it cannot drift between branches.The #847 CI check already pins the symbol table because "a future strip step,
RUSTFLAGS change, or base-image swap would break flame graphs silently".
Frame pointers deserve the same pin once added: assert a sampled function
prologue starts with
push %rbp; mov %rsp,%rbp(e.g.objdump -donmain)in the same job, so the property cannot regress silently either.
Verification