fix(profiling): invalidate stale Echion frames on Python 3.14+ - #19641
fix(profiling): invalidate stale Echion frames on Python 3.14+#19641taegyunkim wants to merge 8 commits into
Conversation
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 6f7325d | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-08-11 23:12:51 Comparing candidate commit 93e735b in PR branch Found 0 performance improvements and 6 performance regressions! Performance is the same for 606 metrics, 10 unstable metrics. scenario:httppropagationinject-ids_only
scenario:iastaspects-add_aspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:span-start
scenario:telemetryaddmetric-1-count-metric-1-times
scenario:tracer-small
|
Circular import analysis
|
Dependency direction analysis
|
Codeowners resolved asResolved from the full PR diff against |
73b02b6 to
1040790
Compare
taegyunkim
changed the title
…ode-generation # Conflicts: # ddtrace/internal/datadog/profiling/stack/test/test_sampling_cycle_state.cpp
|
@codex review |
There was a problem hiding this comment.
Pull request overview
Prevents stale Echion frame attribution on Python 3.14 by tracking interpreter code-object generations and invalidating frame identity caches.
Changes:
- Captures interpreter generation snapshots before stack sampling.
- Clears frame and asyncio/uvloop identity caches when generations change or snapshots are incomplete.
- Adds native and Python regression coverage plus release notes.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
tests/profiling/collector/test_stack.py |
Tests code-object address reuse. |
releasenotes/notes/profiling-echion-code-generation-6df0c11c31a5d70c.yaml |
Documents the fix. |
ddtrace/internal/datadog/profiling/stack/test/test_sampling_cycle_state.cpp |
Tests generation-based invalidation. |
ddtrace/internal/datadog/profiling/stack/src/sampler.cpp |
Captures snapshots before unwinding. |
ddtrace/internal/datadog/profiling/stack/src/echion/interp.cc |
Reads generations and reports snapshot completeness. |
ddtrace/internal/datadog/profiling/stack/include/sampler.hpp |
Adds interpreter snapshot storage. |
ddtrace/internal/datadog/profiling/stack/fuzz/fuzz_echion_interp.cpp |
Adapts the fuzz harness to the new return value. |
ddtrace/internal/datadog/profiling/stack/echion/echion/interp.h |
Extends interpreter metadata and traversal API. |
ddtrace/internal/datadog/profiling/stack/echion/echion/echion_sampler.h |
Implements cache invalidation and fork reset behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ce98e9870e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Suppressed comments (1)
tests/profiling/collector/test_stack.py:193
- The regression does not prove that the old function was sampled before its code object is released. If the sampler misses this first execution window but samples the replacement, both final assertions pass even with generation invalidation disabled because no stale cache entry was ever created. Inspect the profile emitted here and require
old_nameto be present before proceeding.
ddup.upload()
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
d2587c4 to
56a5f8a
Compare
taegyunkim
changed the title
…ode-generation # Conflicts: # ddtrace/internal/datadog/profiling/stack/src/sampler.cpp
Description
On Python 3.14, Echion's frame cache can return stale metadata when CPython reuses a released code object's address. Before each sampling cycle, snapshot every interpreter's
(interpreter ID, code object generation). If that snapshot changes, clear the global frame cache before unwinding stacks. Read the generation field from the running interpreter's patch-stable_PyRuntime.debug_offsetstable instead of compiling a privatePyInterpreterStateoffset into the wheel.If the interpreter snapshot is incomplete, clear the same state and skip that sampling cycle. This preserves Echion's existing multi-interpreter traversal, including processes that host a legacy subinterpreter without importing ddtrace there.
Python 3.13 and earlier are unchanged because they do not expose
_code_object_generation.Testing
mainthroughff80eb3f0and preserved both the frame-invalidation scratch state and the new sampling-error state across the post-fork reset.scripts/lint checks,scripts/lint profiling-native-check, andgit diff --checkpass.py314-echion-genRapid Test Drive and exercised the dynamic code-generation endpoint.bfaf5a1f577fa4730aa15010e96f6f19eecc4502and deployed the exact cp314 wheel to thepy314-echion-bothRapid Test Drive asv132223867-c51014b. Four concurrent two-second dynamic-code-generation requests completed successfully, the pod had no restarts, and the Gunicorn master sampling thread used 0.62 CPU seconds/minute instead of the previously observed 42.75 seconds/minute. Profiles showed coherent dynamic-code-generation stacks and no duplicated-task wall-time inflation.Risks
Workloads that frequently create code objects will clear Echion's frame cache more often. The invalidation is limited to Python 3.14 and keeps the existing cache structure and sampling behavior otherwise unchanged.
An unreadable or inconsistent interpreter snapshot skips one cycle rather than risking stale frame attribution.
The runtime offset relies on CPython 3.14's documented patch-stability for
_PyRuntimeState.debug_offsets; it does not assume compatibility across Python minor versions.The generation snapshot is intentionally lock-free and taken once per sampling cycle. Code-object deletion after that snapshot can leave frame metadata stale for the remainder of the current cycle; the next cycle observes the generation change and clears the cache. Eliminating this bounded race would require additional per-frame synchronization or identity tracking that is outside this change.
Additional Notes
This intentionally does not add per-interpreter frame caches, change frame-cache keys, or add code watchers for older Python versions.
Pre-review checklist