fix(profiling): reject inconsistent Python 3.14+ asyncio task lists

[taegyunkim]

Description

Python 3.14 tracks asyncio tasks in circular linked lists and can move a lingering task from a thread-local list to an interpreter-wide list. Echion samples these lists without acquiring the GIL or stopping Python threads, so it can observe the old thread-list head after the task has moved. Following that stale head enters the interpreter list at the moved task and repeatedly walks the same cycle. In the observed failure, Echion emitted one task 32,768 times before reaching its traversal limit, inflating task wall time and consuming significant profiler CPU.

This change rejects a task source when its links are inconsistent or a node repeats, discarding any tasks already collected from that source. It also deduplicates task identities found across the thread-local, interpreter-wide, scheduled, and eager task sources before accounting and reservoir sampling.

Testing

• Added a deterministic Python 3.14 native regression for the stale-head and moved-cycle topology. It traverses one valid task before reaching the malformed edge, then verifies that partial source result is removed while a preexisting sentinel remains. Before the fix, the test reliably retained 32,768 duplicate tasks, for 32,769 entries including the sentinel.
• CI passes.
• Deployed the production fix at e7c8caf2e4d4dc14cf3bf2e91b4635bb9c670e2e to the Python 3.14 Rapid Test Drive py314-echion-tasks as version v132205293-daa67e1. The pod stayed healthy with no restarts while the standard Rapid Python smoke workload, including dynamic code generation, ran.
• Validated profiles from 2026-08-20 17:58:00 through 18:00:54 UTC. The Python wall-time profile reported a plausible aggregate 3.56 minutes of wall time per minute across three Python runtimes, with no duplicated-task inflation. In the native CPU profile filtered to task traversal, unwind_tasks(), get_all_tasks(), and get_tasks_from_linked_list() together accounted for 27.4 ms/min, or 0.065% of native CPU.

Risks

Normal Python 3.14 task-list traversal now performs one visited-set insertion per task, plus final cross-source deduplication. Typical task lists are small. Inconsistent lock-free snapshots fail closed for only the affected source, while discovery continues with other task sources as before.

Additional Notes

This issue was discovered while testing the Python 3.14+ Echion code-object cache invalidation work in #19641. The task traversal bug predates that PR and also reproduces from its merge base.

The reproducer was committed separately before the fix as 369097c3f. The Python 3.14 task-source roles, traversal validity conditions, earliest-snapshot deduplication policy, and source-local rollback contract are documented next to their implementations. The regression test also documents its synthetic list topology and restoration of CPython-owned links.

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/datadog/profiling/stack/echion/echion/threads.h        @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/echion/threads.cc          @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/test/CMakeLists.txt            @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/test/test_task_traversal.cpp   @DataDog/profiling-python
releasenotes/notes/fix-profiler-asyncio-task-traversal-754a147329fc90cd.yaml  @DataDog/apm-python

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 250 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 250 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=135)
ddtrace.llmobs._integrations.openai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)
ddtrace.llmobs._integrations.google_adk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)
ddtrace.appsec._contrib.flask -×-> ddtrace.trace  (product:appsec -> product:tracing, score=133)
ddtrace.appsec._listeners -×-> ddtrace.trace  (product:appsec -> product:tracing, score=133)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[datadog-datadog-prod-us1-2]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: dc5090f | Docs | View more details | Give us feedback!}

[KowalskiThomas]

Will continue reviewing later.

[taegyunkim]

@codex review

[Copilot]

Pull request overview

Hardens Python 3.14+ asyncio task discovery against inconsistent lock-free snapshots and duplicate task accounting.

Changes:

• Validates linked-list structure and rolls back failed-source results.
• Deduplicates tasks across discovery sources.
• Adds native regression coverage and a release note.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
releasenotes/notes/fix-profiler-asyncio-task-traversal-754a147329fc90cd.yaml	Documents the profiling fix.
ddtrace/internal/datadog/profiling/stack/test/test_task_traversal.cpp	Tests stale-list rejection and deduplication.
ddtrace/internal/datadog/profiling/stack/test/CMakeLists.txt	Registers the native test.
ddtrace/internal/datadog/profiling/stack/src/echion/threads.cc	Adds validation, rollback, and deduplication.
ddtrace/internal/datadog/profiling/stack/echion/echion/threads.h	Grants the test fixture private access.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 0e3726cd99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".