fix(profiling): stack-profiler SIGSEGV on shutdown (PROF-15342)

[vlad-scherbich]

< Prev PR | Next PR

Summary

This PR closes the shutdown race (Layer 2).

Profiler._stop_service runs the final scheduler.flush() before it stops the
collectors. That first flush resolves the code-provenance file, which the very
first time triggers one-time importlib.metadata cold imports
(setuptools / _distutils_hack / packaging). Those imports run while the native
stack sampler is still walking live frames during interpreter teardown, so the
sampler can read a freed PyCodeObject* and fault inside safe_memcpy.

Changes

• Pre-warm code provenance at scheduler start (scheduler.py) so the final
  flush performs no cold imports — done on a safe thread when nothing is being
  torn down.
• Pause the native sampler before the final flush (profiler.py +
  collector/stack.py's pause_for_shutdown_flush). Collectors are not stopped
  yet, so memalloc's snapshot still works; the stack v2 sampler pushes directly
  to ddup and has no meaningful snapshot, so pausing it is safe.
• Py_IsFinalizing guard in the native sampling loop (sampler.cpp) so the
  sampler bails the moment finalization begins (covers teardown paths that don't
  stop the thread first, e.g. uWSGI --skip-atexit, hard exits).

Stacked with #18798 (handler-ownership hardening), which depends on this branch.

Reproduction

Repro script included: scripts/profiling/repro_prof_14568_shutdown.py. It
repeatedly starts/stops the profiler while threads churn frames, clearing the
code-provenance cache each cycle so the final flush does the cold imports while
sampling.

# crashes (intermittently) on main, survives on this branch
DD_PROFILING_STACK_FAST_COPY=1 python scripts/profiling/repro_prof_14568_shutdown.py

Test plan

• Rebuild the stack native extension.
• pytest tests/profiling/test_main.py -k start_stop_under_load
• Run the repro on main (intermittent crash) vs this branch (clean) on Linux.
• Profiling riot venvs.

[cit-pr-commenter-54b7da]

Codeowners resolved as

ddtrace/internal/datadog/profiling/stack/src/sampler.cpp                @DataDog/profiling-python
ddtrace/profiling/collector/stack.py                                    @DataDog/profiling-python
ddtrace/profiling/profiler.py                                           @DataDog/profiling-python
ddtrace/profiling/scheduler.py                                          @DataDog/profiling-python
releasenotes/notes/profiling-fix-stack-sampler-shutdown-segv-9f3c1a7e54b2d806.yaml  @DataDog/apm-python
scripts/profiling/repro_prof_14568_shutdown.py                          @DataDog/apm-core-python
tests/profiling/test_main.py                                            @DataDog/profiling-python

[datadog-prod-us1-6]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 36 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741491-d2b8243-musllinux_1_2_x86_64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]     

View all 36 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 323a9ec | Docs | Datadog PR Page | Give us feedback!}

[Copilot]

Pull request overview

Fixes a stack-profiler shutdown crash (SIGSEGV) by removing cold-import work from the final flush, quiescing the native sampler before the last upload, and adding a native-side guard to stop sampling as soon as interpreter finalization begins.

Changes:

• Pre-warm code-provenance resolution during scheduler startup so shutdown flush avoids one-time importlib.metadata imports.
• Pause the native stack sampler prior to the final flush; add Py_IsFinalizing/_Py_IsFinalizing guard in the sampling loop.
• Add a regression subprocess test, a standalone repro script, and a release note.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ddtrace/profiling/scheduler.py	Pre-warms code provenance at scheduler start to prevent cold imports during shutdown flush.
ddtrace/profiling/profiler.py	Pauses collectors (when supported) before the final scheduler flush.
ddtrace/profiling/collector/stack.py	Implements pause_for_shutdown_flush() for the stack collector via stack.pause_sampling().
ddtrace/internal/datadog/profiling/stack/src/sampler.cpp	Stops sampling immediately when the interpreter is finalizing.
tests/profiling/test_main.py	Adds a regression subprocess test for repeated start/stop under load.
scripts/profiling/repro_prof_14568_shutdown.py	Adds a standalone reproduction script for the shutdown crash scenario.
releasenotes/notes/profiling-fix-stack-sampler-shutdown-segv-9f3c1a7e54b2d806.yaml	Documents the crash fix in release notes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.