fix(tests): sandbox doc-snippet env mutations; order-robust dCDH baseline backend detection#637
Conversation
…line backend detection The doc-snippet runner exec'd documentation code blocks without sandboxing os.environ, so the troubleshooting page's backend-override snippet (os.environ['DIFF_DIFF_BACKEND'] = 'python') leaked into every later test in a full-suite run (7,008 teardowns with perturbed state, traced with a teardown-hook leak detector). The only victim was the dCDH pinned bootstrap baseline test, whose call-time env read selected the pure-Python baseline arm while the fit still dispatched to the already-imported Rust backend (fails under full-suite order, passes standalone — the TODO row's exact signature). Root cause: snapshot/restore os.environ around each snippet exec. Defense in depth: the dCDH test now derives its baseline arm from the same dispatch globals the fit consumes (bootstrap_utils / linalg, bound at import), with a coherence assert between the two. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Overall Assessment✅ Looks good. No unmitigated P0/P1 findings. Executive Summary
MethodologyNo findings. Affected method: Code QualityNo findings. The environment sandbox is correctly placed around snippet execution with PerformanceNo findings. The changes are test-only and add negligible overhead: one MaintainabilityNo findings. The new comments explain the order-dependent failure mode and why call-time Tech DebtNo findings. The PR removes the already-tracked TODO row for the order-dependent dCDH baseline failure at SecurityNo findings. The environment restoration reduces cross-test process-state leakage. No secrets or unsafe external behavior were introduced. Documentation/TestsNo blocking findings.
|
Summary
tests/test_doc_snippets.py) exec'd documentation code blocks without sandboxingos.environ, so the troubleshooting page's backend-override snippet (os.environ['DIFF_DIFF_BACKEND'] = 'python',docs/troubleshooting.rst:329) leaked into every later test of a full-suite run — 7,008 teardowns with perturbed state, traced with a teardown-hook leak detector. The runner now snapshot/restoresos.environin afinallyaround each snippet exec.test_survey_dcdh.py::test_bootstrap_se_matches_pre_pr4_baselineselected its pinned-baseline arm from a call-timeos.environ/diff_diff._backendread — order-fragile, since an env mutation after import flips the detection without changing the already-imported dispatch. It now derives the arm from the same dispatch globals the fit consumes (bootstrap_utils.HAS_RUST_BACKEND/_rust_bootstrap_weights,linalg.HAS_RUST_BACKEND/_rust_solve_ols) with a coherence assert between the two.backend=pure-pythonwhile producing the rust arm's value, passes standalone).Methodology references (required if estimator / math changes)
Validation
tests/test_doc_snippets.py(env sandbox),tests/test_survey_dcdh.py(dispatch-true arm selection).pytest tests/test_doc_snippets.py tests/test_survey_dcdh.py::TestBootstrapCellPeriod::test_bootstrap_se_matches_pre_pr4_baselinefails on main, passes with the fix; full local suite previously 1 failed / 8558 passed (this test the sole failure), the leak detector confirms the first perturbed teardown istest_doc_snippet[troubleshooting:block11].Security / privacy
Generated with Claude Code