From ad88a42f4094eb39e71de523ac1b3e59ef46e268 Mon Sep 17 00:00:00 2001 From: igerber Date: Thu, 9 Jul 2026 18:17:15 -0400 Subject: [PATCH 1/4] docs(todo): close the SE-audit CI-lock row into its Deferred buckets Every sub-item is now resolved or bucketed: (a) G2 hetero/cluster regen DONE (#644, 2026-07-07); C6 CLOSED (#658); C7/C8 TWFE live-R bands tightened + df-convention root-caused (#663, with its own Actionable row). The three remaining letters move to Deferred where they belong: (b) PlaceboTests boundary_gap -> Paper-gated (user-locked 2026-07-09: defer until a derivation/paper source exists); (c) StackedDiD intercept SEs -> Won't-fix/waived (~0.3% reference-cell gap, not lockable); (d) estimatr classical intercept SE -> Won't-fix/waived (documented O(1/n) deviation, reference-only). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Lbd6nqWmg4snvvBmegwqiw --- TODO.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 420c821b..5ac1d69e 100644 --- a/TODO.md +++ b/TODO.md @@ -51,7 +51,6 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m | Issue | Location | Origin | Effort | Priority | |-------|----------|--------|--------|----------| -| SE-audit CI-lock — remaining "fiddly bits" after the second coverage batch landed (that batch pinned C2 `dof_hc2_bm`/`dof_per_coef` via CI-inversion, C3 LOO `df`, C4 estimatr HC1/CR1 intercept SE, C5 Yatchew `p`/`sigma2_lin`/`sigma2_diff`, and the G2 fixest cluster-SE band). Still deferred, each needing a golden regeneration, new computation, or a documented-deviation call — **(a) G2 hetero/cluster is DONE (2026-07-07)**: the unbalanced/heteroskedastic-DGP regen landed, the DiD-path hetero AND cluster CR1 SEs are machine-precision-locked (the plain-OLS CR1 matches fixest exactly — the DOF-convention deviation is absorbed-FE-only), and the TWFE cluster band-pin is retained for the documented non-nested-FE ssc deviation (tracked under "Needs external reference"); TWFE has no public unclustered-hetero surface (auto-cluster convention). Remaining: **(b) PlaceboTests `boundary_gap`** — a permutation randomization-inference margin NOT computed anywhere in code (a new feature + result field, not a coverage lock); **(c) StackedDiD intercept SEs** (`se_cr1/cr2_intercept`, C1) — MEASURED to diverge ~0.3% from R: a nuisance-parameter reference-cell/parameterization gap, NOT machine-precision lockable (the event-study interaction SEs already match ~2e-13; surfacing it would add an unasserted, R-divergent public field); **(d) estimatr `classical` intercept SE** — same documented `O(1/n)` projection/DOF deviation as the slope (reference-only, excluded from parity). C6 CLOSED 2026-07-08 (dr remainder measured ~6e-11 ATT / ~2e-11 rel SE on the no-cov fixture and tightened to 1e-8 with the previously-missing SE assertion added; reg/ipw ATT bands 0.02/0.05 also tightened to 1e-8; the covariate-DR 2e-3 band stays as documented small-sample numerics). C7/C8 re-derivation IN PROGRESS from a fresh loose-tolerance inventory (2026-07-09): the TWFE live-R fixest parity bands were measured and tightened (ATT 1e-3→1e-12 machine, SE/CI 0.01→0.005 documented-CR1-band pins, p-value 0.01→1e-12 both-zero pin, root-caused to the clustered-CR1 inference-df convention: Python t(residual df) vs fixest t(G−1) — REGISTRY Note added + its own Actionable row below); remaining inventory candidates are statistical-by-design (DGP recovery, RNG moments) or fixture-not-committed (triplediff CSVs) — re-measure any new loose golden-backed sites before tightening. | `benchmarks/R/generate_fixest_did_twfe_golden.R`, `tests/test_fixest_did_twfe_parity.py`, `tests/test_methodology_stacked_did.py`, `tests/test_methodology_placebo.py` | SE-audit | Mid | Low | | Clustered-CR1 inference **df convention**: Python uses the residual df (`n − K_full`, e.g. t(148) on the TWFE live-R panel) for clustered t-stats/p-values/CIs while `fixest`/Stata use the cluster df `G − 1` (t(49) same panel) — the common applied recommendation for few-cluster inference. Both are t-distributions; at large \|t\| tails diverge by orders of magnitude (documented as a REGISTRY Note (deviation from R), 2026-07-09). Decide: adopt cluster-df library-wide (moves EVERY clustered p-value/CI — needs a full-suite impact sweep + R parity regen) or keep residual-df as the documented convention. | `diff_diff/linalg.py::LinearRegression.get_inference` + `safe_inference` df-passing callers | SE-audit C7/C8 | Heavy | Medium | --- @@ -64,6 +63,7 @@ Not currently actionable. Retained for provenance + AI-review deviation-document | Issue | Location | PR | Priority | |-------|----------|----|----------| +| `PlaceboTests` `boundary_gap` — a permutation randomization-inference margin (SE-audit item (b)); NOT computed anywhere in code today, so this is a new feature + result field, not a coverage lock. **User-locked 2026-07-09: defer until a derivation/paper source exists** — do not design or implement from scratch. | `tests/test_methodology_placebo.py`, `diff_diff/placebo.py` | SE-audit | Low | | `SyntheticControl` fit-snapshot residency (`_SyntheticControlFitSnapshot`) — **investigated 2026-07-07, parked**: the snapshot ALIASES the fit's own working pivots (zero extra construction cost); the retained residency implements the documented freeze contract (post-fit mutation of estimator inputs must not change `in_space_placebo()` / `leave_one_out()` / conformal output on an already-returned results object, and `__getstate__` already excludes it from pickles). A compact array representation saves only pandas overhead (the float panel dominates); releasing residency needs new API surface (`release`/opt-out flag) or a freeze-contract change. Revisit on user demand for very large donor panels. | `synthetic_control.py`, `synthetic_control_results.py` | follow-up | Low | | Stratified survey-PSU multiplier-weight draw-tiling — **investigated 2026-07-07, parked**: the stratified generator (`generate_survey_multiplier_weights_batch`) consumes ONE sequential rng stream stratum-major (`rng.choice(size=(n_bootstrap, n_h))` per stratum, then lonely-PSU pooling), so draw-chunked assembly CANNOT reproduce the stream bit-identically (contra the old row's parenthetical) — it would need per-stratum generator state skipping (PCG64.advance + per-weight-type variate accounting; fragile) or a stream-layout change (MC-level SE changes → baseline/golden recapture + REGISTRY note). Stratified designs have few PSUs, so the full `(n_bootstrap × n_psu)` matrix rarely matters; unstratified (the large-`n_units` case) is already tiled. Revisit only if a large-PSU stratified design hits memory, as a documented stream change. | `diff_diff/bootstrap_chunking.py::iter_survey_multiplier_weight_blocks` | follow-up | Low | | CBWSDID covariate balancing (`StackedDiD(balance="entropy")`) v1 supports only balanced event windows + `weighting="aggregate"`; unbalanced/ragged panels fail closed (unit-count vs observation-count corrector convention unresolved off balanced panels). Matching-based balancing and the repeated `0→1`/`1→0` episode extension are also deferred. Documented in REGISTRY StackedDiD "Covariate balancing (CBWSDID)" Notes. | `stacked_did.py`, `balancing.py`, REGISTRY | follow-up | Low | @@ -140,6 +140,8 @@ Doable in principle, but no current caller and/or explicitly out of paper scope. | Decision | Location | Verified | |----------|----------|----------| +| StackedDiD intercept SEs (`se_cr1/cr2_intercept`, SE-audit C1/(c)) — measured ~0.3% divergence from R: a nuisance-parameter reference-cell/parameterization gap, not machine-precision lockable (the event-study interaction SEs already match ~2e-13). Surfacing the field would add an unasserted, R-divergent public value; waived 2026-07-09 with the SE-audit row closure. | `tests/test_methodology_stacked_did.py` | SE-audit | Low | +| estimatr `classical` intercept SE (SE-audit (d)) — same documented `O(1/n)` projection/DOF deviation as the slope; reference-only, excluded from parity. Waived 2026-07-09 with the SE-audit row closure. | `tests/test_fixest_did_twfe_parity.py` | SE-audit | Low | | **`bread_inv` reuse not bit-identically achievable.** "Factor `(X'WX)` once, reuse across HC2/HC2-BM" can't be done bit-identically (the bar for a pure perf refactor of the inference path). Internal bread ops solve against *different* RHS (`X.T`, `eye`, `meat`+`temp.T`, `contrasts`); only same-RHS results are bit-reusable. Measured: `lu_solve(lu_factor(A),B)` differs from `solve(A,B)` up to 6.4e-15; the `inv(A)@meat@inv(A)` sandwich differs up to 1.24e-14 — both nonzero and *below* the affected goldens' tolerances (1e-12/1e-10), so a broad reuse would silently shift SEs without tripping the suite. The one genuine bit-identical redundancy (a duplicated `solve(bread, X.T)` in the unweighted one-way `hc2_bm`+`return_dof` path) is dwarfed by that path's dense `M=I−H` build, so the saving is negligible. | `linalg.py::compute_robust_vcov` | 2026-06-01 | | **R-script-per-test consolidation has no CI impact.** No CI workflow installs R, so every R-parity test skips in CI behind a per-file availability gate — consolidating `Rscript` spawns yields zero CI speedup. `test_methodology_twfe.py` already session-caches its R fits. The only residual is a LOCAL-dev micro-opt for `test_methodology_continuous_did.py` / `test_methodology_callaway.py` (re-spawn `library(...)` per call). Low value; retained as a local-dev note. | `tests/test_methodology_continuous_did.py`, `tests/test_methodology_callaway.py` | #139 / 2026-06-07 | | **`HeterogeneousAdoptionDiD` mass-point IV bread is non-symmetric — `_rank_guarded_inv` inapplicable.** The structural rank-guard sweep (continuous_did / two_stage / spillover / conley) excluded `had.py`'s `ZtWX = Zd'WX` ([1, instrument]' × [1, endogenous]): it is a non-symmetric 2×2 Wald-IV bread (`V = ZtWX_inv @ Omega @ ZtWX_inv.T`), and `_rank_guarded_inv` assumes a **symmetric PSD** Gram (symmetric `D=diag(A)` equilibration + eigendecomposition), so applying it would be methodologically wrong. The existing fallback already returns NaN SE on a singular bread; an IV-appropriate near-singular guard would need a different mechanism. | `had.py` | structural-rank-guard / 2026-06-28 | From 248934cc3b727a1f7351678a85f5896bd64172b7 Mon Sep 17 00:00:00 2001 From: igerber Date: Thu, 9 Jul 2026 18:27:41 -0400 Subject: [PATCH 2/4] docs(todo): waiver rows match the Won't-fix table's 3-column shape Review P2: the Won't-fix table is (Decision | Location | Verified); my two rows carried a fourth Priority cell from the Deferred-table shape. Reshaped with bolded decision leads and SE-audit/2026-07-09 Verified stamps, matching the neighboring rows. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Lbd6nqWmg4snvvBmegwqiw --- TODO.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/TODO.md b/TODO.md index 5ac1d69e..addb513f 100644 --- a/TODO.md +++ b/TODO.md @@ -140,8 +140,8 @@ Doable in principle, but no current caller and/or explicitly out of paper scope. | Decision | Location | Verified | |----------|----------|----------| -| StackedDiD intercept SEs (`se_cr1/cr2_intercept`, SE-audit C1/(c)) — measured ~0.3% divergence from R: a nuisance-parameter reference-cell/parameterization gap, not machine-precision lockable (the event-study interaction SEs already match ~2e-13). Surfacing the field would add an unasserted, R-divergent public value; waived 2026-07-09 with the SE-audit row closure. | `tests/test_methodology_stacked_did.py` | SE-audit | Low | -| estimatr `classical` intercept SE (SE-audit (d)) — same documented `O(1/n)` projection/DOF deviation as the slope; reference-only, excluded from parity. Waived 2026-07-09 with the SE-audit row closure. | `tests/test_fixest_did_twfe_parity.py` | SE-audit | Low | +| **StackedDiD intercept SEs not parity-lockable (SE-audit C1/(c)).** Measured ~0.3% divergence from R: a nuisance-parameter reference-cell/parameterization gap, not machine-precision lockable (the event-study interaction SEs already match ~2e-13). Surfacing `se_cr1/cr2_intercept` would add an unasserted, R-divergent public field. Waived with the SE-audit row closure. | `tests/test_methodology_stacked_did.py` | SE-audit / 2026-07-09 | +| **estimatr `classical` intercept SE excluded from parity (SE-audit (d)).** Same documented `O(1/n)` projection/DOF deviation as the slope; reference-only. Waived with the SE-audit row closure. | `tests/test_fixest_did_twfe_parity.py` | SE-audit / 2026-07-09 | | **`bread_inv` reuse not bit-identically achievable.** "Factor `(X'WX)` once, reuse across HC2/HC2-BM" can't be done bit-identically (the bar for a pure perf refactor of the inference path). Internal bread ops solve against *different* RHS (`X.T`, `eye`, `meat`+`temp.T`, `contrasts`); only same-RHS results are bit-reusable. Measured: `lu_solve(lu_factor(A),B)` differs from `solve(A,B)` up to 6.4e-15; the `inv(A)@meat@inv(A)` sandwich differs up to 1.24e-14 — both nonzero and *below* the affected goldens' tolerances (1e-12/1e-10), so a broad reuse would silently shift SEs without tripping the suite. The one genuine bit-identical redundancy (a duplicated `solve(bread, X.T)` in the unweighted one-way `hc2_bm`+`return_dof` path) is dwarfed by that path's dense `M=I−H` build, so the saving is negligible. | `linalg.py::compute_robust_vcov` | 2026-06-01 | | **R-script-per-test consolidation has no CI impact.** No CI workflow installs R, so every R-parity test skips in CI behind a per-file availability gate — consolidating `Rscript` spawns yields zero CI speedup. `test_methodology_twfe.py` already session-caches its R fits. The only residual is a LOCAL-dev micro-opt for `test_methodology_continuous_did.py` / `test_methodology_callaway.py` (re-spawn `library(...)` per call). Low value; retained as a local-dev note. | `tests/test_methodology_continuous_did.py`, `tests/test_methodology_callaway.py` | #139 / 2026-06-07 | | **`HeterogeneousAdoptionDiD` mass-point IV bread is non-symmetric — `_rank_guarded_inv` inapplicable.** The structural rank-guard sweep (continuous_did / two_stage / spillover / conley) excluded `had.py`'s `ZtWX = Zd'WX` ([1, instrument]' × [1, endogenous]): it is a non-symmetric 2×2 Wald-IV bread (`V = ZtWX_inv @ Omega @ ZtWX_inv.T`), and `_rank_guarded_inv` assumes a **symmetric PSD** Gram (symmetric `D=diag(A)` equilibration + eigendecomposition), so applying it would be methodologically wrong. The existing fallback already returns NaN SE on a singular bread; an IV-appropriate near-singular guard would need a different mechanism. | `had.py` | structural-rank-guard / 2026-06-28 | From d34b48ff8421c3fdaa89ad9b53c4e92ecec25d22 Mon Sep 17 00:00:00 2001 From: igerber Date: Thu, 9 Jul 2026 18:30:54 -0400 Subject: [PATCH 3/4] docs(todo): boundary_gap row points at diagnostics.py (placebo.py does not exist) Review P2 x2 (same issue): PlaceboTests is implemented in diff_diff/diagnostics.py; I wrote a nonexistent diff_diff/placebo.py. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Lbd6nqWmg4snvvBmegwqiw --- TODO.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index addb513f..ecd1ef50 100644 --- a/TODO.md +++ b/TODO.md @@ -63,7 +63,7 @@ Not currently actionable. Retained for provenance + AI-review deviation-document | Issue | Location | PR | Priority | |-------|----------|----|----------| -| `PlaceboTests` `boundary_gap` — a permutation randomization-inference margin (SE-audit item (b)); NOT computed anywhere in code today, so this is a new feature + result field, not a coverage lock. **User-locked 2026-07-09: defer until a derivation/paper source exists** — do not design or implement from scratch. | `tests/test_methodology_placebo.py`, `diff_diff/placebo.py` | SE-audit | Low | +| `PlaceboTests` `boundary_gap` — a permutation randomization-inference margin (SE-audit item (b)); NOT computed anywhere in code today, so this is a new feature + result field, not a coverage lock. **User-locked 2026-07-09: defer until a derivation/paper source exists** — do not design or implement from scratch. | `tests/test_methodology_placebo.py`, `diff_diff/diagnostics.py` | SE-audit | Low | | `SyntheticControl` fit-snapshot residency (`_SyntheticControlFitSnapshot`) — **investigated 2026-07-07, parked**: the snapshot ALIASES the fit's own working pivots (zero extra construction cost); the retained residency implements the documented freeze contract (post-fit mutation of estimator inputs must not change `in_space_placebo()` / `leave_one_out()` / conformal output on an already-returned results object, and `__getstate__` already excludes it from pickles). A compact array representation saves only pandas overhead (the float panel dominates); releasing residency needs new API surface (`release`/opt-out flag) or a freeze-contract change. Revisit on user demand for very large donor panels. | `synthetic_control.py`, `synthetic_control_results.py` | follow-up | Low | | Stratified survey-PSU multiplier-weight draw-tiling — **investigated 2026-07-07, parked**: the stratified generator (`generate_survey_multiplier_weights_batch`) consumes ONE sequential rng stream stratum-major (`rng.choice(size=(n_bootstrap, n_h))` per stratum, then lonely-PSU pooling), so draw-chunked assembly CANNOT reproduce the stream bit-identically (contra the old row's parenthetical) — it would need per-stratum generator state skipping (PCG64.advance + per-weight-type variate accounting; fragile) or a stream-layout change (MC-level SE changes → baseline/golden recapture + REGISTRY note). Stratified designs have few PSUs, so the full `(n_bootstrap × n_psu)` matrix rarely matters; unstratified (the large-`n_units` case) is already tiled. Revisit only if a large-PSU stratified design hits memory, as a documented stream change. | `diff_diff/bootstrap_chunking.py::iter_survey_multiplier_weight_blocks` | follow-up | Low | | CBWSDID covariate balancing (`StackedDiD(balance="entropy")`) v1 supports only balanced event windows + `weighting="aggregate"`; unbalanced/ragged panels fail closed (unit-count vs observation-count corrector convention unresolved off balanced panels). Matching-based balancing and the repeated `0→1`/`1→0` episode extension are also deferred. Documented in REGISTRY StackedDiD "Covariate balancing (CBWSDID)" Notes. | `stacked_did.py`, `balancing.py`, REGISTRY | follow-up | Low | From 53a08474ce10daf14d6aa02e3faa380969a508a8 Mon Sep 17 00:00:00 2001 From: igerber Date: Thu, 9 Jul 2026 18:33:43 -0400 Subject: [PATCH 4/4] docs(todo): estimatr intercept waiver points at the estimatr parity file Review P2 x2 (same issue): the classical-intercept O(1/n) waiver context lives in tests/test_estimatr_iv_robust_parity.py (module docstring + test_hc1_intercept_se_matches_estimatr), not the fixest TWFE parity file. Verified against the tree this time, as were the other two rows' paths. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Lbd6nqWmg4snvvBmegwqiw --- TODO.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index ecd1ef50..620a4eb2 100644 --- a/TODO.md +++ b/TODO.md @@ -141,7 +141,7 @@ Doable in principle, but no current caller and/or explicitly out of paper scope. | Decision | Location | Verified | |----------|----------|----------| | **StackedDiD intercept SEs not parity-lockable (SE-audit C1/(c)).** Measured ~0.3% divergence from R: a nuisance-parameter reference-cell/parameterization gap, not machine-precision lockable (the event-study interaction SEs already match ~2e-13). Surfacing `se_cr1/cr2_intercept` would add an unasserted, R-divergent public field. Waived with the SE-audit row closure. | `tests/test_methodology_stacked_did.py` | SE-audit / 2026-07-09 | -| **estimatr `classical` intercept SE excluded from parity (SE-audit (d)).** Same documented `O(1/n)` projection/DOF deviation as the slope; reference-only. Waived with the SE-audit row closure. | `tests/test_fixest_did_twfe_parity.py` | SE-audit / 2026-07-09 | +| **estimatr `classical` intercept SE excluded from parity (SE-audit (d)).** Same documented `O(1/n)` projection/DOF deviation as the slope; reference-only. Waived with the SE-audit row closure. | `tests/test_estimatr_iv_robust_parity.py` | SE-audit / 2026-07-09 | | **`bread_inv` reuse not bit-identically achievable.** "Factor `(X'WX)` once, reuse across HC2/HC2-BM" can't be done bit-identically (the bar for a pure perf refactor of the inference path). Internal bread ops solve against *different* RHS (`X.T`, `eye`, `meat`+`temp.T`, `contrasts`); only same-RHS results are bit-reusable. Measured: `lu_solve(lu_factor(A),B)` differs from `solve(A,B)` up to 6.4e-15; the `inv(A)@meat@inv(A)` sandwich differs up to 1.24e-14 — both nonzero and *below* the affected goldens' tolerances (1e-12/1e-10), so a broad reuse would silently shift SEs without tripping the suite. The one genuine bit-identical redundancy (a duplicated `solve(bread, X.T)` in the unweighted one-way `hc2_bm`+`return_dof` path) is dwarfed by that path's dense `M=I−H` build, so the saving is negligible. | `linalg.py::compute_robust_vcov` | 2026-06-01 | | **R-script-per-test consolidation has no CI impact.** No CI workflow installs R, so every R-parity test skips in CI behind a per-file availability gate — consolidating `Rscript` spawns yields zero CI speedup. `test_methodology_twfe.py` already session-caches its R fits. The only residual is a LOCAL-dev micro-opt for `test_methodology_continuous_did.py` / `test_methodology_callaway.py` (re-spawn `library(...)` per call). Low value; retained as a local-dev note. | `tests/test_methodology_continuous_did.py`, `tests/test_methodology_callaway.py` | #139 / 2026-06-07 | | **`HeterogeneousAdoptionDiD` mass-point IV bread is non-symmetric — `_rank_guarded_inv` inapplicable.** The structural rank-guard sweep (continuous_did / two_stage / spillover / conley) excluded `had.py`'s `ZtWX = Zd'WX` ([1, instrument]' × [1, endogenous]): it is a non-symmetric 2×2 Wald-IV bread (`V = ZtWX_inv @ Omega @ ZtWX_inv.T`), and `_rank_guarded_inv` assumes a **symmetric PSD** Gram (symmetric `D=diag(A)` equilibration + eigendecomposition), so applying it would be methodologically wrong. The existing fallback already returns NaN SE on a singular bread; an IV-appropriate near-singular guard would need a different mechanism. | `had.py` | structural-rank-guard / 2026-06-28 |