feat(imputation): leave-one-out conservative variance (BJS 2024 App. A.9)#630
feat(imputation): leave-one-out conservative variance (BJS 2024 App. A.9)#630igerber wants to merge 1 commit into
Conversation
Overall Assessment
Executive Summary
MethodologyFinding M1 — P3 Informational, No Action Required Severity: P3 Locations: Code QualityNo findings. The parameter is validated in Locations: PerformanceNo findings. The near-cancellation recomputation is scoped to suspect dominated rows and should not change the normal path materially. MaintainabilityNo findings. The LOO logic is isolated in Locations: Tech DebtNo findings. The prior TODO for ImputationDiD LOO variance was removed, and I did not see a new untracked correctness limitation. SecurityNo findings. No secrets or security-sensitive behavior were introduced in the changed code. Documentation/TestsFinding T1 — P1 Missing Parameter Interaction Coverage Severity: P1
Locations: implemented aggregation paths at Path to Approval
|
…A.9) Add the opt-in `leave_one_out` parameter (default False) implementing the Borusyak-Jaravel-Spiess (2024) Supplementary Appendix A.9 finite-sample refinement to ImputationDiD's conservative variance. The non-LOO auxiliary aggregate tau_tilde_g (eq. 8) is built from the fitted tau_hat_it and thus partially overfits to the noise epsilon_it, biasing the variance downward. leave_one_out=True recomputes each unit's group aggregate excluding that unit -- implemented efficiently (A.9) by rescaling each treated auxiliary residual by 1/(1 - v_ig**2/sum_j v_jg**2), which reproduces the direct leave-one-out at the per-unit cluster sum psi_i EXACTLY (self-derived, machine-precision-verified identity). Yields a larger, less-downward-biased SE (Prop. A8: unbiased for an upper bound at the default unit clustering; an equal-weight K-unit group inflates residuals by exactly K/(K-1)). Point estimates and the entire default path (leave_one_out=False) are byte-identical, preserving R didimputation parity. Robustness: - the leave-one-out denominator is the exact sum of the OTHER units' squared weights (drop-then-sum), not D - v_ig**2 after forming the ratio, so an extremely dominated >=2-unit group keeps its large finite factor instead of silently reverting to non-LOO; the singleton test counts positive-weight units (handles effective singletons); - replicate-weight survey designs raise NotImplementedError (their variance bypasses the influence-function path where the rescale lives); - leave_one_out is recorded on ImputationDiDResults, to_dict(), and summary(). Validated by the exact psi-identity, K/(K-1) hand-calc, direction (LOO >= non-LOO at unit clustering), extreme-weight / near-cancellation edges, and MC coverage; the authors' Stata did_imputation ships the same option (no CI-runnable anchor). Fills the A.9 GAP in the BJS review doc (sourced from arXiv 2108.12419v5). Closes the ImputationDiD LOO Actionable TODO row. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LHDijzf8zHXk5T8ahS2mKi
igerber
force-pushed
the
feature/imputation-leave-one-out
branch
from
86cfd13 to
a57a1b6
Compare
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall Assessment✅ Looks good — no unmitigated P0/P1 findings. The previous P1 on missing Executive Summary
MethodologyFinding M1 — P3 Informational, No Action Required Severity: P3 Locations: Code QualityNo findings. Locations: PerformanceNo findings. The near-cancellation exact recomputation is limited to suspect dominated Location: MaintainabilityNo findings. The LOO logic is isolated in Locations: Tech DebtNo findings. The prior TODO entry for ImputationDiD LOO variance was removed after implementation, and I did not find a newly introduced untracked correctness limitation. Location: SecurityNo findings. The PR does not introduce secrets, filesystem/network behavior, or security-sensitive code paths. Documentation/TestsNo blocking findings. The prior P1 is addressed by new tests covering Locations: Verification note: focused test execution was not possible here: |
1 participant
Summary
ImputationDiD(leave_one_out=False)parameter implementing the Borusyak-Jaravel-Spiess (2024) Supplementary Appendix A.9 leave-one-out finite-sample refinement to the conservative variance.tau_tilde_g(eq. 8) is built from the fittedtau_hat_itand so partially overfits to the noiseepsilon_it, biasing the variance downward.leave_one_out=Truerecomputes each unit's group aggregate excluding that unit — implemented efficiently (A.9) by rescaling each treated auxiliary residual by1/(1 - v_ig**2 / sum_j v_jg**2), which reproduces the direct leave-one-out at the per-unit cluster sumpsi_iexactly (self-derived, machine-precision-verified identity). Yields a larger, less-downward-biased SE (Prop. A8: unbiased for an upper bound at the default unit clustering; an equal-weightK-unit group inflates residuals by exactlyK/(K-1)).leave_one_out=False) are byte-identical (Rdidimputationparity preserved). The opt-in SE inflation scales with 1/group-size: ~0.4% on large panels, up to ~13% on small/fine-partition designs.D - v_ig**2, which cancels for dominated>=2-unit groups); replicate-weight survey designs raiseNotImplementedError(their variance bypasses the IF path);leave_one_outis recorded onImputationDiDResults/to_dict()/summary().Methodology references (required if estimator / math changes)
docs/methodology/REGISTRY.md## ImputationDiD(new leave-one-out Note, the derivation's home);docs/methodology/papers/borusyak-jaravel-spiess-2024-review.md(A.9 GAP now filled with a provenance note — sourced from arXiv 2108.12419v5 App. A.9; the REStud Supplementary Material is canonical).tau_tilderesidual rescale (as in the authors' Statadid_imputation); the stronger Prop. A8 variant that also leaves out for the covariatedelta_hatis noted but not implemented. TheLOO >= non-LOO/ upper-bound guarantee is documented as holding at the default unit clustering; coarsercluster=/ analyticalsurvey_design=/n_bootstrapcompositions apply the same rescale as a documented library extension. Default off preserves Rdidimputationparity.Validation
tests/test_methodology_imputation.py::TestB2024AppendixA9LeaveOneOut— the exact psi-identity vs the paper's direct leave-one-out (paper-fidelity gate,atol=1e-12),K/(K-1)hand-calc, direction (LOO >= non-LOOat unit clustering), extreme-weight / positive-near-cancellation edges, effective-singleton fallback, replicate-weight fail-closed, result metadata, and@slowMC coverage. Existing_EQ8_COHORT_SE_PINand Rdidimputationgolden-parity tests pass unchanged (default byte-identity).didimputationomits LOO; the authors' Statadid_imputationships it but Stata is not in CI) — the exact psi-identity anchors the implementation to the paper's owntau_tilde_it^LOdefinition.Security / privacy
🤖 Generated with Claude Code
https://claude.ai/code/session_01LHDijzf8zHXk5T8ahS2mKi