diff --git a/CHANGELOG.md b/CHANGELOG.md index 60e5c940..5575aa5d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `did_had_pretest_workflow`, ...) are unchanged in this release and removed separately. ### Testing +- **`CallawaySantAnna` ipw R-parity yardsticks folded into the golden fixture + no-covariate + ipw structural-parity decision recorded.** `csdid_golden_values.json` regenerated (R 4.5.2, + did 2.5.1, DRDID 1.3.0): all pre-existing data and result blocks reproduced byte-identically; + the ipw scenario now carries the `aggte` simple/dynamic/group blocks (identical to the + previously hardcoded 2026-07-05 yardsticks), and + `test_golden_ipw_aggregation_se_vs_r_did_251` reads them from the JSON. The no-covariate ipw + branch's unconditional-propensity treatment is now a recorded document-only decision + (REGISTRY § CallawaySantAnna): R `did`'s intercept-only logit is deliberately not mirrored — + its estimation-effect correction is identically zero, and no-covariate ipw/reg/dr reduce to + the same difference-in-means IF, locked bit-identical per cell by a new + `TestDRNoCovariateSEUniformity::test_ipw_no_cov_per_cell_identical_to_reg`. - **CI-locked standard-error parity for flagship and previously-unasserted paths (SE-audit coverage batch).** These surfaces computed SEs matching R but had no CI assertion pinning them (the latent-risk pattern that once hid the CallawaySantAnna reg-method gap): diff --git a/TODO.md b/TODO.md index c35b548f..04b74101 100644 --- a/TODO.md +++ b/TODO.md @@ -31,8 +31,6 @@ The `Origin` column (Actionable tables) and the `PR` column (Deferred tables) bo | `SyntheticControl` conformal (CWZ 2021) extensions: (a) one-sided / signed-`t` variants (§7); (b) covariates in the conformal proxy (`X_jt`, eqs 4/6 — current proxy is outcomes-only); (c) AR / innovation-permutation path (Lemmas 5-7) for time-series proxies. The joint test, pointwise CIs, and average-effect CI have landed. | `conformal.py`, `synthetic_control_results.py` | CWZ-2021 | Heavy | Low | | `ContinuousDiD` CGBS-2024 extensions. (a) `covariates=` kwarg — **DONE (reg/dr)**; (b) discrete-treatment saturated regression (`treatment_type="discrete"`) — **DONE**; (c) lowest-dose-as-control per Remark 3.1 when `P(D=0)=0` (`control_group="lowest_dose"`) — **DONE** (discrete + continuous mass-point, single-cohort; estimand `ATT(d)−ATT(d_L)`; see REGISTRY Note #7). Remaining (all deferred `NotImplementedError`, documented): `estimation_method="ipw"` on the dose curve (scalar-adjustment / degenerate); `covariates=` × `survey_design=` (weighted OR + weighted nuisance IF); multi-cohort **heterogeneous-support** discrete aggregation (support-aware: average each dose only over the cohorts that observe it); **multi-cohort `lowest_dose`** (within-cohort `d_L` reference + support-aware cross-cohort aggregation); and **`covariates=` × `lowest_dose`** (conditional-PT-relative-to-`d_L` estimand). Single-cohort / 2-period / shared-support multi-cohort are supported. | `continuous_did.py` | CGBS-2024 | Heavy | Low | | `TwoWayFixedEffects(vcov_type in {hc2, hc2_bm})` with replicate-weight designs raises `NotImplementedError` (`twfe.py:~233`). The replicate path re-demeans per replicate, which doesn't compose with the full-dummy HC2/HC2-BM build — a correct impl needs per-replicate full-dummy refit. Workaround: `hc1` for replicate-weight CR1. | `twfe.py::fit` | follow-up | Heavy | Low | -| `CallawaySantAnna` no-covariate ipw treats the propensity as unconditional (no correction term); R's `did` fits an intercept-only logit whose estimation effect is identically zero in the IF, so values match — decide whether to document-only (current REGISTRY note) or mirror R's code path for structural parity. | `staggered.py::_ipw_estimation` (no-cov branch) | CS-scaling | Quick | Low | -| Fold the R `did` 2.5.1 ipw aggregation yardsticks (hardcoded with provenance in `test_golden_ipw_aggregation_se_vs_r_did_251`) into `csdid_golden_values.json` on the next fixture regeneration — the generator already emits the ipw `aggte` blocks; switch the test to read the JSON. | `benchmarks/R/generate_csdid_test_values.R`, `tests/test_csdid_ported.py` | CS-scaling | Quick | Low | | TWFE's HC2/HC2-BM inline full-dummy build (`twfe.py:280-315`) duplicates the dummy-construction logic in `DifferenceInDifferences(fixed_effects=...)` (`estimators.py:478-486`). Extract a shared helper, or delegate TWFE's HC2/HC2-BM path to DiD's `fixed_effects=` branch (with TWFE-specific cluster-default threading), to reduce drift risk on FE naming / survey behavior / result-surface conventions. Substantive refactor — touches both estimators. | `twfe.py::fit`, `estimators.py::DifferenceInDifferences.fit` | follow-up | Heavy | Low | | `HonestDiD` Δ^SD optimal-FLCI affine-estimator optimizer (Nelder-Mead over slope weights) diverges from R `.FLCI.computeFLCI` at intermediate small M (~`[0.005, 0.02]`): the CI **center** shifts up to ~9% on wide pre/post windows while the **width** matches to ~1e-3. Not a local-minimum artifact (multi-start does not move it); the two implementations land on different affine estimators of near-equal length. Coverage is unaffected. Reconcile our FLCI optimization with R's exact algorithm. Exposed when the identified-set NaN gate was removed (finite CIs now surface at all M). | `honest_did.py::_compute_optimal_flci` | SE-audit | Mid | Low | diff --git a/benchmarks/R/generate_csdid_test_values.R b/benchmarks/R/generate_csdid_test_values.R index b25594a8..f3fa54bf 100644 --- a/benchmarks/R/generate_csdid_test_values.R +++ b/benchmarks/R/generate_csdid_test_values.R @@ -108,10 +108,12 @@ data2_ipw <- build_sim_dataset(sp2_ipw) res2_ipw <- att_gt(yname = "Y", xformla = ~X, data = data2_ipw, tname = "period", idname = "id", gname = "G", est_method = "ipw", bstrap = FALSE, cband = FALSE) -# Aggregations for the ipw scenario. Until the fixture is regenerated, these -# values are hardcoded (from did 2.5.1 / DRDID 1.3.0, 2026-07-05) in -# tests/test_csdid_ported.py::test_golden_ipw_aggregation_se_vs_r_did_251; -# on the next regeneration the test can switch to reading them from the JSON. +# Aggregations for the ipw scenario, read by +# tests/test_csdid_ported.py::test_golden_ipw_aggregation_se_vs_r_did_251. +# Folded into the JSON on the 2026-07-07 regeneration (did 2.5.1 / DRDID +# 1.3.0, R 4.5.2); all pre-existing data and result blocks reproduced +# byte-identically, and the folded values match the previously hardcoded +# 2026-07-05 yardsticks exactly. agg2_ipw_simple <- aggte(res2_ipw, type = "simple", bstrap = FALSE, cband = FALSE) agg2_ipw_dynamic <- aggte(res2_ipw, type = "dynamic", bstrap = FALSE, cband = FALSE) agg2_ipw_group <- aggte(res2_ipw, type = "group", bstrap = FALSE, cband = FALSE) diff --git a/benchmarks/data/csdid_golden_values.json b/benchmarks/data/csdid_golden_values.json index 73f54ca2..99950a88 100644 --- a/benchmarks/data/csdid_golden_values.json +++ b/benchmarks/data/csdid_golden_values.json @@ -97,6 +97,24 @@ "time": [2, 3, 4, 2, 3, 4, 2, 3, 4], "att": [0.93713345895, 1.1102267341, 0.68521672467, -0.67162819411, 0.86720176282, 0.35233061573, -0.58336476425, -0.10949916241, 0.24708100638], "se": [0.28869526544, 0.45992762299, 0.6484493646, 0.2842568482, 0.26539204819, 0.44734379608, 0.27944370832, 0.28444193554, 0.30682305369] + }, + "simple": { + "att": 0.68857256467, + "se": 0.31382230012 + }, + "group": { + "overall_att": 0.58154811452, + "overall_se": 0.28417155702, + "att": [0.91085897258, 0.60976618928, 0.24708100638], + "se": [0.44399702904, 0.33782835505, 0.30682305369], + "egt": [2, 3, 4] + }, + "dynamic": { + "overall_att": 0.68885444012, + "overall_se": 0.38935416434, + "att": [-0.58336476425, -0.40369753414, 0.6831798871, 0.69816670859, 0.68521672467], + "se": [0.27944370832, 0.22022753056, 0.2015332469, 0.38903747086, 0.6484493646], + "egt": [-2, -1, 0, 1, 2] } } }, diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 98fb462d..3177f5fb 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -541,6 +541,12 @@ Aggregations: `pscore_trim` (R drops at `trim.level=0.995`; differs only at extreme propensities); no-covariate ipw is treated as unconditional (R fits an intercept-only logit whose estimation effect is identically zero in the IF, so this is presentation-only). + **Note:** decided document-only (2026-07-07) — the intercept-only logit is deliberately + NOT mirrored structurally: its estimation-effect correction is identically zero at the + MLE, so mirroring would add a per-cell IRLS solve (and its non-convergence failure + surface) for zero numerical change. No-covariate ipw/reg/dr all reduce to the same + difference-in-means IF, bit-identical per cell (locked by + `tests/test_methodology_callaway.py::TestDRNoCovariateSEUniformity::test_ipw_no_cov_per_cell_identical_to_reg`). DR's no-covariate per-cell SE now also uses the IF-based `sqrt(sum(phi^2))` form (it had lagged on the ddof=1 plug-in `sqrt(var_t/n_t + var_c/n_c)`, O(1/n) from R): without covariates DR reduces to difference in means, so its per-cell SE is now bit-identical to diff --git a/tests/test_csdid_ported.py b/tests/test_csdid_ported.py index b829e1a4..ff6f473b 100644 --- a/tests/test_csdid_ported.py +++ b/tests/test_csdid_ported.py @@ -1484,19 +1484,22 @@ def test_golden_reg_aggregation_se(self, golden_values): def test_golden_ipw_aggregation_se_vs_r_did_251(self, golden_values): """Aggregated ipw SEs match R did 2.5.1 ``aggte`` on the golden data. - The golden JSON carries no aggregation blocks for the ipw scenario, - so the expected values are hardcoded from a fresh R run (did 2.5.1, - DRDID 1.3.0, ``bstrap=FALSE``, computed 2026-07-05 on the EXACT - ``with_covariates_ipw`` fixture data; see - benchmarks/R/generate_csdid_test_values.R which will fold these into - the JSON on its next regeneration). Discriminating: the pre-fix - Python simple SE was 0.32125722 (missing PS estimation-effect - correction, ~2.4% off). Observed post-fix agreement ~1e-10; 1e-6 - covers the fixtures' 8-decimal quantization + IRLS-vs-glm headroom. + The expected values are read from the golden JSON's ipw aggregation + blocks (``simple`` / ``dynamic`` / ``group``), folded in on the + 2026-07-07 regeneration (did 2.5.1 / DRDID 1.3.0, ``bstrap=FALSE``; + identical to the previously hardcoded 2026-07-05 yardsticks — see + benchmarks/R/generate_csdid_test_values.R). Discriminating: the + pre-fix Python simple SE was 0.32125722 (missing PS + estimation-effect correction, ~2.4% off). Observed post-fix + agreement ~1e-10; 1e-6 covers the fixtures' quantization + + IRLS-vs-glm headroom. """ if "with_covariates_ipw" not in golden_values: pytest.skip("Scenario not in golden values") scenario = golden_values["with_covariates_ipw"] + r_results = scenario["results"] + if "simple" not in r_results: + pytest.skip("ipw aggregation blocks not in golden values (pre-2026-07 fixture)") data = _golden_to_df(scenario["data"]) cs = CallawaySantAnna(estimation_method="ipw") results = cs.fit( @@ -1509,28 +1512,20 @@ def test_golden_ipw_aggregation_se_vs_r_did_251(self, golden_values): aggregate="all", ) # R did 2.5.1: aggte(type="simple", bstrap=FALSE, cband=FALSE) - assert abs(results.overall_att - 0.68857256) < 1e-6 - assert abs(results.overall_se - 0.31382230) < 1e-6 + assert abs(results.overall_att - r_results["simple"]["att"]) < 1e-6 + assert abs(results.overall_se - r_results["simple"]["se"]) < 1e-6 # aggte(type="dynamic"): (att.egt, se.egt) per event time - r_dynamic = { - -2: (-0.58336476, 0.27944371), - -1: (-0.40369753, 0.22022753), - 0: (0.68317989, 0.20153325), - 1: (0.69816671, 0.38903747), - 2: (0.68521672, 0.64844936), - } - for e, (r_att, r_se) in r_dynamic.items(): - es = results.event_study_effects[e] + r_dyn = r_results["dynamic"] + assert len(r_dyn["egt"]) > 0 + for e, r_att, r_se in zip(r_dyn["egt"], r_dyn["att"], r_dyn["se"]): + es = results.event_study_effects[int(e)] assert abs(es["effect"] - r_att) < 1e-6, f"ipw dyn ATT(e={e})" assert abs(es["se"] - r_se) < 1e-6, f"ipw dyn SE(e={e})" # aggte(type="group"): (att.egt, se.egt) per cohort - r_group = { - 2: (0.91085897, 0.44399703), - 3: (0.60976619, 0.33782836), - 4: (0.24708101, 0.30682305), - } - for g, (r_att, r_se) in r_group.items(): - grp = results.group_effects[g] + r_grp = r_results["group"] + assert len(r_grp["egt"]) > 0 + for g, r_att, r_se in zip(r_grp["egt"], r_grp["att"], r_grp["se"]): + grp = results.group_effects[int(g)] assert abs(grp["effect"] - r_att) < 1e-6, f"ipw group ATT(g={g})" assert abs(grp["se"] - r_se) < 1e-6, f"ipw group SE(g={g})" diff --git a/tests/test_methodology_callaway.py b/tests/test_methodology_callaway.py index a6c6f323..1db5dffa 100644 --- a/tests/test_methodology_callaway.py +++ b/tests/test_methodology_callaway.py @@ -422,6 +422,38 @@ def test_dr_no_cov_aggregated_se_matches_reg(self): else: np.testing.assert_allclose(dr_es["se"], reg_es["se"], rtol=0, atol=1e-12) + def test_ipw_no_cov_per_cell_identical_to_reg(self): + """No-covariate ipw is bit-identical to reg per-cell (effect AND SE). + + Locks the documented decision (REGISTRY § CallawaySantAnna) that the + no-covariate ipw branch treats the propensity as unconditional and + does NOT structurally mirror R ``did``'s intercept-only logit: the + logit's estimation-effect correction is identically zero at the MLE, + so both implementations reduce to the same difference-in-means IF — + mirroring would add a per-cell IRLS solve (and its failure surface) + for zero numerical change. If this test ever breaks, the ipw no-cov + branch has diverged from the diff-in-means contract and the REGISTRY + note must be revisited.""" + data = generate_staggered_data(n_units=120, n_periods=6, never_treated_frac=0.3, seed=7) + reg = self._fit(data, "reg", aggregate="event_study") + ipw = self._fit(data, "ipw", aggregate="event_study") + + assert set(reg.group_time_effects) == set(ipw.group_time_effects) + compared = 0 + for key, reg_cell in reg.group_time_effects.items(): + ipw_cell = ipw.group_time_effects[key] + if not np.isfinite(reg_cell["se"]): + assert not np.isfinite(ipw_cell["se"]) + continue + np.testing.assert_allclose(ipw_cell["effect"], reg_cell["effect"], rtol=0, atol=1e-12) + np.testing.assert_allclose(ipw_cell["se"], reg_cell["se"], rtol=0, atol=1e-12) + compared += 1 + assert compared >= 3, f"expected several finite cells to compare, got {compared}" + + # Aggregated surfaces inherit the identity (same IF feeds aggregation). + np.testing.assert_allclose(ipw.overall_att, reg.overall_att, rtol=0, atol=1e-12) + np.testing.assert_allclose(ipw.overall_se, reg.overall_se, rtol=0, atol=1e-12) + # ============================================================================= # Phase 2: R Benchmark Comparison Tests