perf(efficient_did): omega_ridge stabilization + fused tiled GEMM analytical path (43x)

[igerber]

Summary

• omega_ridge parameter (default 1e-6): ridge-regularizes the Omega* inversion behind the efficient weights — (Omega* + omega_ridge * max(trace/H, 0) * I) x = 1 — replacing the pseudoinverse of the numerically singular Omega* that PT-All's telescoping overidentified moments produce (measured cond 1e17–1e22 for 100% of units). Per-cell ATT(g,t) becomes numerically well-defined and platform-stable: 1-ulp input perturbation moves per-cell values ≤3e-9 relative (previously ~1e-4 through the pseudoinverse's rcond-cutoff cliff). omega_ridge=0 restores the entire legacy code path bit-for-bit.
• Fused unit-tiled GEMM rewrite of the conditional analytical path: kernel-covariance tables via the GEMM identity KCov = W@(A0*B0) − (W@A0)*(W@B0) with (t_pre_j, t_pre_k) dedup (1,830 pairs → ~190 columns), per-group kernel matrices computed once per tile and reused across all cells, batched ridge solves replacing the per-unit SVD + pseudoinverse loop, memory bounded by _TARGET_OMEGA_TILE_BYTES (256 MB). nocov twin: Gram-form Omega* + the same ridge on global weights.
• Measured (3-rep medians, 20 periods, 5 cohorts, 5 covariates): conditional fit 35.9s→1.8s at 500 units (20x), 95.4s→3.5s at 1k (27x), 337.3s→7.8s at 2k (43x), 57.7s at 10k (~2.3h extrapolated before); a 100k-unit fit completes in ~86 min at 4.0 GB peak (previously memory-killed). Survey-weighted 1k: 95.3s→3.3s.
• Degenerate self-pair exclusion (ridge path): the pre-treatment (g'=g, t_pre=t) moment is identically zero by construction (the exact-null Omega* direction); dropping it keeps pre-treatment placebos data-driven (the legacy pseudoinverse truncation had turned them into ±5%-of-effect-size noise; a naive ridge would have made them deterministically zero and disabled the pre-trend diagnostic).
• Warning consolidation: legacy per-cell "using pseudoinverse" warnings (fired on essentially every PT-All nocov cell) become ONE fit-level warning (count + max cond); the conditional-path scalability warning moves from per-cell at n>5,000 to once-per-fit at n>50,000.

Methodology references (required if estimator / math changes)

• Method name(s): EfficientDiD (Chen, Sant'Anna & Xie 2025) — Omega* inversion behind the efficient weights (Eq 3.5/3.13/4.3), conditional Omega*(X) (Eq 3.12), no-covariates Omega* and EIF-based inference
• Paper / source link(s): Chen, Sant'Anna & Xie (2025), "Efficient Difference-in-Differences and Event Study Estimators" (see docs/methodology/REGISTRY.md EfficientDiD section)
• Any intentional deviations from the source (and why): the omega_ridge regularization and the degenerate self-pair exclusion are documented in the new REGISTRY - **Note:** (Omega* ridge regularization). The paper assumes Omega* is invertible and does not prescribe finite-sample handling of numerical singularity; any fixed weights summing to 1 keep the estimator consistent (each moment identifies ATT(g,t)), the plug-in EIF treatment of estimated weights (Remark 4.2) is unaffected, and the default was calibrated against the paper's own HRS Table 6 empirical application (all anchors unchanged, worst deviation 0.0257 SE, shift ≤0.0001 SE) plus Monte Carlo (bias/RMSE/SE-calibration/coverage statistically identical to legacy). One-time value shift on the covariate path is called out in CHANGELOG (worst post cell ~0.6 of its own SE at n=500, shrinking with n).

Validation

• Tests added/updated: tests/test_efficient_did.py (TestOmegaRidge: param surface/validation/transactional set_params, aggregate-warning payload vs legacy per-cell, ridge-engaged lock, degenerate-pair drop, data-driven placebos; TestFusedConditionalPath: dense-reference oracle, tile-forced twins incl. survey weights with a multi-tile execution proof, 1-ulp stability contracts for both paths), tests/test_methodology_efficient_did.py (TestOmegaRidgeWeights hand-computed/scale-equivariance/dose-response/uniform-fallbacks, TestNoCovOmegaGram loop-parity incl. n<2 / sum(w)≤1 / zero-weight-cohort / H=0 guards + fit-level twin, TestKernelCovBatch GEMM-identity oracle weighted+unweighted+row-tile). Existing suites green on both backends; full suite 8,466 passed (single failure = known pre-existing order-dependent test_survey_dcdh item, passes standalone, TODO.md row from fix(staggered): CS panel reg/ipw influence functions + per-cell SEs to DRDID parity #619).
• Backtest / simulation / notebook evidence (if applicable): HRS Table 6 replication anchors unchanged at every candidate lambda; Monte Carlo (2 covariate-confounded configs, true ATT known) legacy vs ridge statistically identical; docs/tutorials/15_efficient_did.ipynb executes clean; measured perf arms in docs/performance-plan.md.

Security / privacy

• Confirm no secrets/PII in this PR: Yes

---

Generated with Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01X4AzrFUMqJxcUumSH31mSr

[github-actions]

Overall Assessment

✅ Looks good. No unmitigated P0/P1 findings.

Executive Summary

• Affected method: EfficientDiD, specifically Omega* weighting/inversion, no-covariate Gram Omega*, and covariate fused tiled analytical path.
• The ridge regularization and degenerate self-pair exclusion are explicitly documented in docs/methodology/REGISTRY.md:L1243-L1252; per review rules, these are not methodology defects.
• The fused GEMM/Gram rewrites preserve the documented five-term Omega* structure and use safe_inference() for inference fields.
• New omega_ridge propagation is present in constructor, validation, get_params(), results, and fit paths.
• I could not run tests locally because this environment lacks pytest and numpy.

Methodology

• Severity: P3 informational
  Impact: omega_ridge changes the finite-sample Omega* inversion from the paper’s inverse-based efficient weight expression, and the ridge path drops the pre-treatment (g'=g, t_pre=t) self-pair. The cited paper derives efficient EIF-based estimators using overidentifying pre-treatment/comparison-group information; the registry documents this PR’s ridge as a finite-sample regularization in the paper’s singular-Omega gap. (arxiv.org)
  Concrete fix: No action required. The deviation is documented with a **Note:** in docs/methodology/REGISTRY.md:L1251-L1252, and implementation follows that note at diff_diff/efficient_did.py:L939-L952, diff_diff/efficient_did_weights.py:L459-L534, and diff_diff/efficient_did_covariates.py:L1385-L1482.

Code Quality

• Severity: None
  Impact: No new inline inference anti-pattern found. Per-cell inference routes through safe_inference() in diff_diff/efficient_did.py:L855-L875, and bootstrap overrides update t-stat via safe_inference() rather than ad hoc effect / se.
  Concrete fix: None.

Performance

• Severity: P3 informational
  Impact: The remaining very-large-n conditional-path optimization is explicitly tracked in TODO.md:L50; this is deferrable performance work, not a correctness blocker.
  Concrete fix: No action required for this PR.

Maintainability

• Severity: None
  Impact: The fused path is centralized in compute_conditional_cells_tiled() and the no-covariate fast path is isolated in _omega_star_nocov_gram(), with omega_ridge=0 preserving the legacy code path selection at diff_diff/efficient_did.py:L1048-L1063 and diff_diff/efficient_did.py:L1116-L1138.
  Concrete fix: None.

Tech Debt

• Severity: P3 informational
  Impact: Cross-cell kernel-table hoisting remains documented as future work in TODO.md:L50. This is properly tracked and does not introduce silent wrong results.
  Concrete fix: No action required.

Security

• Severity: None
  Impact: No secrets, external calls, unsafe deserialization, or credential handling changes found in the diff.
  Concrete fix: None.

Documentation/Tests

• Severity: P3 informational
  Impact: Documentation and methodology notes were updated, and tests were added for ridge behavior, fused conditional equivalence, tiling, survey weights, and 1-ulp stability. I could not execute them here because pytest and numpy are unavailable in the review environment.
  Concrete fix: No PR change required; CI should run the added suites.