Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the legacy pseudoinverse path as described under `omega_ridge` above; at `omega_ridge=0`
results are bit-identical to the previous release. The conditional-path scalability warning
now fires once per fit at n > 50,000 (previously per cell at n > 5,000).
- **`EfficientDiD` conditional-path kernel-covariance tables hoisted across `(g,t)` cells**
(follow-up to the fused tiled pass above). Every Omega* term is a kernel covariance keyed only
by outcome columns, so the tables the tiled pass rebuilt per cell dedup to one table of
distinct product columns per comparison group per unit-tile (~26x fewer kernel GEMM columns
on a PT-All fit); kernel weight matrices are built and freed one group at a time, so the tile
memory budget is governed by the largest single group instead of the sum (proportionally
fatter tiles at large n); each cell's Omega* is gathered from the group tables through
compact per-cell column slices in the legacy per-entry operation order (value-exact, locked
by test). Measured (3-rep medians, 20 periods, 5 cohorts, 5 covariates, `aggregate="all"`):
fit 7.7s → 7.0s at 2k units, 57.2s → 35.7s at 10k (1.6x), 358s → 129s at 30k (2.8x), and
the 100k-unit fit drops from ~86 min to **17.8 min (~4.8x)** at the same ~4 GB peak RSS;
the kernel-covariance stage itself is 40x faster (21.2s → 0.5s at 10k). Results move only at
floating-point reassociation level (post-treatment cells ~1e-12 relative, overall ATT ~1e-13);
the no-covariates path is byte-identical and `omega_ridge=0` still routes the entire legacy
path. A `pt_assumption="post"` covariate fit builds no tables at all (all cells are
just-identified), keeping that path regression-free.
- **CallawaySantAnna multiplier bootstrap rewritten as a fused, column-tiled scatter-GEMM**
(EfficientDiD's bootstrap routes through the same kernel). The former loop sliced the
`(block × n_units)` weight matrix twice per (g,t) cell per weight block — at a 40-period,
Expand Down
2 changes: 1 addition & 1 deletion TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m

| Issue | Location | Origin | Effort | Priority |
|-------|----------|--------|--------|----------|
| `EfficientDiD` conditional path at very large n: the fused tiled GEMM rewrite (v3.7; 43x at 2k units, 100k now completes in ~86 min) computes kernel-covariance tables per cell per tile. Cross-cell table hoisting — Term 5 tables are (g,t)-independent and Term 2 tables depend only on t — would cut the intrinsic per-cell GEMM factor by ~10x at 100k-unit scale if practitioners need routine 100k covariate fits. Follow-up lever from the v3.7 flop analysis; not needed below ~50k units. | `efficient_did_covariates.py::compute_conditional_cells_tiled` | CS-scaling | Mid | Low |
| `EfficientDiD` conditional path: post-hoisting (2026-07, 100k fit ~18 min) the top stage at every scale is `_ridge_solve_weights` — batched `np.linalg.solve` LU over `(n × H × H)` stacks (17.2s of a 35.7s fit at 10k units, ~170s at 100k; linear in n). The matrices are SPD after the ridge, so a batched Cholesky (potrf ~1/2 the LU flops) in the Rust backend — which could also thread across the stack — is the natural lever; numpy has no batched triangular solve, so pure-Python gains are limited. | `efficient_did_covariates.py::_ridge_solve_weights`, `rust/src/` | CS-scaling | Mid | Low |
| `ImputationDiD` dense `(A0'A0).toarray()` scales `O((U+T+K)^2)` — OOM risk on large panels (only triggers when the sparse solver fails). Needs an alternative dense fallback or richer sparse strategy. | `imputation.py` | #141 | Heavy | Medium |
| CR2 Bell-McCaffrey DOF uses a naive `O(n²k)` per-coefficient loop over cluster pairs; Pustejovsky-Tipton (2018) Appendix B has a scores-based formulation avoiding the full `n×n` `M`. Switch when a user hits a large-`n` cluster-robust design. | `linalg.py::_compute_cr2_bm` | Phase 1a | Heavy | Low |
| Rust-backend HC2: the Rust path only supports HC1; HC2 and CR2 Bell-McCaffrey fall through to NumPy. Noticeable for large-`n` fits. | `rust/src/linalg.rs` | Phase 1a | Mid | Low |
Expand Down
Loading
Loading