perf(process-storms): rolling-window AORC scan, read each year once (~3-5x, opt-out)#16
Open
nghiemv wants to merge 1 commit into
Open
perf(process-storms): rolling-window AORC scan, read each year once (~3-5x, opt-out)#16nghiemv wants to merge 1 commit into
nghiemv wants to merge 1 commit into
Conversation
nghiemv
marked this pull request as draft
July 14, 2026 21:24
nghiemv
marked this pull request as ready for review
July 15, 2026 18:46
nghiemv
force-pushed
the
perf/rolling-scan
branch
from
July 16, 2026 14:40
82c6daa to
3a20e63
Compare
nghiemv
force-pushed
the
perf/rolling-scan
branch
2 times, most recently
from
July 16, 2026 15:09
2c8d4e3 to
7b620a9
Compare
The upstream scan re-reads each AORC hour once per overlapping storm window
(storm_duration / check_every_n_hours times: 3x at 72h/24h, ~12x at 72h/6h).
The rolling-window scan sweeps each year ONCE: keep a running cumulative sum,
snapshot it at the indices each storm-date needs, so a window sum is
snapshots[end] - snapshots[start]. Complexity O(area x D x num_dates) ->
O(area x T); peak memory bounded to ~one time-chunk + the snapshot pool.
Ported from the fork's cumsum_scan, renamed rolling (CC_ROLLING_SCAN,
CC_ROLLING_CHUNK_HOURS). Monkey-patches collect_event_stats via
install()/restore() around new_collection, gated on CC_ROLLING_SCAN (default
on; 0 = upstream per-date path).
Validated end-to-end on real AORC against the real indian-creek domain
(fetched geometry, fast aorc-cache-conus, 2-year window, full pipeline):
- Parity: rolling vs baseline storm-stats.csv = 2925/2925 rows byte-exact
(min/mean/max/x/y), 0 diffs. Numerical core also unit-tested against naive
window sums (test/test_rolling_scan.py).
- Perf: rolling 585.8s vs baseline 3137.7s total (~5.4x at 72h/6h).
Default-on is safe: parity is bit-identical, so it changes speed, not results.
nghiemv
force-pushed
the
perf/rolling-scan
branch
from
July 16, 2026 15:10
7b620a9 to
6daa0fb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The upstream scan re-reads each AORC hour once per overlapping storm window
(
storm_duration / check_every_n_hourstimes — 3× at 72h/24h, ~12× at 72h/6h).The rolling-window scan sweeps each year once: keep a running cumulative
sum, snapshot at the indices each storm-date needs, so a window sum is
snapshots[end] - snapshots[start]. ComplexityO(area × D × num_dates)→O(area × T); peak memory bounded to ~one time-chunk + the snapshot pool. Thespeedup scales with cadence: ~3× at 72h/24h, ~5× measured at 72h/6h.
Ported from the fork's
cumsum_scan, renamed rolling. Monkey-patchescollect_event_statsviainstall()/restore()aroundnew_collection, gatedon
CC_ROLLING_SCAN.Default-on: yes, with confidence
Shipped default-on (
CC_ROLLING_SCAN=1). This is justified, not aspirational:end-to-end parity is confirmed bit-identical on a real production domain (see
below), so default-on changes speed, not results.
CC_ROLLING_SCAN=0remains aone-flag escape hatch to the exact upstream per-date path.
Validation (end-to-end, real AORC) — ✅ confirmed
Real indian-creek production domain (fetched geometry), fast
aorc-cache-conus, 2-year window, full pipeline (scan → DSS → grid → upload):CC_ROLLING_SCAN=0)storm-stats.csv: 2925 / 2925 rows byte-exact (min/mean/max/x/y), 0 diffs.test/test_rolling_scan.pyparity-tests the numerical core againstnaive window sums (chunk-size invariant, NaN-as-zero, toggle). 4 pass.
CI gates (verified locally on this rebased branch)
ruff check src/✅ ·ruff format --check src/✅ ·docker compose build✅ ·built image imports
rolling_scan,enabled()→True.Reviewer notes
mainas a single commit; diff isrolling_scan.py(+680) + 8-line wiring in
process_storms.py+ the test. No deletions; noconflict with the merged pre-flight (feat(process-storms): fail-fast AORC pre-flight before the scan #15) or cache-dir attr (Implement cache directory attribute for plugin #17).
CC_ROLLING_SCAN=1then=0and diff the collection's
storm-stats.csv— rows are byte-identical.CC_ROLLING_SCAN10= upstream per-date loopCC_ROLLING_CHUNK_HOURS720