feat(resume): durable local_root + resume the scan instead of restarting it#14
Closed
nghiemv wants to merge 1 commit into
Closed
feat(resume): durable local_root + resume the scan instead of restarting it#14nghiemv wants to merge 1 commit into
nghiemv wants to merge 1 commit into
Conversation
nghiemv
marked this pull request as draft
July 14, 2026 20:29
A pod OOMKill discarded the whole run: local_root was Path("Local") on the
ephemeral overlay, so the catalog, the flushed storm-stats.csv scan progress,
and the DSS grids died with the pod. The .checkpoint skip-list also skipped
process-storms on resume, leaving ctx["collection"] unset and crashing
downstream.
Durability:
- local_root honors CC_LOCAL_ROOT (point at the /model PVC); unset keeps the
Local default. A startup preflight fails fast if the mount isn't writable
and warns if it's object-storage FUSE (stormhub fsyncs the CSV per date).
- the .checkpoint skip-list is removed; resume is a property of the idempotent
actions, so process-storms always runs and always repopulates ctx.
Resume is validated, not merely possible (new scan_state module):
- Param fingerprint — a scan is only reused when the parameters that decide
which dates are searched and how events are ranked are unchanged. A changed
request quarantines the stale state and rebuilds, instead of silently
resuming onto a catalog built for different parameters.
- Completion sentinel — a catalog is only reloaded ("skip the search") once
the whole stage finished. A pod that died during item creation left a
partial collection; without the sentinel we resume rather than ship a
truncated top-N.
- CSV repair — a pod killed mid-write can leave a torn final row; keep every
well-formed row and drop the rest (atomic rewrite) so the missing-date diff
can't miscompute or crash.
- Resume degrades to fresh — a resume that fails for any non-OOM reason
quarantines the partial state and rebuilds once, so a resume-path fault
never strands a runnable job. OOM is re-raised with guidance.
- Quarantine, never delete — rejected state is moved aside for debugging.
process-storms resume ladder: reload complete catalog -> resume_collection over
only the missing dates -> fresh search, each rung guarded as above.
Pairs with an Argo retryStrategy + a durable, block (not FUSE) CC_LOCAL_ROOT.
Tests: scan_state units (fingerprint/sentinel/quarantine/CSV repair) and
process_storms flow tests (fresh / reload / drift->quarantine->fresh / resume /
resume-fail->fresh / incomplete-not-reloaded) under a stubbed stormhub.
nghiemv
force-pushed
the
feat/durable-resume
branch
from
July 14, 2026 20:37
f75a980 to
dcd9459
Compare
Contributor
Author
|
Closing -- might prefer fail-fast instead of resume |
Contributor
Author
|
Closing — declining in favor of the performance direction. Stateful resume is a bridge for when the scan is long and failure-prone. That premise no longer holds once the shippable defaults land:
With the scan fast, fail-fast, and OOM-safe, re-running from scratch is cheap — so the resume machinery here (durable checkpoint, param-fingerprint, quarantine, CSV repair) is complexity that isn't warranted. Superseded, not lost: reopen if a future long/monolithic stage brings the premise back. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pod OOMKill discarded the whole run:
local_rootwasPath("Local")on theephemeral overlay, so the catalog, the flushed
storm-stats.csvscan progress, andthe DSS grids died with the pod. The
.checkpointskip-list also skippedprocess-stormson resume, leavingctx["collection"]unset and crashing downstream.Changes:
local_roothonorsCC_LOCAL_ROOT(point at the/modelPVC); unset keeps theLocaldefault.process-stormsresume ladder: reload complete catalog →resume_collectionoveronly missing dates → fresh search. A mid-scan OOM resumes where it died.
.checkpointskip-list. Actions are idempotent againstlocal_root, soa resumed run re-runs from the top;
process-stormsalways runs, soctx["collection"]is always repopulated.Deploy pairing (required on-cluster): set
CC_LOCAL_ROOTto the durable/modelPVC and add an Argo
retryStrategy. The mount must be block, not object-FUSE —multi_processorfsyncs the CSV per date.Happy path unchanged; resume adds seconds to save hours. New
test/test_process_storms_resume.pycovers the partial-scan detector.