Skip to content

feat(resume): durable local_root + resume the scan instead of restarting it#14

Closed
nghiemv wants to merge 1 commit into
USACE-Cloud-Compute:mainfrom
nghiemv:feat/durable-resume
Closed

feat(resume): durable local_root + resume the scan instead of restarting it#14
nghiemv wants to merge 1 commit into
USACE-Cloud-Compute:mainfrom
nghiemv:feat/durable-resume

Conversation

@nghiemv

@nghiemv nghiemv commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

A pod OOMKill discarded the whole run: local_root was Path("Local") on the
ephemeral overlay, so the catalog, the flushed storm-stats.csv scan progress, and
the DSS grids died with the pod. The .checkpoint skip-list also skipped
process-storms on resume, leaving ctx["collection"] unset and crashing downstream.

Changes:

  • local_root honors CC_LOCAL_ROOT (point at the /model PVC); unset keeps the
    Local default.
  • process-storms resume ladder: reload complete catalog → resume_collection over
    only missing dates → fresh search. A mid-scan OOM resumes where it died.
  • Remove the .checkpoint skip-list. Actions are idempotent against local_root, so
    a resumed run re-runs from the top; process-storms always runs, so
    ctx["collection"] is always repopulated.

Deploy pairing (required on-cluster): set CC_LOCAL_ROOT to the durable /model
PVC and add an Argo retryStrategy. The mount must be block, not object-FUSE —
multi_processor fsyncs the CSV per date.

Happy path unchanged; resume adds seconds to save hours. New
test/test_process_storms_resume.py covers the partial-scan detector.

@nghiemv
nghiemv marked this pull request as draft July 14, 2026 20:29
A pod OOMKill discarded the whole run: local_root was Path("Local") on the
ephemeral overlay, so the catalog, the flushed storm-stats.csv scan progress,
and the DSS grids died with the pod. The .checkpoint skip-list also skipped
process-storms on resume, leaving ctx["collection"] unset and crashing
downstream.

Durability:
  - local_root honors CC_LOCAL_ROOT (point at the /model PVC); unset keeps the
    Local default. A startup preflight fails fast if the mount isn't writable
    and warns if it's object-storage FUSE (stormhub fsyncs the CSV per date).
  - the .checkpoint skip-list is removed; resume is a property of the idempotent
    actions, so process-storms always runs and always repopulates ctx.

Resume is validated, not merely possible (new scan_state module):
  - Param fingerprint — a scan is only reused when the parameters that decide
    which dates are searched and how events are ranked are unchanged. A changed
    request quarantines the stale state and rebuilds, instead of silently
    resuming onto a catalog built for different parameters.
  - Completion sentinel — a catalog is only reloaded ("skip the search") once
    the whole stage finished. A pod that died during item creation left a
    partial collection; without the sentinel we resume rather than ship a
    truncated top-N.
  - CSV repair — a pod killed mid-write can leave a torn final row; keep every
    well-formed row and drop the rest (atomic rewrite) so the missing-date diff
    can't miscompute or crash.
  - Resume degrades to fresh — a resume that fails for any non-OOM reason
    quarantines the partial state and rebuilds once, so a resume-path fault
    never strands a runnable job. OOM is re-raised with guidance.
  - Quarantine, never delete — rejected state is moved aside for debugging.

process-storms resume ladder: reload complete catalog -> resume_collection over
only the missing dates -> fresh search, each rung guarded as above.

Pairs with an Argo retryStrategy + a durable, block (not FUSE) CC_LOCAL_ROOT.

Tests: scan_state units (fingerprint/sentinel/quarantine/CSV repair) and
process_storms flow tests (fresh / reload / drift->quarantine->fresh / resume /
resume-fail->fresh / incomplete-not-reloaded) under a stubbed stormhub.
@nghiemv
nghiemv force-pushed the feat/durable-resume branch from f75a980 to dcd9459 Compare July 14, 2026 20:37
@nghiemv

nghiemv commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Closing -- might prefer fail-fast instead of resume

@nghiemv nghiemv closed this Jul 14, 2026
@nghiemv

nghiemv commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Closing — declining in favor of the performance direction.

Stateful resume is a bridge for when the scan is long and failure-prone. That premise no longer holds once the shippable defaults land:

With the scan fast, fail-fast, and OOM-safe, re-running from scratch is cheap — so the resume machinery here (durable checkpoint, param-fingerprint, quarantine, CSV repair) is complexity that isn't warranted. Superseded, not lost: reopen if a future long/monolithic stage brings the premise back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant