Skip to content

fix: reconcile interrupted relaunch tasks - #73

Merged
sjug merged 1 commit into
quadsproject:mainfrom
sjug:fix-reconcile-interrupted-relaunch-task
Aug 6, 2026
Merged

fix: reconcile interrupted relaunch tasks#73
sjug merged 1 commit into
quadsproject:mainfrom
sjug:fix-reconcile-interrupted-relaunch-task

Conversation

@sjug

@sjug sjug commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Mark stale provisioning tasks failed when startup reconciles an interrupted managed llama.cpp relaunch.
  • Repair both newly transitioned relaunching nodes and existing relaunch_failed records, covering gateways that crash between node and task persistence.
  • Preserve the task start time and unknown future fields while recording the interrupted step and a teardown-required error.
  • Use a one-shot modification-revision CAS so concurrent or newer task records are never overwritten.

Safety

  • Complete terminal node reconciliation before task repair, so task read, parse, write, or CAS failures cannot weaken the fail-closed node state.
  • Skip unrelated and already-terminal task steps, preserving more specific completion or failure records.
  • Keep task records persistent with an unleased CAS write, matching the existing /provisioning/{hostname} lifetime.

Validation

  • Python 3.12: 1,449 passed.
  • Python 3.13: 1,449 passed.
  • Coverage: 92.57% against the 92% floor.
  • Interrupted relaunch recovery: 100% statement and branch coverage.
  • mypy: 142 source files, no errors.
  • ruff check and format, shellcheck, JavaScript syntax checks, and git diff --check: clean.

Live finding

The Phase 3 hard-kill staging test correctly produced a persistent, teardown-only relaunch_failed node but left its provisioning task at starting_llamacpp. This follow-up makes the next gateway startup reconcile that existing record to a failed task with the interrupted step.

Mark the stale provisioning task failed when startup recovers an
interrupted llama.cpp relaunch. Record the interrupted step and a clear
teardown-required error so the dashboard does not show phantom work.

Repair tasks for both newly reconciled relaunching nodes and existing
relaunch_failed records. Preserve task start time and unknown fields,
and use a one-shot modification-revision CAS so newer task state is
never overwritten.

Keep task repair subordinate to the terminal node transition, document
the startup behavior, and cover malformed records, write failures, CAS
conflicts, terminal tasks, and rolling-upgrade fields.
@sjug
sjug force-pushed the fix-reconcile-interrupted-relaunch-task branch from 12020c9 to 58558a3 Compare August 6, 2026 02:22
@sjug
sjug merged commit f32f0c4 into quadsproject:main Aug 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant