fix(supervisor): make a watchdog restart resumable, bounded, and sized by the backend it interrupts

CHANGELOG.md

@@ -47,6 +47,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
   third-party provider plugins published before the rename are still
   discovered, and it is not a CLI surface.
 
+### Fixed
+
+- **Supervisor watchdog restarts are resumable and bounded.** A wedged
+  coordinator receives SIGHUP, preserving the interrupted phase segment without
+  creating a session outcome; repeated wedges become terminal after three
+  restart attempts, counted durably before the signal goes out. Inline role
+  turns now share an explicit total wall-clock timeout, independent of backend
+  streamed-message idle timeouts and retries. Reactor-stage boundaries refresh
+  supervisor progress, while a stage cancelled at the total timeout counts toward
+  the crash emergency stop. A failed restart-counter write refuses the resumable
+  restart and sends SIGTERM, and a cleanly ended leg can resume as soon as its
+  owner pid is gone instead of being held alive by the final state write.
+- **The robustness monitor reads the real stop-reason vocabulary.** It imported
+  a module that does not exist and silently fell back to a subset missing 16
+  terminal reasons, so a finished session could be relaunched.
+
 ## [v1.1.0] - 2026-09-09
 Current packaged version (`pyproject.toml`). See
 [release notes](docs/release-notes.md) and the

docs/reference/environment-variables.md

@@ -681,9 +681,10 @@ guarantee that varies silently by host is not a guarantee.
 | Variable | Default | Description |
 |---|---|---|
 | `HYPERLOOM_REAP_BACKEND` | `process_group` | Which unit ends a bring-up round's processes: `process_group`, `cgroup` or `container`. Only `cgroup` and `container` produce a reap that is *proof* the tree is gone — the kernel (or the container runtime) owns the membership list, so nothing can leave it by forking or re-parenting. `process_group` reaches only what it could enumerate from procfs before it signalled. A unit that cannot run on this host falls back to `process_group`, which weakens the claim rather than faking it. |
-| `HYPERLOOM_SUPERVISOR` | `1` | Whether the optimizer starts an out-of-band supervisor process that watches for a coordinator that died or whose tick stopped advancing. |
-| `HYPERLOOM_SUPERVISOR_ENFORCE` | unset (off) | Whether the supervisor may end a process tree. A wedged coordinator is sent SIGTERM regardless — that is the channel its signal drain reads while the loop is busy. Off by default, a coordinator that does not answer that stop, and a dead one's leftovers, are left alone and the refusal is recorded in `runtime/supervisor/status.json`. Ending a tree additionally requires a reap backend whose success is proof, so enforcement with the default `process_group` unit will refuse to kill and say so. |
-| `HYPERLOOM_SUPERVISOR_TICK_STALL_SEC` | half of `--max-hours`, capped at `3600` and floored at `1800` | How long the coordinator's tick may go without advancing before the supervisor calls it wedged. Derived from the session budget so the window always fits inside the run it watches; setting this overrides the derivation. |
+| `HYPERLOOM_SUPERVISOR` | `1` | Whether the optimizer starts an out-of-band supervisor process that watches for a coordinator that died or whose tick stopped making progress. |
+| `HYPERLOOM_SUPERVISOR_ENFORCE` | unset (off) | Whether the supervisor may end a process tree. A wedged coordinator receives SIGHUP for each resumable restart and SIGTERM after the restart limit. Off by default, a coordinator that does not answer that stop, and a dead one's leftovers, are left alone and the refusal is recorded in `runtime/supervisor/status.json`. Ending a tree additionally requires a reap backend whose success is proof, so enforcement with the default `process_group` unit will refuse to kill and say so. |
+| `INFERENCE_OPTIMIZER_REACTOR_TURN_TIMEOUT_SEC` | `1800` | Total wall-clock limit for each reactor stage, including backend startup, streamed output, retries, backoff, and cleanup. This is independent of backend `*_CALL_TIMEOUT_SEC` settings: for streamed Claude turns those settings bound idle time between SDK messages, and activity resets that idle timer. Reaching this total limit cancels the stage and records a crash; a shorter remaining session bound still ends the stage without recording a crash. |
+| `HYPERLOOM_SUPERVISOR_TICK_STALL_SEC` | half of `--max-hours`, capped at the larger of `3600` or the reactor timeout plus `30`, and floored at the reactor timeout plus `30` | How long coordinator progress may remain unchanged before the supervisor calls it wedged. Progress is refreshed at every reactor-stage boundary, so the derived window covers one total reactor timeout plus one supervisor poll rather than the sum of all sequential roles. Raising `INFERENCE_OPTIMIZER_REACTOR_TURN_TIMEOUT_SEC` automatically raises the derived floor and default; an explicit value overrides the derivation and is the operator's responsibility. |
 
 The supervisor never opens `coordinator.db` — it sits on a network filesystem
 where a second writer risks the message bus and the task registry — and never
@@ -693,6 +694,13 @@ writes `reports/final.json` itself, marked `producer: "supervisor"`; that record
 never replaces a full report, and the coordinator's own crash-safe fallback never
 replaces it.
 
+A successful watchdog stop records a leg boundary but no session `stop_reason`
+or final artifacts, allowing the monitor to resume it. The restart count is
+persisted in `runtime/supervisor/status.json`; after three resumable restarts the
+next watchdog request uses SIGTERM and the coordinator follows its normal
+terminal path. The monitor treats `final.json`, `final.md`, a completed CLOSE
+sequence, or a vocabulary stop reason as terminal.
+
 ---
 
 ## Framework / source-tree discovery

src/hyperloom/agents/robustness/role/envelope.py

@@ -68,6 +68,7 @@ class IntentType(str, Enum):
         "current_best",
         "stop_reason",
         "stop_ts",
+        "leg_ended_ts",
         "last_tick_exception",
         "cumulative_gain_validated",
         "cumulative_gain_validated_ts",

src/hyperloom/inference_optimizer/SKILL.md

@@ -761,6 +761,11 @@ for the kernel dispatch and artifact layout.
 
 ### Recovery
 
+The out-of-band supervisor uses SIGHUP for a resumable watchdog restart. This
+records the interrupted leg boundary without producing a stop reason or final
+report; the monitor resumes the same session with `--resume-from`. After three
+watchdog restarts, the next watchdog stop is terminal instead.
+
 If the CLI exits with `Claude SDK exit code 1` or `Primus.00009 token not present`,
 the gateway rejected the request. Check that `OPENAI_BASE_URL` / `OPENAI_API_KEY`
 are set in `.env` (or the calling shell) and that the gateway is reachable:
@@ -1217,10 +1222,10 @@ Three exceptions:
 
 For runs > 5 min, start a monitor in its own `setsid nohup` process. It polls
 `state.json` every 5 min, exits without resuming when the session is terminal
-(any `stop_reason` in `STOP_REASON_VOCAB`, `phase=CLOSE`, or
-`reports/final.md` present — including failure sentinels like
-`baseline_failed`), and resumes via `--resume-from` only when the optimizer
-dies without those markers (unexpected crash).
+(any `stop_reason` in `STOP_REASON_VOCAB`, a completed CLOSE sequence, or a
+`reports/final.json` / `final.md` artifact — including failure sentinels like
+`baseline_failed`), and resumes via `--resume-from` when the optimizer dies
+without those markers, including a resumable watchdog restart.
 
 ```bash
 export RUN_DIR="${USER_DATA_PATH:-/workspace/hyperloom}/optimizer_runs"

src/hyperloom/inference_optimizer/cli/__init__.py

@@ -102,6 +102,7 @@
     default_enabled_actions,
 )
 from hyperloom.orchestrator.supervisor import spawn_supervisor, stop_supervisor
+from hyperloom.orchestrator.supervisor.watch import SUPERVISOR_RESTART_REASON
 
 from ..session.lock import SessionAlreadyRunning, SessionLock
 from ..session.paths import (
@@ -1304,6 +1305,65 @@ def _exit_code_for_stop_reason(stop_reason: str | None) -> int:
     return 0 if (stop_reason or "") in SUCCESS_STOP_REASONS else 1
 
 
+def _write_cli_terminal_artifacts(session_dir: Path, state: SharedState, stop_reason: str | None) -> None:
+    """Write the session's terminal artifacts in their established order.
+
+    A watchdog restart ends a leg, not the session, so it produces none of them
+    -- including the close-out package, which speaks for a finished run.
+    """
+    if stop_reason == SUPERVISOR_RESTART_REASON:
+        return
+    try:
+        from ..breakdown import write_minimal_final_json
+
+        with timed_teardown_step(state, "final_json"):
+            final_json = write_minimal_final_json(session_dir)
+        print(f"Final summary     : {final_json}")
+    except Exception:  # noqa: BLE001
+        log.exception("crash-safe final.json write failed (non-fatal)")
+    if state.close_sequence_done:
+        print("Session breakdown : (already written by CLOSE phase sequencer; skipping cli.finally safety-net write)")
+    else:
+        try:
+            from ..breakdown import write_breakdown_json
+
+            with timed_teardown_step(state, "session_breakdown"):
+                breakdown_path = write_breakdown_json(session_dir)
+            print(f"Session breakdown : {breakdown_path}")
+        except Exception:  # noqa: BLE001
+            log.exception("session_breakdown finalize failed (non-fatal)")
+        try:
+            from ..breakdown import write_minimal_final_report
+
+            with timed_teardown_step(state, "final_md"):
+                final_md = write_minimal_final_report(session_dir)
+            print(f"Final report      : {final_md}")
+        except Exception:  # noqa: BLE001
+            log.exception("emergency final report write failed (non-fatal)")
+    try:
+        from hyperloom.orchestrator.trace.langfuse_emitter import flush_session, record_session_breakdown
+
+        with timed_teardown_step(state, "langfuse"):
+            flush_session(session_dir)
+            from ..breakdown import patch_breakdown_langfuse
+
+            patch_breakdown_langfuse(session_dir)
+            record_session_breakdown(session_dir)
+    except Exception:  # noqa: BLE001
+        log.debug("langfuse flush_session failed", exc_info=True)
+    # Safety net for paths that leave close_sequence_done False and never run
+    # the sequencer; ordered after langfuse so the package carries its patch.
+    try:
+        from ..breakdown import package_session_artifacts
+
+        with timed_teardown_step(state, "artifact_package"):
+            pkg_path = package_session_artifacts(session_dir, session_id=str(getattr(state, "session_id", "") or ""))
+        if pkg_path is not None:
+            print(f"Artifact package  : {pkg_path}")
+    except Exception:  # noqa: BLE001
+        log.exception("session artifact package failed (non-fatal)")
+
+
 def _new_preflight_failure_session_dir(
     args: argparse.Namespace,
     *,
@@ -2306,6 +2366,7 @@ def _backend_kind(role: str) -> str:
         log.warning("supervisor: could not start (%s); the run proceeds unwatched", exc)
         print(f"[supervisor] could not start ({exc}); the run proceeds unwatched", file=sys.stderr)
 
+    stop_reason: str | None = None
     try:
         stop_reason = await coordinator.run(
             objective=objective,
@@ -2317,6 +2378,7 @@ def _backend_kind(role: str) -> str:
         )
     finally:
         state = coordinator.shared_state
+        effective_stop_reason = stop_reason or coordinator.stop_classification
         # Stopping the leases and the agent subprocesses is itself a step that
         # can hang, so it happens while the supervisor is still watching; the
         # supervisor is stood down only once it has returned.
@@ -2329,90 +2391,18 @@ def _backend_kind(role: str) -> str:
         # exit anyway; this just frees it promptly for an intentional resume.
         with timed_teardown_step(state, "session_lock"):
             session_lock.release()
-        # Crash-safe reports/final.json.
-        try:
-            from ..breakdown import write_minimal_final_json
-
-            with timed_teardown_step(state, "final_json"):
-                final_json = write_minimal_final_json(session_dir)
-            print(f"Final summary     : {final_json}")
-        except Exception:  # noqa: BLE001 — safety net must never mask stop_reason
-            log.exception("crash-safe final.json write failed (non-fatal)")
-        # End-of-session safety net: always materialize session_breakdown.json (best-effort; never mask stop_reason).
-        sequencer_done = getattr(state, "close_sequence_done", False)
-        if sequencer_done:
-            print(
-                "Session breakdown : (already written by CLOSE phase sequencer; skipping cli.finally safety-net write)"
-            )
-            # Re-run the Langfuse flush idempotently as a safety net.
-            try:
-                from hyperloom.orchestrator.trace.langfuse_emitter import (
-                    flush_session,
-                    record_session_breakdown,
-                )
-
-                with timed_teardown_step(state, "langfuse"):
-                    flush_session(session_dir)
-                    from ..breakdown import patch_breakdown_langfuse
-
-                    patch_breakdown_langfuse(session_dir)
-                    record_session_breakdown(session_dir)
-            except Exception:  # noqa: BLE001
-                log.debug("langfuse flush_session (post-sequencer) failed", exc_info=True)
-        else:
-            try:
-                from ..breakdown import write_breakdown_json
-
-                with timed_teardown_step(state, "session_breakdown"):
-                    breakdown_path = write_breakdown_json(session_dir)
-                print(f"Session breakdown : {breakdown_path}")
-            except Exception:  # noqa: BLE001
-                log.exception("session_breakdown finalize failed (non-fatal)")
-            # Safety-net reports/final.md write (no-op when the sequencer's final.md already exists).
-            try:
-                from ..breakdown import write_minimal_final_report
-
-                with timed_teardown_step(state, "final_md"):
-                    final_md = write_minimal_final_report(session_dir)
-                print(f"Final report      : {final_md}")
-            except Exception:  # noqa: BLE001
-                log.exception("emergency final report write failed (non-fatal)")
-            # Live Langfuse push (opt-in, default off): reconcile + flush, then splice the post-flush receipt into the
-            # session_breakdown.json langfuse section.
-            try:
-                from hyperloom.orchestrator.trace.langfuse_emitter import (
-                    flush_session,
-                    record_session_breakdown,
-                )
-
-                with timed_teardown_step(state, "langfuse"):
-                    flush_session(session_dir)
-                    from ..breakdown import patch_breakdown_langfuse
-
-                    patch_breakdown_langfuse(session_dir)
-                    record_session_breakdown(session_dir)
-            except Exception:  # noqa: BLE001
-                log.debug("langfuse flush_session failed (non-fatal)", exc_info=True)
-
-        # Safety-net artifact package -> /workspace, for paths that leave close_sequence_done False and never run the
-        # sequencer.
-        try:
-            from ..breakdown import package_session_artifacts
-
-            with timed_teardown_step(state, "artifact_package"):
-                pkg_path = package_session_artifacts(
-                    session_dir,
-                    session_id=str(getattr(state, "session_id", "") or ""),
-                )
-            if pkg_path is not None:
-                print(f"Artifact package  : {pkg_path}")
-        except Exception:  # noqa: BLE001
-            log.exception("session artifact package failed (non-fatal)")
+        _write_cli_terminal_artifacts(session_dir, state, effective_stop_reason)
         try:
             state.save(session_dir)
         except Exception:  # noqa: BLE001
             log.exception("failed to persist teardown timings (non-fatal)")
 
+    if stop_reason == SUPERVISOR_RESTART_REASON:
+        # The leg is over, the session is not: the monitor resumes it, and a
+        # final summary here would speak for a run that has not ended.
+        print(f"Leg stopped for a supervisor restart; session {session_dir} stays resumable.")
+        return _exit_code_for_stop_reason(stop_reason)
+
     _reconcile_crash_count(coordinator.shared_state, session_dir)
     # NOTE: conc_sweep is now a SWEEP-phase action auto-enqueued by the Coordinator, not a post-hook here.
 

src/hyperloom/inference_optimizer/cli/bootstrap.py

@@ -408,7 +408,8 @@ def _print_final_summary(
 
 def _bank_previous_leg_phase_segment(state: SharedState) -> None:
     """Bank the phase time the stopped leg spent but never recorded."""
-    stop_unix = min(to_unix(state.stop_ts, 0.0) or 0.0, time.time())
+    boundary = state.leg_ended_ts or state.stop_ts
+    stop_unix = min(to_unix(boundary, 0.0) or 0.0, time.time())
     if stop_unix <= 0.0:
         return
     bank_phase_segment(state, until_unix=stop_unix)
@@ -438,6 +439,7 @@ def _begin_resume_leg(state: SharedState) -> str:
     state.resumed_ts = now_iso()
     state.stop_reason = ""
     state.stop_ts = ""
+    state.leg_ended_ts = ""
     state.closing_phase = False
     state.closing_started_unix = 0.0
     state.closing_report_task_id = ""

src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py

@@ -569,6 +569,20 @@ def test_a_resume_banks_what_the_stopped_leg_spent_in_its_phase() -> None:
     assert state.stop_ts == ""
 
 
+def test_a_resume_banks_a_resumable_legs_own_boundary() -> None:
+    state = SharedState(session_id="s", start_ts="2026-08-01T00:00:00+00:00")
+    state.phase = "PRELUDE"
+    state.phase_started_ts = "2026-08-01T00:00:00+00:00"
+    state.phase_started_unix = 1785_542_400.0
+    state.leg_ended_ts = "2026-08-01T00:30:00+00:00"
+
+    cb._begin_resume_leg(state)
+
+    assert state.phase_elapsed_totals == {"PRELUDE": 1800.0}
+    assert state.leg_ended_ts == ""
+    assert state.stop_ts == ""
+
+
 def test_a_second_resume_banks_only_the_leg_that_just_stopped() -> None:
     """The first leg's segment is already durable; re-banking it would double-charge the phase."""
     state = SharedState(session_id="s", start_ts="2026-08-01T00:00:00+00:00")