Watchdog: pause a running campaign at a done signal, persist its state, resume from a new session

[hieplam]

Summary

A campaign that is already running cannot be paused cleanly partway through. The owner wanted to stop an overnight build at a safe point to save account usage, then continue it later from a new Claude Code session. Nothing in the tooling does that today, so the orchestrating session hand-wrote a shell script that watched git for a specific commit and then killed three processes. This issue asks for pause and resume as a first-class watchdog feature: request it once, let the zero-token watchdog wait for a "done signal", stop everything in a safe order, write every piece of state to disk, and resume later with one documented command.

Background: the moving parts

These are the pieces this issue touches. All of them are in this repo under plugins/tribe/.

• Campaign. A batch of one or more idea cards built unattended. Its machine-local state lives in a campaign home: ~/.tribe/<repo-key>/campaigns/<slug>/. That directory holds campaign-state.json, answers.md, escalations/, runs/<run-id>/run.json, runs/<run-id>/logs/, .runner.lock and the optional STOP file.
• Runner. plugins/tribe/scripts/runner/run.ts. A Bun process with no LLM of its own. It spawns one executor session per card through the Claude Agent SDK and records progress in the campaign home. The executor session is a headless Claude Code session acting as the card's "warchief". It dispatches its own subagents inside that same process and makes the card's git commits.
• Watchdog. run.ts watchdog …, the code in core/watchdog/ plus adapters/watchdog-io.adapter.ts. A zero-token supervisor. It launches the runner, waits out account-limit (quota) resets, backs off on overload errors, relaunches after a crash, and exits only when a human must act. Its own files are <home>/watchdog/status.json, events.jsonl and runner-stdout/.
• Done signal. Used here for any observable, durable marker that a unit of work finished. The concrete one in this incident is a git commit on the card branch whose message carries the trailer Tribe-Task: 20/33. Every task commit in a tribe build carries Tribe-Card: and Tribe-Task: N/M trailers.

What happened (2026-09-13, campaign viewer-consolidation)

• The campaign has one card: a 33-task build that runs for many hours inside a single executor session.
• Partway through task 20, the owner said: "once this finish, stop this, i need to save tokens … make sure save it to the current state, so that i can open a new session and continue."
• The only stop mechanism is the STOP file, and it does not apply mid-card (see the next section). The orchestrating session therefore wrote and detached a one-off script. The script polls git log origin/master..HEAD in the card's worktree every 10 seconds for a commit with Tribe-Task: 20/33. When it sees one, it sends SIGTERM to the watchdog, then to the runner's child (the SDK/Claude executor process), then to the runner itself, and writes a marker file. On this machine the script is ~/.tribe/-Users-hip-repo-tribe/campaigns/viewer-consolidation/watchdog/stop-after-task20.sh and the marker is stop-after-task20.done next to it.
• That script is a workaround with real gaps. It runs outside the watchdog, so the watchdog's own status.json does not know about it. It hard-codes one trailer value. It kills with raw signals and never records a paused state. After the stop, resuming depends on a human remembering the exact relaunch command.

Why the existing mechanisms do not cover this

1. STOP acts only between cards. From the runner README, "STOP file and the lock file", verbatim:

   STOP — the owner's soft-stop. If present when a run starts, the runner exits cleanly (code 0) before touching any card. If it appears mid-run, the loop finishes the in-flight card and stops before starting the next one. Delete it to resume.

   In a one-card campaign, "finishes the in-flight card" means the whole build. The loop checks STOP only at start-up (core/loop/run-loop.ts L187-L195) and between cards.

2. The watchdog treats STOP as "do not start new work", not as "stop". See core/watchdog/decide.ts L38-L57 and every if (o.stopFilePresent) return STOP; branch below it. While the runner is alive, the watchdog attaches and never reaches those branches.

3. The watchdog is forbidden to kill. From the README, "What it never does", verbatim:

   Never kills the runner or a session. The runner's own --session-timeout owns that; the watchdog only ever observes, waits and relaunches.

   A pause feature has to change this rule deliberately and narrowly: the watchdog may stop the runner and its session only when an owner has requested a pause and its done signal has fired. The README must say so.

4. Resume works today, but only by accident of the D4 resume matrix. When a runner starts, it re-derives each card's phase from git and GitHub (README "Resume semantics"). In this campaign, three quota-driven relaunches each came back as a fresh executor session. Each one inspected the worktree and continued. Nothing records that the stop was deliberate, where it stopped, or what was uncommitted at that moment.

What to build (the contract; the design is yours)

1. Request a pause without an LLM in the loop. One documented way to say "pause this campaign when a given done signal happens". Examples: a subcommand such as run.ts watchdog pause --home <home> --after-commit-trailer 'Tribe-Task: 20/33', or a PAUSE request file under the campaign home. Required conditions:
   • now: stop at once.
   • After the next commit on the card branch that matches a given trailer or pattern.
   • After the next task commit of any number, which is the common "finish what you're doing" ask.
     The waiting is done by the watchdog process (zero tokens), not by a script an LLM writes.
     Which card the signal comes from. A campaign can have several cards, and with --max-concurrent N several can be in flight at once, each on its own branch.
   • A pause request may name one card, for example --card <id>. The signal then counts only on that card's branch.
   • Without a card named, the signal counts on the branch of any card that is in flight.
   • A pause is always campaign-wide: once the first matching commit is seen, every in-flight session is stopped, not just the one that made the commit.
   • Commits that other in-flight cards land during the ordered shutdown are expected. They must be listed in the persisted pause record, and they are not a bug.
   • "Stopping later than the first matching done signal" means the shutdown began after a later signal. It does not mean a concurrent card committed while the shutdown was already under way.
2. Stop in a safe order when the signal fires.
   • Stop the watchdog's own relaunch behaviour first, so nothing restarts.
   • Then end the executor session gracefully.
   • Then end the runner, letting it finalise run.json and campaign-report.json if it can.
   • Release .runner.lock.
   • No process from the campaign may be left running afterwards, whether runner, SDK session or watchdog. Name how you verify that.
3. Persist everything needed to resume, on disk, before exiting.
   • status.json gets a terminal state that is distinct from done and from needs_human, for example paused, with its own exit code.
   • Record the pause request, the done signal that fired (commit sha and subject), the time, and the card's branch and worktree.
   • Record a snapshot of git status --porcelain in the worktree, so uncommitted work from an in-flight subagent is visible rather than silently lost.
   • Record the exact command that resumes the campaign.
   • Add an events.jsonl line for each step.
4. Resume with one documented command from any new session. It clears the pause state and relaunches through the normal path. A resumed campaign must not be mistaken for a stall, a crash or an escalation.
5. Documentation. Update the runner README's STOP, Watchdog and "What it never does" sections. Also update plugins/tribe/skills/orchestrate-campaign/SKILL.md, so an orchestrating session reaches for pause instead of writing a script.

Acceptance: how this is proven

Correctness is decided by a real process tree, not by mocks. Per plugins/tribe/rules/fixtures-mirror-reality.md, the proof is an end-to-end run.

• Start a watchdog-supervised campaign whose executor makes several commits. A fixture card with a scripted executor is acceptable.
• Request "pause after the next commit". Exactly one more commit lands. Then no campaign process remains, and status.json shows the paused terminal state with that commit's sha. The worktree snapshot matches git status.
• Run the resume command. The campaign continues from the paused point and reaches done.
• pause now while a quota wait is in progress also ends in the paused state, and resume then honours the original reset time.
• Unit tests cover every new row in the pure decision table (decide.ts stays pure).

A pause that leaves any campaign process alive is a bug. A pause that stops later than the first matching done signal is a bug. Stopping earlier than requested is never acceptable either.

Scope fence

• Do not change what the runner does between cards, beyond honouring the pause.
• Keep the STOP file's existing meaning, and do not silently reinterpret it as pause.
• Do not add any LLM call to the watchdog.
• Do not change the executor brief, other than the one line that tells a session a pause may end it.

Decided in advance, not a finding

• The watchdog stopping the runner after an owner-requested pause fired is the intended change to the "never kills" rule. It is not a violation of it.
• Work an in-flight subagent had not committed may be lost, provided the persisted snapshot records it.

Related, out of scope for this issue: a false stalled exit right after a quota relaunch

This showed up three times in the same campaign and interacts with resume, so it is recorded here for whoever picks this up. It needs its own issue.

• The watchdog relaunches the runner when a quota reset arrives. On the very next tick, about 7 ms later, it judges a stall from the previous run's log mtime and exits needs_human: stalled. The new runner keeps working unsupervised until someone relaunches the watchdog.
• Evidence from events.jsonl on this machine (~/.tribe/-Users-hip-repo-tribe/campaigns/viewer-consolidation/watchdog/events.jsonl):

  2026-09-12T17:30:30.009Z relaunch cause=quota
  2026-09-12T17:30:30.016Z stall logPath=.../runs/2026-09-12T12-39-23-572Z-cd0a/logs/... (the OLD run)
  2026-09-12T17:30:30.016Z exit needs_human stalled
  (the same three lines recur at 2026-09-12T22:30:30 and 2026-09-13T03:30:30)
  

• Likely cause, not yet confirmed: observe() (core/watchdog/watch-loop.ts L69-L72) picks the newest runs/<run-id> directory. The just-relaunched runner has not created its run directory yet, so the live pid is paired with the old run's stale log. isStale (core/watchdog/select.ts) then fires.

Source map

What	Where
Watchdog decision table (pure)	core/watchdog/decide.ts
Watchdog loop, observe, and the STOP check in waits	core/watchdog/watch-loop.ts and L408-L418
Watchdog status file shape	core/watchdog/status.ts and model.ts
Watchdog flags	core/watchdog/args.ts
Runner STOP and lock	core/loop/lock.ts and core/loop/run-loop.ts L187-L195
Runner README: STOP, resume, watchdog	README
Orchestrating skill that drives all this	plugins/tribe/skills/orchestrate-campaign/SKILL.md
The workaround script (this machine only)	~/.tribe/-Users-hip-repo-tribe/campaigns/viewer-consolidation/watchdog/stop-after-task20.sh
The watchdog event log from the incident (this machine only)	~/.tribe/-Users-hip-repo-tribe/campaigns/viewer-consolidation/watchdog/events.jsonl
Governing rules	plugins/tribe/rules/pure-core.md, fail-closed-edges.md (every subprocess.run/spawn carries a timeout; isolate git config), fixtures-mirror-reality.md

🤖 Generated with Claude Code