fix(recovery): decide stuck-task recovery under the dispatch state lock (#977)

[markhayden]

Summary

Closes #977 — the residual recorded in #975's round-3 review.

Both recovery engines judged a workflow task stranded from a scope read before awaiting hooks. A nested/map child spawned and claimed between that walk and the synchronous check was invisible (parent row untouched, child run keyed on the child id), so the parent could be recovered or blocked while a child was executing.

Every ledger claim is already made under withStateLock, so the decision and the write now happen under that lock:

• Watchdog: recoverStuckTask runs per task under the lock — re-reads liveness (re-walks the workflow topology; a scope that changed since the scan is left for the next tick), validates the row fresh (inProgress, same version), then supersede / recover / block with nothing awaiting between the read and the store write. The scan's read stays as the pre-filter and feeds alert-only mode.
• Restart recovery: each candidate is decided in settleCandidate under the lock, so a claim in flight (HTTP is live during startup recovery) lands before the re-assessment reads the ledger.

Deadlock fixed on the way (Codex review of this PR)

resolveTeamAssignmentForStep runs inside dispatchWorkflowTask, which the cycle/single dispatch already call under withStateLock; on a transient team-routing failure it awaited withStateLock again to record the ladder entry. The lock is a non-reentrant promise queue, so that dispatch cycle never finished — on main today — and with recovery queued on the same lock the watchdog and startup recovery would have hung too. The resolver now records into the caller's state (new StepTeamResolutionArgs.state) and never takes the lock; the lock holder saves as it already did.

dispatch-state.ts imports only the logger — check:cycles unchanged.

Verification

• New tests (fail against the previous engines): a child spawned and claimed during the scan leaves the parent alone; the watchdog and restart recovery park behind a held lock and see the claim its holder made; a topology that changed with nothing live is skipped that tick and recovered on the next; a claim racing the decision queues behind recovery's lock; the step resolver completes a transient failure while the caller holds the lock.
• tests/core/{watchdog,restart-recovery,dispatch,dispatch-handoff,dispatch-team-step,continuation}.test.ts, tests/plugins/workflows/team-step-token.test.ts, health checks route suite green; typecheck, lint (3 pre-existing warnings), check:cycles green.
• Isolated boot on 3799 with a 1.5 s stuck threshold: a task moved to In Progress with no run is auto-recovered to Todo by the lock-guarded path.

Docs: .claude/knowledge/dispatch.md § Restart recovery, .claude/knowledge/agent-system.md § Watchdog, .claude/knowledge/team-aware-assignment.md (step ladder lives in the caller's state), spec D9 residual marked closed.

🤖 Generated with Claude Code