Skip to content

Recover threads whose checkout folder was deleted - #161

Merged
badcuban merged 4 commits into
mainfrom
missing-checkout-recovery
Aug 15, 2026
Merged

Recover threads whose checkout folder was deleted#161
badcuban merged 4 commits into
mainfrom
missing-checkout-recovery

Conversation

@badcuban

Copy link
Copy Markdown
Collaborator

Fixes the three failure modes from the deleted-worktree incident: a thread whose git worktree vanished broke with a misleading "Claude binary not found" error and a Retry that could never work, new draft threads inherited the dead folder with the checkout picker hidden (bricking the project), and nothing prevented or detected the deletion in the first place.

What changes

Detection and recovery

  • A single shared module (vcs/CheckoutPresence.ts) answers "is this checkout still on disk?" for every consumer, with a three-state design: only a confirmed absence triggers recovery; a check error always fails open.
  • Session pre-flight: every provider start/restart/handoff verifies the checkout exists before spawning, failing with a typed CheckoutMissingError instead of reaching the SDK. Spawn ENOENT against a confirmed-missing cwd is reclassified honestly (the SDK wraps the error and drops the errno, so the directory check decides).
  • The thread view replaces the dead-end "Turn failed / Retry" with a recovery notice: "This thread's folder no longer exists" plus Switch to local checkout and (when the branch still exists) Recreate worktree. The notice keys off live VCS status, so it self-clears once the folder is back.
  • A presence watcher (2s poll, 2s confirmation window to ride out git's in-place renames) announces out-of-band deletions within seconds and fans the recovery state out to every thread bound to the path — no failed turn required. Reappearance pushes a fresh status so surfaces clear promptly.

No more inherited traps

  • New drafts drop a known-missing checkout (and its branch and worktree env mode) and fall back to the project root.
  • The checkout picker survives an invalid selection: it stays visible, annotates the selection "(missing)" with the thread's own branch name, and lists refs from the project root so there is always a way out.
  • VCS status distinguishes "path is gone" (pathMissing) from "not a repository", so the source panel shows recovery actions instead of offering git init on a deleted folder, and the panel launcher rows say the folder is missing instead of describing a tree that is not there.

Guardrails

  • vcs.removeWorktree refuses when a live session or an active thread still uses the worktree, naming the blocking threads (typed VcsWorktreeInUseError, no force flag). Archived threads do not block; a projection read failure fails open.
  • Recreating a worktree over the stale registration an out-of-band deletion leaves behind prunes dead registrations and retries — without this, the recovery action can never work.
  • Sessions running in a managed worktree get one standing instruction telling the agent the app owns worktree lifecycle, aimed at the "helpful post-merge cleanup" that caused the incident.

Plumbing fixes found by live verification

  • Status cache/stream keys canonicalize through the nearest existing ancestor, so a subscription opened while the folder is missing keys identically to statuses published after recreation (realPath alone flips between /tmp and /private/tmp forms across the existence boundary and stranded the recovery UI after navigation).
  • The local half of a status keeps pathMissing across remote updates (it was silently dropped, collapsing the recovery UI moments after it appeared).
  • Warning activities are excluded from timeline semantic summarization (the keyword heuristic saw "folder" and rendered the notice as "Explored project · 1 directory list").

Testing

  • vp fmt, vp lint, vp run typecheck, full server (1878) and web (2222) unit suites, browser suite (471).
  • New regression tests: checkout-presence classification, removal guard policy, watcher disappearance/blip/key-stability (it.live with symlinked temp dirs), stale-registration recreate against real git, pathMissing retention across remote updates, draft fallback, picker survival, launcher missing states, recovery view-model.
  • Live end-to-end verification against a throwaway stack with Playwright, twice (an initial external review pass found three defects; after fixes, every checkpoint passes): deletion detected in 3.6s with the correct notice, branch label, and source-panel recovery; state survives navigation; recreate restores the folder and the notice self-clears in 3s; switch-to-root recovers; new drafts land on the project root; in-use removal is refused naming the blocking thread. No provider sessions were started at any point.

A thread running in a git worktree broke permanently when that folder was
removed: the next turn failed inside the provider SDK as "Claude Code native
binary not found", the only affordance was a Retry that failed identically,
new threads inherited the dead path, and the source panel offered to run
`git init` on a directory that was not there.

Detection and recovery:

- Pre-flight the checkout at the single seam every provider start, restart and
  handoff passes through (ProviderCommandReactor). A confirmed-missing
  directory fails with CheckoutMissingError and records a thread.checkout.missing
  activity; no process is ever spawned in a directory that is gone.
- Classify spawn ENOENT honestly. A missing cwd and a missing executable fail
  with the same errno, and the SDK wraps it as a binary problem. A confirmed
  missing working directory now outranks that guess.
- Offer the two things that resolve it, in the composer notice and the source
  control panel: move the thread to the project root, or recreate the worktree
  at the same path (only when its branch still exists).

Fallbacks so one broken thread stays one broken thread:

- Drafts no longer inherit a checkout already known to be missing.
- The checkout picker survives an invalid selection: it stays visible while the
  project root is a repository, marks the selection "(missing)", and lists refs
  from the project root so there is always a way out.
- VCS status distinguishes "path does not exist" from "not a repository", so
  Initialize Git is only offered for a directory that is actually there.

Guardrails:

- vcs.removeWorktree refuses a path a non-stopped session or a live thread still
  uses, failing with VcsWorktreeInUseError naming them. No force override.
- The status broadcaster polls watched checkouts and reports a disappearance
  after a confirmation window, so recovery appears between turns rather than
  after the next crash.
- Sessions running in a managed worktree are told not to remove or prune it.

The missing-checkout determination lives in one place (vcs/CheckoutPresence),
shared by the pre-flight, the status typing and the watcher. A check error is
never treated as missing: only a confirmed absence opens the new paths.
The thread view decides whether to show the recovery actions from the
checkout's VCS status, which can still be a healthy snapshot taken before the
folder was deleted. The pre-flight now refreshes that status when it finds a
checkout missing, so the affordance appears with the failed turn instead of
waiting for the watcher's next pass.

Test fallout from the new behaviour, all real rather than cosmetic:

- GitWorkflowService stats the requested cwd now, so its test layers need a
  filesystem, and the two "no repository detected" cases were pointed at a path
  that does not exist. They now use a directory that is really there, and a new
  case pins the distinction that Initialize Git depends on.
- The reactor and integration harnesses stub the status broadcaster and needed
  the new missing-checkout stream.
- The reactor suite's project root was an invented path, which the pre-flight
  correctly refuses; it now uses real temp directories.
- Four browser suites stub gitStatusState with a partial factory that the draft
  seeder's new synchronous snapshot read was missing.
…reate, honest launcher rows

A full Playwright pass against a throwaway stack (delete the worktree out
from under a bound thread, recover both ways, navigate around) surfaced
four defects the unit suites could not:

- Status cache/stream keys now canonicalize through the nearest existing
  ancestor, so a subscription opened while the checkout is missing keys
  identically to the statuses published after it is recreated. realPath
  alone flips form across the existence boundary (/tmp vs /private/tmp on
  macOS), which stranded the recovery UI after any navigation.
- Recreating a worktree over the stale registration a deleted folder
  leaves behind now prunes and retries; git otherwise refuses the path.
- The presence watcher pushes a fresh status when a missing checkout
  reappears, so recovery surfaces clear in seconds instead of at the next
  scheduled refresh.
- Local status parts keep pathMissing across remote updates; the checkout
  picker labels the thread's own branch rather than the fallback ref
  list's; warning activities are excluded from timeline summarization (the
  keyword heuristic read "folder" and filed the notice as exploration);
  the panel launcher rows say the folder is missing instead of describing
  a tree that is not there.
@vercel

vercel Bot commented Aug 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
threadlines Skipped Skipped Aug 15, 2026 10:20am

Request Review

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Aug 15, 2026
… stubs

Caught by CI's cold typecheck: the worktree's vp cache replayed a stale
typecheck success locally, so three errors shipped — the Codex adapter
now provides the FileSystem service the session runtime requires for its
managed-worktree check, and two reactor test stubs gained the broadcaster's
new observeMissingCheckouts method.
@badcuban
badcuban merged commit 3ffada2 into main Aug 15, 2026
15 checks passed
@badcuban
badcuban deleted the missing-checkout-recovery branch August 15, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant