feat(s19): snapshot producer, restore-check, live doctor probes, bundle folding

AUDIT_OPEN.md

@@ -54,8 +54,13 @@ All off by default or observe-only; flags are in the programme's wave-4 table.
 - **S15/S17 (PR #46):** inert; callers must redact log excerpts.
 - **S22 (PR #47):** the coordination record does not keep the revision the
   client was planned against.
-- **S19 (PR #49):** doctor's store and clock checks report unknown until real
-  probes exist; the restore comparison has no snapshot producer.
+- **S19 (PR #49; tail landed 2026-10-04):** doctor now answers store liveness
+  and clock skew read-only when `NUCLEUS_URL` is set (unknown otherwise, and
+  unknown when the engine answers no time query — honest, not a gap to paper
+  over); `teploy-ship snapshot` / `restore-check` exist with the script's
+  invariants. Still open: the unpack and rehearsal remain script-only by
+  design; restore-readiness has no command; none of the S19 tail has run
+  against a live store.
 - **S07 (PR #53):** wired at the plan park 2026-10-04 behind `SHIP_PLAN_GROUNDING` (default off): the grounding report is an additive `grounding` field on the `plan-think` step result — no durable step added, flag-off bytes unchanged — and the run page's plan-review card renders it as advisory text with the name-existence caveat. Still open: the rest of S07 (planning-time use beyond the park, other consumers).
 - **Agent hygiene:** parallel agents share one scratch directory and the fixed
   grading port 8901. A backup-file collision corrupted one agent's mutation

CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to Teploy Ship are recorded here.
 ## [Unreleased]
 
 ### Added
+- **S19 tail: `teploy-ship snapshot` and `teploy-ship restore-check`, live
+  doctor probes, and doctor + backup receipt in the support bundle.** The
+  practiced procedure in `scripts/ship-backup.sh` is now productized where
+  that is safe: `snapshot` produces the identical archive shape (tar of the
+  nucleus data dir + sha256 sidecar + manifest under
+  `$SHIP_BACKUP_DIR/<label>-<date>/`), refuses while any `ship-*` container
+  it can see is running — printing the exact coordinated `docker stop` it
+  expects — unless the operator attests with `--i-stopped-writers`, never
+  overwrites, and never deletes anything except its own partial output after
+  a failed production (so a failed tar cannot wedge the label+date slot;
+  retention stays the operator's). `restore-check` verifies an archive
+  WITHOUT unpacking it: sha256 sidecar in `sha256sum -c` form (the sidecar
+  must travel with its archive), full-stream gzip integrity, and the content
+  listing, failing on absolute or `..` paths; its verdict is integrity-only
+  and names the rehearsal as the restore proof — the unpack and rehearse
+  steps stay in the script, deliberately. `doctor` stops reporting blind
+  unknowns for the store and clock when `NUCLEUS_URL` is set: read-only
+  liveness (`SELECT 1`) and skew against the store's own clock
+  (`SELECT now()`), each degrading to unknown rather than guessing, and
+  store consistency deliberately not probed at all (read-only, writers may
+  run — that proof is a rehearsal). The support bundle (`scriptVersion` 2)
+  now carries `doctor.json` and the latest backup manifest — the receipt,
+  never the archive — both degrading to note files.
+
 - **The S03 requirements store has its migration, rehearsed on the real
   store's backup (008).** `ship_task_requirements` — the record of which
   statements were ACCEPTED as a task's requirements and which were later

NEXT_SESSION.md

@@ -15,9 +15,9 @@ Wave 4's state ("implemented and checked by automated tests only") is now partly
 ## Next work, in order
 
 1. **Shadow-log review** after soak (placement/policy/budget/tool-manifest/knowledge-provenance JSONL + reports); set `SHIP_MODEL_ROUTING_POLICY` so the routing shadow has something to record. Only then discuss any `on` flips.
-2. **S08 wiring** (approved): BUILT in wave 5 (this PR) — findings in the PR body + run-completion webhook behind `SHIP_TEST_INTEGRITY_SURFACING=off|shadow|on`, default off. Still to do: soak `shadow` on the real worker, then decide the `on` flip.
-3. **Live proofs still open**: S01 credential proofs on a real sandbox with a private repo; S27 real teploy-adapter run against a scratch target; S19 doctor probes against the real store/clock; human observation of dashboard users. (S03's storage migration is done: migration 008 rehearsed on the restored `pre-80c9187` copy, receipt in `evals/receipts/2026-10-04-s03-requirements-migration.json`; the store stays unwired — a write path is separate future work. S07's plan-park wiring landed in #59.)
-4. **Code still unfinished**: S04 and S12 barely started; most of S07 beyond the grounding check; S10 follow-ups (offline/error states, other roles, screen readers); wiring the inert modules (S15/S17 into delivery+incidents, S19 snapshot producer + restore-check, S18 worker tree provisioning, S07 into the plan-park point).
+2. **S08 wiring** (approved): BUILT in wave 5 (#63) — findings in the PR body + run-completion webhook behind `SHIP_TEST_INTEGRITY_SURFACING=off|shadow|on`, default off. Still to do: soak `shadow` on the real worker, then decide the `on` flip.
+3. **Live proofs still open**: S01 credential proofs on a real sandbox with a private repo; S27 real teploy-adapter run against a scratch target; S19 doctor probes against a live store; human observation of dashboard users. (S03's storage migration is done: migration 008 rehearsed on the restored `pre-80c9187` copy, receipt in `evals/receipts/2026-10-04-s03-requirements-migration.json`; the store stays unwired — a write path is separate future work. S07's plan-park wiring landed in #59.)
+4. **Code still unfinished**: S04 and S12 barely started; most of S07 beyond the grounding check; S10 follow-ups (offline and error states, other roles, screen readers); wiring the inert modules (S15/S17 into delivery+incidents, S18 worker tree provisioning). Done in wave 5: S07's plan-park wiring (#59), S19's snapshot producer + restore-check + doctor probes (this PR).
 5. **Re-grade retained runs where possible**: only transcripts were preserved for pre-batch runs, so regrades are limited to transcript+fixture-verifiable scenarios; write supplementary `regrade-*.json` beside the records, never replacing originals.
 6. **Deferred**: configuration B of the S02 comparison (needs real API spend, owner-gated).
 

docs/BACKUP.md

@@ -62,6 +62,38 @@ Environment: `SHIP_ROOT` (default `/deployments/ship`), `SHIP_BACKUP_DIR`
 (default `$SHIP_ROOT/_backups`), and for `rehearse` only: `SHIP_BIN`
 (default `teploy-ship`), `SHIP_NUCLEUS_IMAGE`, `SHIP_REHEARSE_TIMEOUT_S`.
 
+## The CLI commands: `snapshot` and `restore-check`
+
+The producer and the verifier are also first-class CLI commands, with the
+same invariants and the same archive shape — a backup taken with either tool
+verifies with both:
+
+```sh
+teploy-ship snapshot [--label NAME] [--ship-root DIR] [--backup-dir DIR]
+                     [--dry-run] [--i-stopped-writers] [--json]
+teploy-ship restore-check <archive> [--json]
+```
+
+- `snapshot` writes the identical `<label>-<date>/nucleus-data-full.tgz` +
+  `.sha256` sidecar + `manifest.txt` under `SHIP_BACKUP_DIR` (flags
+  `--ship-root`/`--backup-dir`, env `SHIP_ROOT`/`SHIP_BACKUP_DIR`, same
+  defaults as the script). It refuses while `ship-*` containers run unless
+  you attest the coordinated stop with `--i-stopped-writers`, never
+  overwrites an existing backup dir, and never deletes anything — except its
+  own partial output when a production fails, so a failed tar does not
+  occupy the label+date slot forever. Retention stays yours.
+- `restore-check` verifies an archive WITHOUT unpacking it: sha256 sidecar
+  (in `sha256sum -c` form — the sidecar must travel with its archive),
+  full-stream gzip integrity, and the tar content listing, failing on
+  absolute or `..` paths that would escape an unpack target. It is integrity
+  only: it says plainly that no engine has booted on the archive and points
+  at `scripts/ship-backup.sh rehearse` as the restore proof.
+
+The unpack (`restore --into` an empty isolated dir) and the rehearsal stay
+in the script, deliberately: they are the steps that touch a live recovery,
+and a shell script an operator reads line by line is the right shape for
+them.
+
 ## What the archive contains
 
 The complete engine data directory (`accessories/nucleus/nucleus-data`):

docs/SHIP_RELEASE_PROGRAMME_2026-09-21.md

@@ -546,7 +546,7 @@ Same rules as wave 3: every change is off by default or observe-only, with tests
 | S15, S17 | #46 | none (inert modules) | `src/recovery-compat.ts` (migration-aware recovery check: unread or unclassified state holds; a newer contract or irreversible migration is incompatible; a recovery plan never counts as a pass) and `src/incident-evidence.ts` (facts, hypotheses and unknowns kept apart; other-service evidence set aside; fewer than five samples is unknown; confidence capped at medium) | 19 tests, 8 negative controls | Unwired; no migration-state reader; no page timeline; log excerpts clamped not redacted (callers must redact); no live incident |
 | S22 | #47 | `SHIP_MISSION_VIEW=on` (off) | Read-only mission view on the coordination page; accepted only when the task record says accepted; waivers shown as waived | 23 tests, 4 negative controls | No launching, persistence or replanning; planned-on revision not recorded on the client child, so a moved upstream with no delivery record is invisible |
 | S26 | #48 | `SHIP_PLACEMENT=shadow` (off) | Placement decision recorded beside the sandbox pool's host choice and the host-loss path; `placement shadow-report` | 19 tests through the real pool and durable path, 3 mutations | No enforcement; per-run architecture or browser needs not recorded; no drain concept in the pool; a failed liveness probe looks like a TTL expiry |
-| S19 | #49 | none (new command) | `teploy-ship doctor`: install prerequisites with pass, fail or unknown (unknown is never pass), secrets reported by presence only, redacted output; restore-readiness comparison module | 20 tests, 8 negative controls, planted-secret control | Store and clock probes report unknown; no snapshot producer or restore-check command; not in the support bundle; S04 not started |
+| S19 | #49 | none (new command) | `teploy-ship doctor`: install prerequisites with pass, fail or unknown (unknown is never pass), secrets reported by presence only, redacted output; restore-readiness comparison module. **S19 tail:** doctor answers store liveness (`SELECT 1`) and clock skew (against the store's own clock, `SELECT now()`) read-only when `NUCLEUS_URL` is set, unknown otherwise; `teploy-ship snapshot` produces the script's archive shape (tar + sha256 sidecar + manifest) and refuses running writers without `--i-stopped-writers`, never overwrites, never deletes; `teploy-ship restore-check` verifies an archive (sidecar + gzip integrity + content listing, no unpacking; integrity-only verdict names the rehearsal as the restore proof); the support bundle carries `doctor.json` + the latest backup manifest (never the archive) | 20 doctor tests + 4 live-probe tests (fake store client), 8 negative controls, planted-secret control; 11 snapshot/restore-check tests incl. negative controls (running-writer refusal with exact stop line, overwrite refusal, tampered sidecar, truncated archive with re-computed checksum caught by gzip, path-escape listing, failed-production cleanup); 2 bundle tests (doctor+receipt included, degrade to notes); CLI wiring type-checked only | S04 not started; doctor's clock stays unknown with no configured store or when the engine does not answer a time query (honest by design); the unpack and rehearsal remain script-only (`scripts/ship-backup.sh restore/rehearse`) — the CLI verifies without unpacking by design; restore-readiness comparison still has no command; nothing in the tail has run against a live store or live docker |
 | S13 | #50 | `SHIP_HARNESS_RECORD=on` (off) | Probes for plan-review, steer and investigate across native, claude-code and opencode (fake binaries); harness identity recorded as a sibling key on run-started (step fingerprint unchanged); "Ran on" block on the run page; credential-lifecycle tests | 15 root and 4 web tests, negative controls | Real vendor binaries; five operations unprobed; probes not run at startup or in CI |
 | S21 | #51 | `SHIP_KNOWLEDGE_PROVENANCE=off|shadow|on` (off) | Sidecar provenance store (new table, no migration); notes screened and labelled fresh, stale or unknown in `on`; redaction cascades on delete; summaries recorded as agent claims | 25 tests, 6 negative controls, durable step sequence identical across modes | No live Nucleus; freshness coarse; replay-exact derivation not available; cli.ts and agent.ts paths unwired; project scope is repo-only |
 | S24 | #52 | `SHIP_TOOL_MANIFEST=shadow|on`, `SHIP_EVENT_ENVELOPE=on` (both off) | `tool validate` dry-run CLI; manifest conformance findings recorded per tool call (never block); signed event envelope as extra webhook headers; client verification example | 14 tests, 4 negative controls, default-off header and body byte identity | The loop sees only the action kind, so host and secret findings cannot fire until the executor reports them; no tool registry or web view |

docs/SUPPORT.md

@@ -29,6 +29,8 @@ states are recorded as note files inside the bundle instead.
 | `config-summary.json` | The known-safe Ship settings — a fixed **whitelist** of model ids, numbers, booleans, policy maps and URL hosts (`SHIP_MODEL`, `SHIP_MAX_STEPS`, `SHIP_SANDBOX_TTL_SEC`, `SHIP_WARM_PARKS`, `SHIP_PUBLIC_URL` as host, `SHIP_TELEMETRY`, `SHIP_INTAKE_POLICIES`, `SHIP_MIN_FREE_MB`, `SHIP_MAX_CONCURRENT_RUNS`, `SHIP_HARNESS_MODEL`). |
 | `runs-summary.json` | Counts by state over the read window, plus the 20 most recent runs: id, state, repo, createdAt, cost when attributed, and a one-line terminal error for failed runs. |
 | `selfwatch.txt` | The health snapshot from `src/selfwatch.ts` (queue depth, worker staleness, stuck runs) computed against the same store, plus its warnings. |
+| `doctor.json` | The redacted `teploy-ship doctor` report, run at assembly time with the same probes the command uses — live store liveness and clock skew when `NUCLEUS_URL` is set. Degrades to `doctor-unavailable.txt` when the doctor did not run or failed. |
+| `backup-receipt.txt` | The manifest of the newest backup under `SHIP_BACKUP_DIR` (default `$SHIP_ROOT/_backups`) — the receipt only; **the archive itself is never part of a bundle**. A note names where it looked when no backup exists. |
 | `logs/` | The last `--log-lines` lines of each `ship-*` container (`docker logs --tail`), or a note file explaining why not. |
 | `REDACTION-REPORT.txt` | Counts of every redaction the gate performed, by category. |
 
@@ -73,11 +75,21 @@ Every check reports `pass`, `fail` or `unknown`; **unknown is not pass**. The
 verdict is `ready` only when all pass, `not-ready` on any fail, `incomplete`
 when nothing failed but something could not be established. Exit code is 0 only
 for `ready`. Passing the `store-url` shape check does not mean the store was
-reached: `store-connectivity` stays `unknown` until a live check answers (use
-`teploy-ship preflight`). `--out` writes the redacted JSON, suitable to attach
-to a support request. The restore side is `src/restore-readiness.ts`, which
-compares backup and restored history snapshots and reports `unverified`
-whenever it did not actually compare; it is not yet exposed as a command.
+reached: with `NUCLEUS_URL` set, `store-connectivity` is probed read-only
+(`SELECT 1` — the rehearse proof's own liveness question) and `clock` measures
+skew against the store's own clock (`SELECT now()`); without a configured store,
+or when the engine does not answer a time query, both honestly stay `unknown` —
+store CONSISTENCY is deliberately not probed at all, because it cannot be
+established read-only while writers run (that proof is a snapshot rehearsal).
+`--out` writes the redacted JSON, suitable to attach to a support request.
+
+The snapshot side is now productized: `teploy-ship snapshot` produces the same
+archive shape as `scripts/ship-backup.sh backup` (see [BACKUP.md](BACKUP.md)),
+and `teploy-ship restore-check <archive>` verifies an archive — sha256 sidecar,
+gzip integrity, content listing — without unpacking it. The restore comparison
+module (`src/restore-readiness.ts`) still has no command of its own; the
+rehearsal (`scripts/ship-backup.sh rehearse`) plus `preflight` remain the
+restore proof.
 
 ## The support policy shape
 

src/args.ts

@@ -76,6 +76,8 @@ export const COMMAND_FLAGS: Record<string, FlagSpec> = {
   web: { boolean: ["dev"], value: ["port", "token", "model"] },
   support: { value: ["out", "log-lines", "days"] },
   doctor: { boolean: ["json"], value: ["out"] },
+  snapshot: { boolean: ["dry-run", "i-stopped-writers", "json"], value: ["label", "ship-root", "backup-dir"] },
+  "restore-check": { boolean: ["json"] },
   eval: { boolean: ["critic", "settle"], value: ["model", "suite", "repeats"] },
 };