From 6798074932faa3ca43defbb63bea55908ee9b1de Mon Sep 17 00:00:00 2001 From: Tyler <53561637+im-tyler@users.noreply.github.com> Date: Sun, 4 Oct 2026 02:14:13 -0700 Subject: [PATCH] feat(s03): additive migration for task requirements, rehearsed on a restored store Migration 008-ship-task-requirements follows the 006/007 new-table convention exactly: ledger-recorded on every store, guarded by the write-shaped shape probe (hasColumns) and the DDL-parity test, and a rename-aside rebuild limb that is unreachable today because no other shape of ship_task_requirements was ever released (nothing copied, nothing dropped, the aside keeps what it held). The store itself stays deliberately unwired: taskRecord() remains a read-only projection, no route/worker/UI writes the table, and a write path is future work gated on the programme's S03 rollout list. Rehearsed on a restored copy of the production store (verified pre-80c9187-shadow-deploy-2026-10-04 backup; isolated throwaway engine on a private docker network; live ship-* containers never touched): - additive-only: 45 tables / 4223 rows identical before and after, the only change being ship_migrations 7 -> 8 (the 008 ledger row) - replay: a second migrate() applies nothing - real-engine store proof (not the unit-test fake): DDL creates the table, idempotent add, conflict refusal via the primary key, exactly-once waiver via the conditional UPDATE, waived rows stay listed; own proof rows deleted afterwards Driver scripts/check-task-requirements-migration.mjs (the check-intake-concurrency isolation contract: SHIP_ISOLATED_CHECK=1 + NUCLEUS_URL); receipt evals/receipts/2026-10-04-s03-requirements-migration.json. Unit tests: 008 write-shaped probe, no-op on fresh install, rename-aside on a stale shape, ledger-only on a current shape, replay no-op, plus the DDL-parity owner entry for task-requirements.ts. Full suite green (2163 + 222 pass, 1 documented local skip). No upstream defect found; no Neutron or Nucleus change. --- CHANGELOG.md | 22 +++ NEXT_SESSION.md | 2 +- docs/SHIP_RELEASE_PROGRAMME_2026-09-21.md | 10 +- ...2026-10-04-s03-requirements-migration.json | 63 +++++++ scripts/check-task-requirements-migration.mjs | 173 ++++++++++++++++++ src/migrations.test.ts | 71 +++++++ src/migrations.ts | 51 ++++++ src/task-requirements.ts | 13 +- 8 files changed, 395 insertions(+), 10 deletions(-) create mode 100644 evals/receipts/2026-10-04-s03-requirements-migration.json create mode 100644 scripts/check-task-requirements-migration.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 71bcc9a..8c896f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,28 @@ All notable changes to Teploy Ship are recorded here. ## [Unreleased] ### Added +- **The S03 requirements store has its migration, rehearsed on the real + store's backup (008).** `ship_task_requirements` — the record of which + statements were ACCEPTED as a task's requirements and which were later + waived, by whom and why (`src/task-requirements.ts`, drafted in an earlier + slice) — now has migration `008-ship-task-requirements` in `src/migrations.ts`, + the same additive new-table convention as 006/007: ledger-recorded on every + store, guarded by the write-shaped shape probe and the DDL-parity test, and a + rename-aside rebuild for a shape that has in fact never been released (so the + limb is unreachable today and nothing is ever copied or dropped). The store + itself stays deliberately UNWIRED — no route, worker path or UI writes it; + the `taskRecord()` projection is still read-only and a write path remains + future work gated on the programme's S03 rollout list. The migration was + rehearsed on a restored copy of the production store (verified + `pre-80c9187-shadow-deploy-2026-10-04` backup, isolated throwaway engine, + live containers untouched): additive-only across 45 tables / 4223 rows (only + `ship_migrations` moved, 7→8), replay a no-op, and the store's primary-key + and conditional-update semantics (idempotent add, conflict refusal, + exactly-once waiver) verified on the real engine rather than the unit-test + fake. Driver: `scripts/check-task-requirements-migration.mjs` + (`SHIP_ISOLATED_CHECK=1` + `NUCLEUS_URL`, the check-intake-concurrency + isolation contract); receipt: + `evals/receipts/2026-10-04-s03-requirements-migration.json`. - **The takeover panel has a BROWSER tab (S12's last surface).** While a lease is held, the operator drives a real headless Chromium running inside the sandbox — the same one the visual and flow rungs use (the node and go diff --git a/NEXT_SESSION.md b/NEXT_SESSION.md index e9468d6..c431330 100644 --- a/NEXT_SESSION.md +++ b/NEXT_SESSION.md @@ -16,7 +16,7 @@ Wave 4's state ("implemented and checked by automated tests only") is now partly 1. **Shadow-log review** after soak (placement/policy/budget/tool-manifest/knowledge-provenance JSONL + reports); set `SHIP_MODEL_ROUTING_POLICY` so the routing shadow has something to record. Only then discuss any `on` flips. 2. **S08 wiring** (approved): findings into PR body + webhook behind a default-off flag; worker-path change, so shadow-first per the standing rule. -3. **Live proofs still open**: S01 credential proofs on a real sandbox with a private repo; S03 storage migration (write the additive migration, rehearse on a restored copy — backup `pre-80c9187-shadow-deploy-2026-10-04` is available and verified); S27 real teploy-adapter run against a scratch target; S19 doctor probes against the real store/clock; human observation of dashboard users. +3. **Live proofs still open**: S01 credential proofs on a real sandbox with a private repo; S27 real teploy-adapter run against a scratch target; S19 doctor probes against the real store/clock; human observation of dashboard users. (S03's storage migration is done: migration 008 rehearsed on the restored `pre-80c9187` copy, receipt in `evals/receipts/2026-10-04-s03-requirements-migration.json`; the store stays unwired — a write path is separate future work.) 4. **Code still unfinished**: S04 and S12 barely started; most of S07 beyond the grounding check; S10 follow-ups (offline/error states, other roles, screen readers); wiring the inert modules (S15/S17 into delivery+incidents, S19 snapshot producer + restore-check, S18 worker tree provisioning, S07 into the plan-park point). 5. **Re-grade retained runs where possible**: only transcripts were preserved for pre-batch runs, so regrades are limited to transcript+fixture-verifiable scenarios; write supplementary `regrade-*.json` beside the records, never replacing originals. 6. **Deferred**: configuration B of the S02 comparison (needs real API spend, owner-gated). diff --git a/docs/SHIP_RELEASE_PROGRAMME_2026-09-21.md b/docs/SHIP_RELEASE_PROGRAMME_2026-09-21.md index 837d271..f605a20 100644 --- a/docs/SHIP_RELEASE_PROGRAMME_2026-09-21.md +++ b/docs/SHIP_RELEASE_PROGRAMME_2026-09-21.md @@ -483,14 +483,14 @@ Baseline reproduced in this session: `pnpm run lint` clean; root suite 1,412/1,4 | S01 | Git credential placement audited; scrub now runs in the same shell as the credentialed clone/warm fetch | Yes | Sequencing test fails on previous code; failed-fetch probe is a regression pin only; env-mode real-git probe fails in argv mode (negative control) | No | `SHIP_GIT_CREDENTIAL=env` (off by default) keeps the token out of argv and config; proven on `LocalExecutor` with a real git and a local HTTP server, not through the Sandbox daemon. Needs: daemon `env` forwarding, sandbox git ≥ 2.31, a live private-repo proof on Forgejo and GitHub, then flipping the default (see `AUDIT_OPEN.md`). Preview egress is owned by teploy-cli, not this repo; no change made. Command-regex item `teploy-ship-04` unchanged | | S02 | Approval stops classified `authority-hold`, separate from `harness-error`; hash-linked reclassification of the migration run | Yes | Executor test fails on the old classification; manifest validation passes | No new model runs | Held-out set, clean current baseline, review-grader lexical limit, authorised-approval path for the migration scenario, n=3 repeats | | S28 | `runTiming(events)` derives first-step, human-wait, timer and active time with explicit unknowns | Yes, not yet wired into any export or UI | 7 unit tests including open waits and missing timestamps; not exercised against live logs | No | Wire into audit/JSON and UI; queue-claim event; OpenTelemetry export; review precision and rework measures | -| S03 | `taskRecord()` projects execution / acceptance / delivery from existing records; `src/task-requirements.ts` is a drafted, unwired store for accepted and waived requirements (new table, no migration) | Yes, a read-only projection; not wired into a route or UI | 8 unit tests: finished ≠ accepted, revision supersedes approval, denied ≠ absent, delivery never inferred | No | Persisting requirements, acceptance criteria and waivers (needs an additive migration rehearsed on a restored copy); wiring the projection; versioned plans; lost-response / two-tab / steer scenarios on a real run | +| S03 | `taskRecord()` projects execution / acceptance / delivery from existing records; `src/task-requirements.ts` is a drafted, unwired store for accepted and waived requirements (migration 008 added and rehearsed on a restored production copy, 2026-10-04) | Yes, a read-only projection; not wired into a route or UI; no write path enabled | 8 unit tests: finished ≠ accepted, revision supersedes approval, denied ≠ absent, delivery never inferred; migration unit tests + rehearsal receipt (`evals/receipts/2026-10-04-s03-requirements-migration.json`) | No | Wiring the projection; authorisation for accept/waive; versioned plans; lost-response / two-tab / steer scenarios on a real run | | S13 | Steering is refused (route) and not offered (UI) for runs whose harness cannot consume it, from one capability table | Yes, steering only | 4 unit tests on the table; web tests and build pass; the route refusal itself has no automated test | No | Full capability declaration (tools, browser, interruption, recovery, accounting); conformance journeys per harness; comparative results | | S05, S06, S08, S09 | Not started this session | Existing foundations only | | | Greenfield and unfamiliar-repo journeys (first-batch item 5) | | S18 | Not started this session | Existing API/client path only | | | Recorded two-repository integration-check kind (first-batch item 6) | | S18 | `integrationStatus()` in `src/integration-evidence.ts`: the contract an executed two-repository test's evidence must satisfy (exact producer/consumer revisions, static never satisfies executed, stale on upstream move, failed/not-run/unknown all block) | Yes, a pure contract; no executor and not wired into coordination | 8 unit tests | No | The executed integration-check kind itself (a non-scan journey with authority over a two-repo workspace and the real fixture run), wiring the contract into the coordination completion gate behind an opt-in, live producer/consumer run | | All others | Not started | Per plan table | | | As in the plan | -First-executable-batch items: **(1)** mostly done — origin/main identity matches the plan, the baseline is reproduced and the [capability inventory](CAPABILITY_INVENTORY_2026-10-03.md) is written (subagent sweep, partly spot-checked); the S28 fields are only the derivation above. **(2)** partly done — hold classification fixed; deterministic preflight run (manifest valid, 12/12 dry-runs exit 0, grader negative controls 8/8, unauthorised spend refused with exit 2); the paid canary and batch proposal are not done (no spend authorised or attempted). **(3)** partly done — records mapped and projected (S03 row); no migration, no live lost-response/restart/steer rehearsal. **(4)** partly done — credential placement above; preview isolation is a teploy-cli change plus a live proof on the preview target, neither possible from here. **(5)** and **(6)** not started. +First-executable-batch items: **(1)** mostly done — origin/main identity matches the plan, the baseline is reproduced and the [capability inventory](CAPABILITY_INVENTORY_2026-10-03.md) is written (subagent sweep, partly spot-checked); the S28 fields are only the derivation above. **(2)** partly done — hold classification fixed; deterministic preflight run (manifest valid, 12/12 dry-runs exit 0, grader negative controls 8/8, unauthorised spend refused with exit 2); the paid canary and batch proposal are not done (no spend authorised or attempted). **(3)** partly done — records mapped and projected (S03 row); migration 008 written and rehearsed on a restored production copy (2026-10-04); no live lost-response/restart/steer rehearsal. **(4)** partly done — credential placement above; preview isolation is a teploy-cli change plus a live proof on the preview target, neither possible from here. **(5)** and **(6)** not started. ### Execution status, wave 2 (2026-10-03, origin/main after PR #31) @@ -574,9 +574,9 @@ Small specifications per the slice discipline above, recorded against their exis - **Evidence today:** `taskRecord()` can only show what each run's input said; nothing records which statements were accepted as requirements, or that one was waived and by whom (inventory, "Stored records"). - **Intended behaviour:** a per-task requirement record. Adding is idempotent for identical content and a refused conflict for different content under the same id (no silent last-write-wins). A waiver needs an actor and a reason, applies once, keeps the requirement visible, and never becomes evidence that it was met. Who may accept or waive is the caller's policy, not the store's. - **Owning layer / contracts:** Ship (`src/task-requirements.ts`: interface plus memory, file and Nucleus stores). No Neutron or Nucleus change. -- **Migration / recovery:** none required. The store creates its own new table (`ship_task_requirements`, primary key `req_key`) with `CREATE TABLE IF NOT EXISTS`, the same additive pattern as the other stores, so no existing run, parked fingerprint or populated table is altered. Revert is not wiring it; the table is inert and can stay. -- **Independent acceptance (done):** 7 tests across all three stores: idempotent add, conflict refusal, concurrent-writer race on the primary key, waiver rules and exactly-once, per-task ordering, input limits, additive-only SQL. The Nucleus tests use a fake of my own writing, **not a real Nucleus**. -- **Not done, and needed before any rollout:** rehearse on a restored production copy (`docs/UPGRADING.md`); a real-Nucleus run of the primary-key and conditional-update behaviour; wiring into enqueue (record the initial request and each follow-up as requirements) and the run page; authorisation for accept/waive on the existing approval grants; feeding waived/active requirements into `taskRecord()` acceptance; scenarios for two tabs, lost response and steering on a real run. +- **Migration / recovery:** additive migration `008-ship-task-requirements` (src/migrations.ts, the 006/007 new-table convention: ledger-recorded, write-shaped shape probe, DDL-parity guard; rename-aside limb unreachable because no other shape was ever released). Rehearsed 2026-10-04 on a restored copy of the production store (backup `pre-80c9187-shadow-deploy-2026-10-04`): additive-only on 45 tables / 4223 rows (only ship_migrations moved, 7→8), replay a no-op, and the store's primary-key and conditional-update behaviour verified on the real engine. Receipt: `evals/receipts/2026-10-04-s03-requirements-migration.json`, driver `scripts/check-task-requirements-migration.mjs`. Revert is not wiring it; the table is inert and can stay. +- **Independent acceptance (done):** 7 tests across all three stores: idempotent add, conflict refusal, concurrent-writer race on the primary key, waiver rules and exactly-once, per-task ordering, input limits, additive-only SQL. The Nucleus tests use a fake of my own writing, **plus the real-engine rehearsal above**. +- **Not done, and needed before any rollout:** wiring into enqueue (record the initial request and each follow-up as requirements) and the run page; authorisation for accept/waive on the existing approval grants; feeding waived/active requirements into `taskRecord()` acceptance; scenarios for two tabs, lost response and steering on a real run. - **Evidence location:** `src/task-requirements.test.ts`. #### S01 — environment-supplied Git credential (implemented, off by default) diff --git a/evals/receipts/2026-10-04-s03-requirements-migration.json b/evals/receipts/2026-10-04-s03-requirements-migration.json new file mode 100644 index 0000000..5d095f1 --- /dev/null +++ b/evals/receipts/2026-10-04-s03-requirements-migration.json @@ -0,0 +1,63 @@ +{ + "rehearsal": { + "date": "2026-10-04", + "purpose": "S03: prove migration 008-ship-task-requirements and the task-requirements store against a real Nucleus holding a restored copy of the production store", + "archive": "/deployments/ship/_backups/pre-80c9187-shadow-deploy-2026-10-04/nucleus-data-full.tgz", + "archiveSha256": "verified against its .sha256 sidecar (sha256sum -c) + gzip -t", + "engineImage": "ghcr.io/neutron-build/nucleus:v1.1.1 (the image of the live ship-nucleus container, read from docker ps -a)", + "isolation": "restore unpacked into /root/s03-rehearsal-s03104851 only; throwaway engine container ship-s03-proof-104851 on a private docker network (no published port); check script run from a node:22-bookworm container on the same network; only the proof container, network and directory were removed afterwards", + "liveContainersUntouched": "ship-web-80c9187, ship-worker-80c9187, ship-nucleus, ship-gateway-web-c0b07dd Up throughout (verified before and after; no restart)", + "engineBootMs": 3000, + "checkWallMs": 951, + "cleanup": "proof container + network + rehearsal dir removed; /deployments/ship and all ship-* containers never touched" + }, + "check": { + "pass": true, + "scope": "migration 008 + task-requirements store on a restored copy; no production path", + "startedAt": "2026-10-04T09:10:35.393Z", + "finishedAt": "2026-10-04T09:10:35.564Z", + "url": "postgres://ship-s03-proof-104851:5432/nucleus", + "tablesSource": "SHOW TABLES", + "tablesPopulatedBefore": 37, + "totalRowsBefore": 4223, + "tablesBefore": { + "ship_akiroo_cursor": 2, "ship_artifact_expiry": 0, "ship_artifacts": 217, + "ship_attributed_spend": 89, "ship_bulletin_boards": 0, "ship_bulletin_posts": 0, + "ship_code_chunks": 329, "ship_code_files": 244, "ship_code_rates": 2, + "ship_code_repos": 6, "ship_connect_requests": 5, "ship_delivery": 7, + "ship_docs": 778, "ship_docs_001": 170, "ship_docs_004": 178, + "ship_evidence": 1, "ship_fleet": 1, "ship_fleet_capacity": 1, + "ship_fleet_load": 1, "ship_governance": 1, "ship_launch_chunks": 149, + "ship_launch_commits": 140, "ship_launch_dispositions": 0, "ship_launches": 140, + "ship_live": 0, "ship_memory": 205, "ship_memory_003": 13, + "ship_migrations": 7, "ship_outbox": 0, "ship_placement": 384, + "ship_policies": 1, "ship_projects": 18, "ship_projects_v2": 24, + "ship_repo_stats": 206, "ship_runtime_config": 531, "ship_spend": 253, + "ship_spend_holds": 16, "ship_steer": 5, "ship_steer_002": 1, + "ship_tasks": 58, "ship_tasks_005": 18, "ship_tasks_v2": 2, + "ship_unpriced_runs": 20, "ship_users": 0, "ship_workspace_content": 0 + }, + "tablesAfter": "identical to tablesBefore on every table except ship_migrations: 7 -> 8 (the 008 ledger row). ship_task_requirements was absent before the store-proof leg and is created by the store DDL, empty", + "migrate": { + "firstPassAppliedIds": [], + "firstPassNote": "ledger-only, the designed path: the table does not exist on the real store (the store is unwired), so needed() is false and 008 is recorded without table surgery. The rename-aside limb is covered by the unit tests against a stale shape.", + "firstPassMs": 34, + "ledgerIds": [ + "001-ship-docs-source-ranon", "002-ship-steer-consumed-turn", "003-ship-memory-note-id", + "004-ship-docs-actor", "005-ship-tasks-requested-by", "006-ship-runtime-config", + "007-ship-connect-requests", "008-ship-task-requirements" + ], + "replayAppliedIds": [], + "replayMs": 12 + }, + "storeProof": { + "ms": 40, + "idempotentAdd": true, + "conflictRefused": true, + "waiverExactlyOnce": true, + "waivedStaysListed": true, + "rowsCreated": 2, + "cleanup": "proof rows deleted (final ship_task_requirements count 0, all other tables unchanged); the empty table and the 008 ledger row stay in the throwaway copy only" + } + } +} diff --git a/scripts/check-task-requirements-migration.mjs b/scripts/check-task-requirements-migration.mjs new file mode 100644 index 0000000..8ee8441 --- /dev/null +++ b/scripts/check-task-requirements-migration.mjs @@ -0,0 +1,173 @@ +#!/usr/bin/env node +// S03 rehearsal: prove migration 008 + the requirements store against a REAL +// Nucleus holding a restored copy of the production store — never production +// itself. Mirrors scripts/check-intake-concurrency.mjs's isolation contract +// (SHIP_ISOLATED_CHECK=1 + an explicit NUCLEUS_URL) and scripts/ship-backup.sh +// rehearse's posture (isolated restore, throwaway engine, own rows cleaned up). +// +// What it proves, in order: +// 1. additive-only: every ship_* table that exists before migrate() exists +// after it with an IDENTICAL row count; the only ledger/table change is +// the 008 row in ship_migrations. +// 2. the store DDL creates ship_task_requirements on a real engine and its +// primary-key / conditional-update semantics behave as the unit tests' +// fake claims (idempotent add, conflict refusal, exactly-once waiver). +// 3. replay idempotency: a second migrate() applies nothing. +// +// It creates only rows under its own proof task root and deletes them before +// exit; the empty table and the ledger row stay (the schema change is the +// point of the rehearsal). Output is a JSON receipt on stdout. +import assert from 'node:assert/strict'; +import { NucleusPgwire } from '../dist/nucleus-pgwire.js'; +import { migrate } from '../dist/migrations.js'; +import { NucleusTaskRequirements } from '../dist/task-requirements.js'; + +if (process.env.SHIP_ISOLATED_CHECK !== '1' || !process.env.NUCLEUS_URL) { + throw new Error('Set SHIP_ISOLATED_CHECK=1 and NUCLEUS_URL for an isolated test engine'); +} + +// Fallback table list in case the engine answers no SHOW TABLES; a live +// Nucleus does (the receipt records which path was taken). +const SHIP_TABLES = [ + 'ship_akiroo_cursor', 'ship_akiroo_receipts', 'ship_artifact_expiry', 'ship_artifacts', + 'ship_attributed_spend', 'ship_bulletin_boards', 'ship_bulletin_posts', 'ship_code_chunks', + 'ship_code_files', 'ship_code_rates', 'ship_code_repos', 'ship_connect_requests', + 'ship_delivery', 'ship_docs', 'ship_evidence', 'ship_fleet', 'ship_fleet_capacity', + 'ship_fleet_load', 'ship_governance', 'ship_knowledge_records', 'ship_launch_chunks', + 'ship_launch_commits', 'ship_launch_dispositions', 'ship_launches', 'ship_live', + 'ship_memory', 'ship_migrations', 'ship_model_segments', 'ship_outbox', 'ship_placement', + 'ship_policies', 'ship_projects', 'ship_projects_v2', 'ship_repo_stats', 'ship_run_images', + 'ship_runtime_config', 'ship_spend', 'ship_spend_holds', 'ship_steer', + 'ship_task_requirements', 'ship_tasks', 'ship_tasks_v2', 'ship_unpriced_runs', + 'ship_users', 'ship_workspace_content', +]; + +const db = new NucleusPgwire(process.env.NUCLEUS_URL, 's03-requirements-rehearsal'); +const receipt = { + pass: false, + scope: 'migration 008 + task-requirements store on a restored copy; no production path', + startedAt: new Date().toISOString(), + url: process.env.NUCLEUS_URL.replace(/:\/\/[^@]*@/, '://'), + tablesBefore: {}, + tablesAfter: {}, + migrate: {}, + storeProof: {}, +}; + +async function countRows(table) { + try { + const rows = await db.query(`SELECT COUNT(*) AS n FROM ${table}`); + return Number(rows[0]?.n ?? 0); + } catch { + // Fallback for an engine without aggregate support: page through the + // table in bounded slices and count client-side. + let total = 0; + for (;;) { + const rows = await db.query(`SELECT 1 FROM ${table} LIMIT 1000 OFFSET ${total}`); + total += rows.length; + if (rows.length < 1000) return total; + } + } +} + +async function tableList() { + try { + const rows = await db.query('SHOW TABLES'); + if (Array.isArray(rows) && rows.length > 0) return { tables: rows.map((r) => String(r.table_name)).sort(), source: 'SHOW TABLES' }; + } catch { /* fall through to the static list */ } + return { tables: SHIP_TABLES, source: 'static list' }; +} + +async function counts() { + const out = {}; + for (const table of (await tableList()).tables) { + try { + out[table] = await countRows(table); + } catch { + out[table] = 'absent'; + } + } + return out; +} + +try { + receipt.tablesSource = (await tableList()).source; + receipt.tablesBefore = await counts(); + const populated = Object.entries(receipt.tablesBefore).filter(([, v]) => typeof v === 'number' && v > 0); + const proofRoot = `s03-proof-${Date.now().toString(36)}`; + + let t0 = Date.now(); + const applied = await migrate(db, (line) => process.stderr.write(`${line}\n`)); + let t1 = Date.now(); + receipt.migrate.firstPassAppliedIds = applied; + receipt.migrate.firstPassMs = t1 - t0; + assert.ok(applied.includes('008-ship-task-requirements') || applied.length === 0, + '008 must run (stale shape) or be ledger-recorded by the winner/no-op path'); + + receipt.tablesAfter = await counts(); + for (const [table, before] of Object.entries(receipt.tablesBefore)) { + const after = receipt.tablesAfter[table]; + if (table === 'ship_migrations') { + assert.ok(after === 'absent' || after >= before, `ship_migrations must not lose rows (${before} -> ${after})`); + } else { + assert.equal(after, before, `additive-only violated: ${table} ${before} -> ${after}`); + } + } + const ledger = await db.query("SELECT id FROM ship_migrations"); + receipt.migrate.ledgerIds = ledger.map((r) => String(r.id)).sort(); + assert.ok(receipt.migrate.ledgerIds.includes('008-ship-task-requirements'), '008 must be in the ledger'); + + t0 = Date.now(); + const appliedAgain = await migrate(db); + t1 = Date.now(); + receipt.migrate.replayAppliedIds = appliedAgain; + receipt.migrate.replayMs = t1 - t0; + assert.deepEqual(appliedAgain, [], 'replay must apply nothing'); + + const store = new NucleusTaskRequirements(db); + t0 = Date.now(); + const base = { taskRootRunId: proofRoot, requirementId: 'req-1', statement: 'The tool must ship a single static binary.', source: 'request', sourceRunId: 'run-proof-1', createdBy: 'rehearsal' }; + const added = await store.add(base); + assert.equal(added.created, true); + const again = await store.add(base); + assert.equal(again.created, false, 'identical re-add must be idempotent'); + assert.equal(again.requirement.statement, added.requirement.statement); + let conflict; + try { await store.add({ ...base, statement: 'A different statement under the same id.' }); } catch (e) { conflict = e; } + assert.equal(conflict?.name, 'RequirementConflictError', 'different content under the same id must be refused'); + const second = await store.add({ taskRootRunId: proofRoot, requirementId: 'req-2', statement: 'The tool must not require a config file.', source: 'request', createdBy: 'rehearsal' }); + assert.equal(second.created, true); + const waived = await store.waive(proofRoot, 'req-2', 'rehearsal-actor', 'superseded by req-1'); + assert.equal(waived.ok, true); + const rewaived = await store.waive(proofRoot, 'req-2', 'rehearsal-actor', 'twice'); + assert.equal(rewaived.ok, false, 'a second waiver must be refused'); + assert.equal(rewaived.failure, 'already-waived'); + const listed = await store.list(proofRoot); + assert.equal(listed.length, 2, 'a waived requirement stays in the list'); + assert.equal(listed.filter((r) => r.state === 'waived').length, 1); + t1 = Date.now(); + receipt.storeProof = { + ms: t1 - t0, + idempotentAdd: true, + conflictRefused: true, + waiverExactlyOnce: true, + waivedStaysListed: true, + rowsCreated: 2, + }; + + await db.query('DELETE FROM ship_task_requirements WHERE task_root_run_id = $1', [proofRoot]); + const finalCounts = await counts(); + assert.equal(finalCounts.ship_task_requirements, 0, 'only own proof rows may be removed'); + for (const [table, before] of Object.entries(receipt.tablesAfter)) { + if (table === 'ship_task_requirements') continue; + assert.equal(finalCounts[table], before, `cleanup touched more than its own rows: ${table}`); + } + + receipt.pass = true; + receipt.finishedAt = new Date().toISOString(); + receipt.tablesPopulatedBefore = populated.length; + receipt.totalRowsBefore = Object.values(receipt.tablesBefore).reduce((a, v) => a + (typeof v === 'number' ? v : 0), 0); + console.log(JSON.stringify(receipt, null, 2)); +} finally { + await db.close().catch(() => {}); +} diff --git a/src/migrations.test.ts b/src/migrations.test.ts index 239602a..2b998e8 100644 --- a/src/migrations.test.ts +++ b/src/migrations.test.ts @@ -4,6 +4,7 @@ import test from "node:test"; import { DOC_FIELDS } from "./nucleus-pgwire.js"; import { MIGRATIONS, hasColumns, migrate } from "./migrations.js"; import { RUN_META_FIELDS } from "./run-store.js"; +import { TASK_REQUIREMENT_COLUMNS } from "./task-requirements.js"; import type { NucleusPgwire } from "./nucleus-pgwire.js"; /** @@ -229,6 +230,7 @@ test("each migrated table's shape matches the store DDL that creates it fresh", { table: "ship_memory", file: "repo-memory.ts" }, { table: "ship_runtime_config", file: "runtime-config.ts" }, { table: "ship_connect_requests", file: "connect-requests.ts" }, + { table: "ship_task_requirements", file: "task-requirements.ts" }, ]; for (const { table, file } of owners) { @@ -338,3 +340,72 @@ test("007 probes ship_connect_requests write-shaped, and is a no-op on a fresh i assert.doesNotMatch(joined, /DROP TABLE|TRUNCATE/i); assert.doesNotMatch(joined, /INSERT INTO ship_connect_requests/); }); + +/** + * 008's probe, and the slice it exists for. ship_task_requirements is S03's + * requirement record and it is deliberately unwired: no route or worker path + * writes it, so on a real deployment the migration lands as a ledger row and + * nothing else. The dangerous fault is therefore the SILENT one a SELECT probe + * would leave: `waived_reason` read back as NULL is a waiver with no recorded + * reason — a decision that looks made when nothing was — and `state` read back + * as NULL collapses active and waived into one list. A SELECT-shaped probe + * would report every column present forever, record 008 as applied, and leave + * exactly that behind. + */ +test("008 probes ship_task_requirements write-shaped, and is a no-op on a fresh install", async () => { + const fresh = fakeDb(); + assert.deepEqual(await migrate(fresh), [], "the store DDL creates the table; there is nothing to migrate"); + + const stale = fakeDb({ + existingTables: new Set(["ship_task_requirements"]), + // A shape from before the waiver columns existed. + columns: { ship_task_requirements: ["req_key", "task_root_run_id", "requirement_id", "statement", "source", "source_run_id", "state", "created_at", "created_by"] }, + }); + assert.deepEqual(await migrate(stale), ["008-ship-task-requirements"]); + + const joined = stale.sql.join("\n"); + assert.match( + joined, + /UPDATE ship_task_requirements SET req_key = req_key, task_root_run_id = task_root_run_id, requirement_id = requirement_id, statement = statement, source = source, source_run_id = source_run_id, state = state, created_at = created_at, created_by = created_by, waived_at = waived_at, waived_by = waived_by, waived_reason = waived_reason WHERE 1 = 0/, + "the shape probe must be an UPDATE", + ); + assert.doesNotMatch( + joined, + /SELECT[^\n]*\bwaived_reason\b/, + "a SELECT probe cannot see a missing column on Nucleus — that is the bug this rule exists for", + ); + assert.match(joined, /ALTER TABLE ship_task_requirements RENAME TO ship_task_requirements_008/); + // The rebuilt table is empty and nothing is destroyed: the shape has never + // been released, so its columns are not knowable here — the aside table + // keeps whatever they held. + assert.doesNotMatch(joined, /DROP TABLE|TRUNCATE/i); + assert.doesNotMatch(joined, /INSERT INTO ship_task_requirements/); +}); + +/** + * 008 against a store that already has the table in the CURRENT shape — + * populated or not — records the migration without table surgery, and a second + * migrate() is a no-op. This is the leg the real deployment takes: the store + * is drafted, not wired, so its table (when it exists at all) was created by + * the store DDL itself and the migration must only enter the ledger. + */ +test("008 is ledger-only on a current-shape table, and replay is a no-op", async () => { + const current = fakeDb({ + existingTables: new Set(["ship_task_requirements"]), + columns: { ship_task_requirements: TASK_REQUIREMENT_COLUMNS }, + }); + assert.deepEqual(await migrate(current), []); + assert.doesNotMatch(current.sql.join("\n"), /RENAME TO/, "a current-shape table must not be rebuilt"); + const ledger = await current.query("SELECT id FROM ship_migrations"); + assert.ok( + ledger.some((r) => String(r.id) === "008-ship-task-requirements"), + "008 must still be recorded as applied", + ); + + const stale = fakeDb({ + existingTables: new Set(["ship_task_requirements"]), + columns: { ship_task_requirements: ["req_key", "statement"] }, + }); + assert.deepEqual(await migrate(stale), ["008-ship-task-requirements"]); + assert.deepEqual(await migrate(stale), [], "second call must be a no-op"); +}); diff --git a/src/migrations.ts b/src/migrations.ts index b235771..28ff451 100644 --- a/src/migrations.ts +++ b/src/migrations.ts @@ -1,5 +1,6 @@ import { CONNECT_REQUEST_COLUMNS } from "./connect-requests.js"; import { RUNTIME_CONFIG_COLUMNS } from "./runtime-config.js"; +import { TASK_REQUIREMENT_COLUMNS } from "./task-requirements.js"; import type { NucleusPgwire } from "./nucleus-pgwire.js"; /** @@ -357,6 +358,55 @@ const connectRequestsTable: Migration = { }, }; +/** + * 008 — ship_task_requirements: S03's record of which statements were ACCEPTED + * as a task's requirements and which were later waived, by whom and why (see + * task-requirements.ts). The store is drafted but deliberately unwired — no + * route, worker path or UI writes it — so on the real deployment this migration + * lands as a ledger row and nothing else: the table does not exist there, the + * store's own CREATE TABLE IF NOT EXISTS is what will bring it in on first use, + * and a write path is future work gated on the programme's S03 rollout list. + * + * A no-op on a fresh install, exactly like 006 and 007, and written now for the + * same reason: the write-shaped probe in `needed()` and the DDL-parity test in + * migrations.test.ts only exist for a table that has an entry here, and the day + * this table grows a column is the day both are needed. + * + * The rename-aside limb below is unreachable today — ship_task_requirements has + * never been released in any other shape. If it ever fires, the rebuilt table + * is EMPTY and the old rows survive in ship_task_requirements_008: nothing is + * copied, because a shape we have never released has no knowable column names + * to copy from, and a requirement record guessed at is worse than one kept + * aside intact. Nothing is dropped, ever. + */ +const taskRequirementsTable: Migration = { + id: "008-ship-task-requirements", + description: "rebuild ship_task_requirements when its shape is behind the store DDL", + async needed(db) { + if (!(await tableExists(db, "ship_task_requirements"))) return false; // fresh install: the store DDL creates it + return !(await hasColumns(db, "ship_task_requirements", TASK_REQUIREMENT_COLUMNS)); + }, + async run(db) { + await db.query("ALTER TABLE ship_task_requirements RENAME TO ship_task_requirements_008"); + await db.query( + `CREATE TABLE ship_task_requirements ( + req_key TEXT PRIMARY KEY, + task_root_run_id TEXT, + requirement_id TEXT, + statement TEXT, + source TEXT, + source_run_id TEXT, + state TEXT, + created_at TEXT, + created_by TEXT, + waived_at TEXT, + waived_by TEXT, + waived_reason TEXT + )`, + ); + }, +}; + export const MIGRATIONS: Migration[] = [ docsSourceColumn, steerConsumedTurn, @@ -365,6 +415,7 @@ export const MIGRATIONS: Migration[] = [ tasksRequestedBy, runtimeConfigTable, connectRequestsTable, + taskRequirementsTable, ]; /** diff --git a/src/task-requirements.ts b/src/task-requirements.ts index e138a4f..f1d6a6b 100644 --- a/src/task-requirements.ts +++ b/src/task-requirements.ts @@ -10,9 +10,11 @@ import type { NucleusPgwire } from "./nucleus-pgwire.js"; * Today a task's intent exists only as the text of each run's input * (task-record.ts projects it). Nothing says which statements were ACCEPTED as * requirements, which were later waived, or by whom. This store is that - * record. It is a NEW table: stores create their own tables with - * CREATE TABLE IF NOT EXISTS, so no migration runs, no existing run, parked - * fingerprint or populated table is touched, and nothing reads it yet. + * record. It is a NEW table: the store creates it with CREATE TABLE IF NOT + * EXISTS and migration 008 (src/migrations.ts) carries the ledger entry, the + * write-shaped shape probe and the DDL-parity guard — the same additive + * convention as 006/007. No existing run, parked fingerprint or populated + * table is touched, and nothing reads it yet. * * Rules, each pinned in task-requirements.test.ts: * - Adding the same requirement twice is idempotent (a lost response retried @@ -26,7 +28,10 @@ import type { NucleusPgwire } from "./nucleus-pgwire.js"; * records the actor it is given and does not decide. * * Rollout is separate and not done: wiring the store into enqueue and the run - * page, and rehearsing it on a restored production copy (docs/UPGRADING.md). + * page (a write path remains future work; nothing here enables one). The + * migration has been rehearsed on a restored production copy — + * scripts/check-task-requirements-migration.mjs, receipt under + * evals/receipts/ (docs/UPGRADING.md). */ export const TASK_REQUIREMENT_STATEMENT_LIMIT = 4000;