From 0371201b4e575ac9a9be9161546a96d3679fcdd8 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 28 Jul 2026 02:10:03 +0400 Subject: [PATCH 1/3] adds unattended plan --- README.md | 2 +- apps/backend/README.md | 17 ++++++++++------- docs/technical-plan.md | 19 ++++++++++--------- 3 files changed, 21 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 135ae70..ae0f988 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ The Release workflow publishes the promoted versions with npm provenance. It rej Do not merge `master` back into `dev`. `NPM_TOKEN` needs publish access to the `@understudy` scope. -Backend deployment remains a separate Wrangler operation. Keep `UNATTENDED_ENABLED_TENANTS=[]` until the canary extension passes the production acceptance suite. +Backend deployment remains a separate Wrangler operation. Keep `UNATTENDED_ENABLED_TENANTS=[]` until the canary extension passes the production acceptance suite. Follow the [unattended production rollout runbook](docs/unattended-production-rollout.md) for deployment order, evidence gates, and rollback. ## Preserve attended proof history diff --git a/apps/backend/README.md b/apps/backend/README.md index 3fec4cb..80489bd 100644 --- a/apps/backend/README.md +++ b/apps/backend/README.md @@ -162,19 +162,22 @@ The Miniflare test suite needs permission to bind a loopback port. ## Deploy safely -Deploy the dual-protocol backend with unattended creation disabled: +Use the [unattended production rollout runbook](../../docs/unattended-production-rollout.md) as the canonical deployment, evidence, and rollback procedure. Deploy the dual-protocol backend with unattended creation disabled: ```bash pnpm --filter @understudy/backend exec wrangler deploy ``` -After one canary extension reports protocol 2, enable only its tenant. Complete the production Chromium acceptance suite and 24-hour soak before broad enablement. +After deployment, record the exact migration-`v2`, flags-off version as the rollback baseline. After one canary extension reports protocol 2, enable only its tenant. Complete the production Chromium acceptance suite and 24-hour soak before broad enablement. A rollback must: -1. Disable new unattended leases -2. Drain or terminalize active leases and granted commands -3. Roll back application code -4. Retain the additive Durable Object migrations +1. Return the consumer to attended mode +2. Roll back to the recorded migration-`v2`, flags-off version +3. Confirm new unattended leases are disabled +4. Delete and poll active leases while the durable sweeper retains unresolved cleanup +5. Retain the additive Durable Object migrations and coordinator data -Do not deploy protocol-1-only code while protocol-2 leases exist. +Migration `v2` is additive and irreversible. Cloudflare blocks rollback across incompatible Durable Object class lifecycle changes, so the active migration-`v1` version cannot be assumed to remain a valid rollback target after `v2`. See [Cloudflare Worker rollback constraints](https://developers.cloudflare.com/workers/versions-and-deployments/rollbacks/). + +Do not remove migration `v2` or deploy protocol-1-only code while protocol-2 leases exist. diff --git a/docs/technical-plan.md b/docs/technical-plan.md index aa1fdb1..03a1b16 100644 --- a/docs/technical-plan.md +++ b/docs/technical-plan.md @@ -214,18 +214,19 @@ Analytics Engine receives content-free events for authentication, allocation, li ## Rollout gates -Production configuration starts with `UNATTENDED_ENABLED_TENANTS=[]`. Deploy the additive Durable Object migration and dual-protocol backend before enabling any tenant. +Production configuration starts with `UNATTENDED_ENABLED_TENANTS=[]`. Deploy the additive Durable Object migration and dual-protocol backend before enabling any tenant. The [unattended production rollout runbook](unattended-production-rollout.md) is the canonical execution procedure and status ledger. -Rollout order: +The conceptual rollout order is: -1. Build and load the production extension in a tenant-dedicated profile -2. Enroll one canary device and confirm protocol 2 -3. Enable one tenant -4. Complete the real-Chromium acceptance suite -5. Run a 24-hour read-only soak -6. Enable remaining tenants after zero unexpected unknown writes and correct expiry behavior +1. Make the governed Metamind consumer compatible and recoverable while it remains attended +2. Establish a migration-`v2`, flags-off Understudy rollback baseline +3. Accept one canary device and complete the real-Chromium suite +4. Run a 24-hour read-only soak +5. Prove one unattended Metamind workflow with correlated audit and durable cleanup +6. Ramp allowlisted traffic through `1 → 5 → all` +7. Complete the final 24-hour operational soak -Rollback disables new leases first, drains or terminalizes active leases and granted commands, then rolls back application code. Never remove the additive migration or deploy protocol-1-only code while protocol-2 leases exist. +Rollback returns Metamind to attended mode, returns Understudy to the recorded migration-`v2`, flags-off version, confirms new leases are disabled, and drains active leases. Never remove the additive migration, target the earlier migration-`v1` version, or deploy protocol-1-only code while protocol-2 leases exist. ## Historical attended proof From ee105ac7de0b0a8022e815f601fc991a0c8144d7 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 28 Jul 2026 02:10:09 +0400 Subject: [PATCH 2/3] adds unattended plan --- docs/unattended-production-rollout.md | 862 ++++++++++++++++++++++++++ 1 file changed, 862 insertions(+) create mode 100644 docs/unattended-production-rollout.md diff --git a/docs/unattended-production-rollout.md b/docs/unattended-production-rollout.md new file mode 100644 index 0000000..a669261 --- /dev/null +++ b/docs/unattended-production-rollout.md @@ -0,0 +1,862 @@ + + +# Finish the unattended production rollout + +This runbook is the canonical operator record for Understudy’s unattended production rollout. Keep it current in every implementation, deployment, canary, and ramp pull request until the final 24-hour operational soak passes. + +## Define the finish line + +The rollout is complete only when every condition below holds: + +- Understudy’s additive Durable Object migration `v2` and dual-protocol backend are live +- The production Chromium acceptance suite passes +- The read-only 24-hour soak passes +- Metamind proves an unattended workflow governed by FlowSafe approvals and Breakwater connectors +- Correlated audit evidence proves durable session cleanup and no write before approval +- Allowlisted production traffic completes the `1 → 5 → all` ramp +- The final 24-hour operational soak passes +- This runbook records full release SHAs, deployment and version IDs, proof artifacts, metrics evidence, and the rollback baseline + +Do not declare the project finished from a code merge, package release, Worker deployment, or single proof run. + +## Update the status ledger + +Update this table in the pull request that changes a gate. Use only `Not started`, `In progress`, `Blocked`, `Passed`, or `Rolled back` as the state. + +| Field | Required evidence | +|---|---| +| Gate | Named phase and acceptance gate | +| State | `Not started`, `In progress`, `Blocked`, `Passed`, or `Rolled back` | +| Approved SHA | Full Git commit, never a branch name | +| Deployment | Cloudflare deployment and version IDs, or an explicit reason that no deployment applies | +| Evidence | Continuous integration (CI) run, proof artifact, query result, or operator record | +| Completed | Coordinated Universal Time (UTC) timestamp | +| Owner | Engineering, release operator, or canary operator | + +Never mark a gate `Passed` without its approved SHA, deployment disposition, evidence, completion timestamp, and owner. Use `Pending` for evidence that does not exist yet. Do not use a branch name as temporary SHA evidence. + +| Gate | State | Approved SHA | Deployment | Evidence | Completed | Owner | +|---|---|---|---|---|---|---| +| Phase 0a: release-flow baseline | Passed | `4843b6bccd8e1028c8fb6dba7812d643a4106778` | Not applicable: package and branch-flow gate | [PR #20](https://github.com/ProofOfTechOrg/understudy/pull/20), [PR #21](https://github.com/ProofOfTechOrg/understudy/pull/21), [master CI](https://github.com/ProofOfTechOrg/understudy/actions/runs/30255565127), [Version](https://github.com/ProofOfTechOrg/understudy/actions/runs/30255262164), [Release](https://github.com/ProofOfTechOrg/understudy/actions/runs/30255565196) | `2026-07-27T09:51:30Z` | Release operator | +| Phase 0b: rollout runbook merged | In progress | Pending | Not applicable: documentation gate | This file; merge pull request pending | Pending | Engineering | +| Phase 1: Metamind compatible and recoverable in attended mode | Not started | Pending | Current baseline: deployment `f85a53c6-7b90-4b6f-a427-2e8cef7df637`, version `b7890ecc-b4c4-489f-a67f-3cafbca67b6a` | Pending | Pending | Engineering | +| Phase 2: Understudy `v2`, flags-off rollback baseline | Not started | Pending | Current baseline: deployment `b73220f0-8035-40d2-9987-243770d96306`, version `41434382-ecdd-4f95-a27c-811c4337b6bd` | Pending | Pending | Release operator | +| Phase 3a: canary device acceptance | Not started | Pending | Pending | Pending | Pending | Canary operator | +| Phase 3b: 24-hour read-only soak | Not started | Pending | Pending | Pending | Pending | Canary operator | +| Phase 4: governed Metamind unattended proof | Not started | Pending | Pending | Pending | Pending | Canary operator | +| Phase 5a: one-record production gate | Not started | Pending | Pending | Pending | Pending | Canary operator | +| Phase 5b: five-record production gate | Not started | Pending | Pending | Pending | Pending | Canary operator | +| Phase 5c: all-allowlisted 24-hour gate | Not started | Pending | Pending | Pending | Pending | Canary operator | +| Phase 6: rollout closeout | Not started | Pending | Pending | Pending | Pending | Release operator | + +## Start from the verified baseline + +The following state was verified on 2026-07-27. Recheck it before operational work because deployments and remote refs can change. + +### Repository and release baseline + +| Repository state | Verified value | +|---|---| +| Understudy working baseline | `dev@e4b98e6824b2dbee078a7c57da37a11f389010b9` | +| Understudy release baseline | `origin/master@4843b6bccd8e1028c8fb6dba7812d643a4106778` | +| Understudy local `master` | Stale at `46b210f745793cbc3d57fb06c96b28a627552429`; never deploy it without refreshing remote refs | +| Metamind release baseline | Clean `master@ee94790ddf92b8fabebba10a502e76005a57e17d` | +| Metamind baseline CI | [Passing run](https://github.com/ProofOfTechOrg/metamind/actions/runs/30179532157) | +| Published packages | `@understudy/protocol@0.8.0`, `@understudy/connector@0.5.1` | + +The package release is complete. Do not create another package release unless later work changes a published package. + +### Production deployment baseline + +| Service | Deployment | Version | Known state | +|---|---|---|---| +| Understudy | `b73220f0-8035-40d2-9987-243770d96306` | `41434382-ecdd-4f95-a27c-811c4337b6bd` | Migration `v1`; no unattended Durable Object bindings, telemetry binding, rate limiter, or rollout variables | +| Metamind | `f85a53c6-7b90-4b6f-a427-2e8cef7df637` | `b7890ecc-b4c4-489f-a67f-3cafbca67b6a` | `COMMIT=local`; deployment provenance is not verifiable | + +### Operator prerequisites + +Real-Chromium verification works only under these conditions: + +- Chrome 125 or newer +- One tenant-dedicated Chrome profile +- Chrome startup set to **New Tab** +- The production extension build, not WXT development mode +- No DevTools attached to a controlled tab +- An awake machine, browser, and network for each soak +- Access to both GitHub repositories, both Cloudflare Workers, production D1, Worker secrets, and the canary profile + +## Phase 0: Freeze the release baseline + +Freeze branch and evidence state before implementation begins. Merge this runbook before any operational deployment. + +1. Refresh remote refs in each repository: + + ```bash + git fetch --prune + git status --short + git rev-parse HEAD + git rev-parse origin/dev + git rev-parse origin/master + ``` + +2. Confirm each working tree is clean. Resolve any output from `git status --short` before deployment. +3. Merge feature work into `dev` through reviewed pull requests. +4. Promote reviewed `dev` commits to `master` through a `dev → master` pull request. +5. Never merge `master` back into `dev`. +6. Merge the rollout-document pull request before Phase 1. +7. Update this runbook in every later implementation and deployment pull request. + +Record the merged SHA for this runbook in Phase 0b. Do not copy the pre-merge `dev` SHA into that field. + +## Phase 1: Make Metamind compatible and recoverable + +Complete and deploy this phase while production remains in attended mode. Understudy’s current backend must continue to serve the attended proof during this phase. + +### Upgrade the Understudy client packages + +In Metamind, change `packages/worker/package.json` to: + +```json +{ + "@understudy/connector": "^0.5.1", + "@understudy/protocol": "^0.8.0" +} +``` + +Run `pnpm install` to update `pnpm-lock.yaml`, then review the dependency diff. Do not publish new Understudy package versions unless their package code changes. + +### Add the mode configuration + +Add these non-secret bindings to both `wrangler.jsonc` files, `packages/worker/src/env.ts`, and generated Worker types: + +| Binding | Contract | +|---|---| +| `UNDERSTUDY_SESSION_MODE` | `attended` or `unattended`; default and initial production value is `attended` | +| `UNDERSTUDY_DEVICE_ID` | Required UUID in unattended mode | +| `UNDERSTUDY_ALLOWED_ORIGINS` | JSON array of canonical exact origins; maximum 32 | +| `UNDERSTUDY_PROFILE_STATE_KEY` | Required non-secret account-state identifier in unattended mode | + +Set the synthetic canary origins to exactly: + +```json +["https://example.com", "https://practice.expandtesting.com"] +``` + +Reject a record origin that is absent from this list before creating an Understudy session. Expanding production scope requires one reviewed change that updates both Metamind’s source-controlled allowlist and the extension’s local origin policy. + +### Create attended and unattended sessions correctly + +Attended creation must omit both `body` and `Content-Type`. Understudy treats any body, including `"{}"`, as an unattended request. + +Unattended creation must send the JSON request with: + +- `mode: "unattended"` +- The configured device UUID +- The exact allowed origins required by the record +- The configured profile-state key + +If unattended creation returns `202`, poll its status URL every 2s for at most 30s. Accept only a connected `200` or `201`. Treat `410` as terminal. Treat a timeout or any other failure as an operational error. + +### Return a mode-discriminated enrichment response + +Change `POST /v1/intake/records/:id/enrich-website` to return these fields in both modes: + +- `workflowId` +- `runId` +- `sessionId` +- `mode` +- `status` +- `approvalIds` + +Attended responses also return `webSocketUrl`. Unattended responses must never expose or require a session WebSocket URL. + +The response contract must discriminate on `mode`: + +```typescript +type EnrichmentStartResponse = { + workflowId: string; + runId: string; + sessionId: string; + status: string; + approvalIds: string[]; +} & ( + | { mode: "attended"; webSocketUrl: string } + | { mode: "unattended" } +); +``` + +### Centralize connector outcome handling + +Handle protocol-2 outcomes in one shared path: + +| Outcome | Required handling | +|---|---| +| `pending` | Poll the same command ID | +| `not_started` | Retry the same logical command and business idempotency key | +| `timed_out` | Retry reads and dry runs only | +| `unknown` | Never retry; fail the workflow and preserve audit evidence | + +Do not infer retry safety from an HTTP status alone. Use the connector’s typed outcome and `safeToRetry` contract. + +### Add durable browser-session leases + +Add `browser_session_leases` to `packages/worker/migrations/0001_init.sql` for new databases. Add the same schema as `packages/worker/sql/add-browser-session-leases.sql` for the existing production database. Keep Metamind’s single consolidated baseline migration; do not add a second file to `packages/worker/migrations/`. + +The table must contain: + +- `run_id` +- `record_id` +- `idempotency_key` +- `mode` +- `request_json` +- Nullable `session_id` +- `state` +- Nullable `last_error` +- `created_at` +- `updated_at` + +Use this state machine: + +```text +minting → active → cleanup_pending → released +``` + +Persist the mint intent before calling Understudy. A crash must not lose the business idempotency key or request needed to recover the mint. + +On workflow-start failure or a terminal FlowSafe status, move the lease to `cleanup_pending`. Terminal statuses are `success`, `failed`, `tripwire`, `canceled`, `bailed`, and `skipped`. + +Preserve leases for the nonterminal statuses `pending`, `running`, `waiting`, `suspended`, and `paused`. + +Extend the existing 15-minute maintenance cron to recover `minting` rows and retry `DELETE` for `cleanup_pending` rows: + +- Treat `204` as released +- Treat `202` as cleanup still pending +- Retain and alert on persistent `401`, `403`, `404`, or `5xx` +- Never mark an unresolved cleanup as released +- Use compare-and-clear updates so a stale sweep cannot overwrite newer state + +Apply the production D1 migration before deploying code that queries the table: + +```bash +pnpm exec wrangler d1 execute metamind --remote \ + --file packages/worker/sql/add-browser-session-leases.sql +pnpm exec wrangler d1 execute metamind --remote \ + --command "SELECT name FROM sqlite_schema WHERE name='browser_session_leases'" +``` + +Apply the additive file once. Save both command results as Phase 1 evidence. + +### Update the production proof runner + +Update `packages/worker/scripts/enrich-browser-runbook.mjs` to support attended and unattended modes. Its self-test must cover both. + +The unattended path must: + +- Verify the configured device is online and has capacity +- Create or replay the unattended lease +- Poll creation until the session is connected +- Avoid extension-token input +- Avoid manual tab attachment +- Verify cleanup and device usage after terminal workflow completion + +Keep the existing attended path and extension-token handling for attended proof only. + +### Stamp Metamind deployment provenance + +Replace the static `COMMIT=local` deployment path with an automated clean-tree deploy. The source-controlled deployment command must calculate the full SHA, reject a dirty tree, pass the SHA through Wrangler’s `COMMIT` variable, and include it in the deployment message. + +The automated command must implement this sequence: + +```bash +metamind_release_sha="$(git rev-parse HEAD)" +test -z "$(git status --short)" +pnpm build +test -z "$(git status --short)" +pnpm exec wrangler deploy \ + --var "COMMIT:$metamind_release_sha" \ + --message "release $metamind_release_sha" +``` + +After deployment, verify provenance: + +```bash +metamind_release_sha="$(git rev-parse HEAD)" +curl --fail-with-body --silent \ + https://metamind.proofof.tech/health | + jq -e --arg sha "$metamind_release_sha" \ + '.status == "ok" and .commit == $sha' +pnpm exec wrangler deployments status --json +``` + +Reject the deployment if `/health.commit` differs from the approved SHA. + +### Verify and deploy Phase 1 + +Run the Metamind Worker lane from the Metamind repository: + +```bash +pnpm --filter @repo/worker typecheck +pnpm --filter @repo/worker test +pnpm exec biome check packages/worker +node packages/worker/scripts/enrich-browser-runbook.mjs --self-test +pnpm build +bash scripts/validate-build.sh +git diff --check +``` + +Promote the reviewed Metamind change to `master`, deploy it with `UNDERSTUDY_SESSION_MODE=attended`, and rerun the existing attended production proof. Record: + +- The full Metamind SHA +- CI URL +- D1 migration result +- Deployment and version IDs +- `/health` result +- Attended proof artifact and UTC completion time + +Do not begin Phase 2 until the attended proof passes. + +## Phase 2: Deploy the Understudy migration-v2 baseline + +Deploy migration `v2` with all unattended tenant flags off. This deployment becomes the rollback baseline for every later Understudy configuration deployment. + +### Verify the exact release source + +Refresh refs and check out the approved full SHA. Do not deploy the stale local `master`. + +Set `understudy_release_sha` to the full SHA approved in the ledger: + +```bash +understudy_release_sha="full_approved_sha_here" +git fetch --prune +test -z "$(git status --short)" +git cat-file -e "$understudy_release_sha^{commit}" +git switch --detach "$understudy_release_sha" +test "$(git rev-parse HEAD)" = "$understudy_release_sha" +``` + +Run from the Understudy repository: + +```bash +pnpm install --frozen-lockfile +pnpm build +pnpm typecheck +pnpm test +pnpm --filter @understudy/backend exec wrangler deploy --dry-run \ + --outdir /tmp/understudy-unattended-worker +pnpm --filter @understudy/backend exec wrangler secret list +``` + +Confirm these six secret names exist: + +- `AUTH_HMAC_SECRET` +- `CALLER_TOKENS` +- `EXTENSION_TOKENS` +- `DEVICE_TOKENS` +- `WS_TICKET_SECRET` +- `VAULT_MASTER_KEY` + +Do not print or record secret values. + +### Deploy with flags off + +Confirm the source-controlled configuration contains: + +```json +{ + "UNATTENDED_ENABLED_TENANTS": "[]", + "SAFE_WRITE_REQUIRED_TENANTS": "[]" +} +``` + +Deploy with a message containing the full approved SHA: + +```bash +understudy_release_sha="$(git rev-parse HEAD)" +test -z "$(git status --short)" +pnpm --filter @understudy/backend exec wrangler deploy \ + --message "release $understudy_release_sha" +pnpm --filter @understudy/backend exec wrangler deployments status --json +``` + +Read the active version ID from the status result, then inspect it: + +```bash +understudy_version_id="active_version_uuid_here" +pnpm --filter @understudy/backend exec wrangler versions view \ + "$understudy_version_id" --json +``` + +Verify the active version contains: + +- Migration tag `v2` +- Exports for `SessionAgent`, `DeviceAgent`, and `TenantDeviceCoordinator` +- Durable Object bindings `SESSION`, `DEVICE`, and `TENANT_CONTROL` +- `ANALYTICS` +- `RATE_LIMITER` +- `VAULT` +- All six required secrets +- Quota configuration +- Both rollout variables + +Then verify: + +```bash +curl --fail-with-body \ + https://understudy-backend.gcharang.workers.dev/health +``` + +Send one attended session request with no body and no `Content-Type`: + +```bash +curl --fail-with-body \ + --request POST \ + --header "Authorization: Bearer caller_token_here" \ + --header "Idempotency-Key: 00000000-0000-4000-8000-000000000021" \ + https://understudy-backend.gcharang.workers.dev/v1/sessions +``` + +Confirm it succeeds, then rerun the Metamind attended proof. + +Record the exact active version ID as `UNDERSTUDY_V2_FLAGS_OFF_VERSION` in the Phase 2 ledger row and operator record. Also record the deployment ID, approved SHA, status JSON, health result, and attended proof. + +The initial `v1 → v2` migration is a stop-and-forward-fix boundary. Do not attempt to remove `v2` or roll back to the current `v1` version. + +## Phase 3: Provision and accept the canary device + +Use one enrolled, tenant-dedicated production profile for acceptance. Build the extension from the same approved Understudy SHA deployed in Phase 2. + +### Build and load the extension + +Run: + +```bash +pnpm --filter @understudy/protocol build +pnpm --filter @understudy/extension typecheck +pnpm --filter @understudy/extension test +pnpm --filter @understudy/extension build +``` + +Load `apps/extension/.output/chrome-mv3/` through `chrome://extensions`. Do not use a development build. + +### Enroll the canary + +Provision one device UUID and one raw credential. Store only the credential’s SHA-256 digest in `DEVICE_TOKENS`. + +Enroll the dedicated profile with exactly: + +- `https://example.com` +- `https://practice.expandtesting.com` + +Confirm the device reports: + +- Protocol 2 +- Expected extension and browser versions +- Capacity 2 +- Usage 0 +- A recent heartbeat + +After the upgraded Metamind connector is live, deploy this Understudy configuration: + +```json +{ + "UNATTENDED_ENABLED_TENANTS": "[\"metamind\"]", + "SAFE_WRITE_REQUIRED_TENANTS": "[\"metamind\"]" +} +``` + +Do not use `"*"`. + +Record the allowlist deployment and version IDs before running acceptance. + +### Run the Chromium acceptance suite + +Execute every scenario in [`apps/extension/RUNBOOK.md`](../apps/extension/RUNBOOK.md): + +- Two-tab routing isolation +- Capacity failure +- Origin collision failure +- Profile-state collision failure +- Redirect containment +- Paused-popup containment +- One-slot cleanup +- Extension service-worker eviction +- Chrome restart +- Unknown write outcome +- Device credential rotation +- Hard and idle expiry +- Attended compatibility + +Stop the rollout on any cross-tab routing, duplicate tab, capacity leak, unexpected recovery, origin escape, expiry mismatch, or granted write replay. + +### Run the read-only soak + +Run the complete 24-hour read-only soak. Do not proceed if any event remains unexplained: + +- `command_unknown` +- Duplicate controlled tab +- Capacity leak +- Unexpected browser recovery +- Origin escape +- Hard-expiry or idle-expiry mismatch + +Record the device acceptance operator record, extension SHA, Understudy deployment and version IDs, start and end timestamps, telemetry, and final device usage. + +## Phase 4: Switch Metamind to unattended and prove governance + +Switch only the synthetic canary workflow. FlowSafe remains the approval authority, and Breakwater remains the connector governance and audit boundary. + +### Configure and deploy Metamind + +Set production Metamind to: + +```json +{ + "UNDERSTUDY_SESSION_MODE": "unattended", + "UNDERSTUDY_DEVICE_ID": "enrolled_canary_uuid_here", + "UNDERSTUDY_ALLOWED_ORIGINS": + "[\"https://example.com\",\"https://practice.expandtesting.com\"]", + "UNDERSTUDY_PROFILE_STATE_KEY": "metamind-practice-account" +} +``` + +Verify the Phase 1 D1 lease migration before deploying. Do not rerun the additive SQL if the table exists: + +```bash +pnpm exec wrangler d1 execute metamind --remote \ + --command "SELECT name FROM sqlite_schema WHERE name='browser_session_leases'" +``` + +Deploy a clean, approved Metamind `master` SHA through the stamped deployment command. Confirm `/health.commit` equals that full SHA. + +### Run the governed unattended proof + +The proof must complete every step: + +1. Create or replay the unattended lease. +2. Reach connected status without manual tab attachment. +3. Produce both dry-run previews. +4. Suspend at the FlowSafe approval. +5. Prove no real write occurred before approval. +6. Approve with the expected actor and suspension fingerprint. +7. Read the public page. +8. Type the non-secret username. +9. Fill the vaulted password. +10. Observe the authenticated marker. +11. Persist enrichment evidence. +12. Correlate FlowSafe, Breakwater, Metamind audit, Understudy session, and command IDs. +13. Reach a terminal workflow state. +14. Confirm `DELETE` reaches `204`. +15. Confirm device usage returns to zero. +16. Confirm the lease row reaches `released`. + +Fail the gate if the run retries an unknown outcome, writes before approval, loses audit correlation, or leaves a stale lease. + +### Store the proof evidence + +Store the proof artifact with mode `0600`. It must contain no raw credentials, tokens, secret values, password text, or credential-bearing URLs. + +Record: + +- Artifact path and SHA-256 hash +- Proof start and end timestamps +- Understudy and Metamind full release SHAs +- Both deployment IDs +- Both version IDs +- D1 migration evidence +- Session, command, workflow, run, approval, and audit correlation IDs +- Final lease state and device usage + +Update the Phase 4 ledger row in the same evidence pull request. + +## Phase 5: Ramp allowlisted production traffic + +Add a real website only when its exact origin appears in both Metamind’s source-controlled allowlist and the extension enrollment. Do not use `"*"` or dashboard-only configuration. + +Keep execution sequential while all workflows use `metamind-practice-account`. The shared profile-state key is an intentional account-concurrency fence. + +### Stage 1: Run one record + +Run one operator-selected production record. Observe it for 2 hours. + +### Stage 2: Run five records + +After Stage 1 passes, run five production records sequentially. Observe them for 8 hours. + +### Stage 3: Enable all reviewed origins + +After Stage 2 passes, enable all traffic for the reviewed allowlisted origins. Observe it for 24 hours. + +### Apply every ramp gate + +Require all conditions at each stage: + +- No unexpected unknown write +- No retry after an unknown outcome +- No write before approval +- No stale `minting`, `active`, or `cleanup_pending` lease after a terminal workflow +- Device usage returns to zero +- No origin-policy rejection for an approved origin +- No cross-tab routing +- No duplicate controlled tab +- No unexpected browser recovery +- Correct FlowSafe decision and Breakwater audit correlation +- Expected session creation, provisioning, release, and expiry telemetry +- No material increase in Worker error rate or handler duration + +Before each stage, record the comparison interval and numeric error-rate and handler-duration thresholds. Do not choose a threshold after observing the stage. + +Record the Analytics Engine result with this query: + +```sql +SELECT + blob1 AS event, + blob2 AS outcome, + count() AS total +FROM understudy_telemetry +WHERE timestamp > NOW() - INTERVAL '1' DAY +GROUP BY event, outcome +ORDER BY event, outcome +``` + +Run the query through the [Analytics Engine SQL API](https://developers.cloudflare.com/analytics/analytics-engine/sql-api/). Preserve the query, UTC interval, raw result, and operator interpretation with each stage’s evidence. + +Do not advance a stage on an unexplained anomaly. Set the current gate to `Blocked`, attach the evidence, and either fix forward or execute rollback. + +## Phase 6: Close the rollout + +Close the rollout only after the all-allowlisted 24-hour soak passes. + +1. Leave `UNATTENDED_ENABLED_TENANTS=["metamind"]`. +2. Leave `SAFE_WRITE_REQUIRED_TENANTS=["metamind"]`. +3. Do not switch either flag to wildcard enablement. +4. Fill every ledger field with full SHAs, CI URLs, deployment and version IDs, D1 evidence, device acceptance, telemetry, proof artifacts, and UTC timestamps. +5. Verify the recorded rollback version still exists among Cloudflare’s available Worker versions. +6. Mark Phase 6 `Passed`. +7. Move optional future work into separate issues. + +Do not convert a failed release gate into a TODO. + +## Roll back safely + +Migration `v2` is additive and irreversible. Cloudflare blocks rollback when a Durable Object class lifecycle change separates the active and target versions. The current migration-`v1` production version is therefore not a valid rollback target after `v2` deploys. See [Cloudflare Worker rollback constraints](https://developers.cloudflare.com/workers/versions-and-deployments/rollbacks/). + +Execute rollback in this order: + +1. Switch Metamind back to `UNDERSTUDY_SESSION_MODE=attended`. +2. Roll Understudy back to `UNDERSTUDY_V2_FLAGS_OFF_VERSION`, or redeploy its exact approved source state. +3. Confirm new unattended leases are disabled. +4. `DELETE` every active unattended lease and poll each `202` until `204`. +5. Let the durable sweeper retain and retry unresolved cleanup. +6. Preserve migration `v2`, coordinator data, audit evidence, and unknown-outcome records. +7. Never replay a command with an unknown external outcome. +8. Rerun the attended production proof. +9. Mark affected ledger gates `Rolled back` with deployment IDs, evidence, UTC time, and owner. + +Use the recorded version ID: + +```bash +UNDERSTUDY_V2_FLAGS_OFF_VERSION="version_uuid_here" +pnpm --filter @understudy/backend exec wrangler rollback \ + "$UNDERSTUDY_V2_FLAGS_OFF_VERSION" \ + --message "rollback to migration-v2 flags-off baseline" +pnpm --filter @understudy/backend exec wrangler deployments status --json +``` + +If Cloudflare refuses the rollback, stop. Confirm the version’s exports, migrations, and bindings. Fix forward from migration `v2`; never remove its Durable Object classes. + +Rollback is complete only when attended proof passes, no new unattended lease can start, all resolvable leases reach `released`, and unresolved leases remain durable with alerts. + +## Verify implementation changes + +Run the full Understudy checks for any Understudy implementation or configuration change: + +```bash +pnpm build +pnpm typecheck +pnpm test +git diff --check +``` + +Run the Metamind Worker lane for every Metamind implementation or configuration change: + +```bash +pnpm --filter @repo/worker typecheck +pnpm --filter @repo/worker test +pnpm exec biome check packages/worker +node packages/worker/scripts/enrich-browser-runbook.mjs --self-test +pnpm build +bash scripts/validate-build.sh +git diff --check +``` + +The required automated scenarios are: + +- Attended creation sends no request body and works with old and new Understudy backends +- Unattended creation validates exact origins, device UUID, profile key, idempotent replay, `202` polling, terminal status, and timeout +- Attended responses contain `webSocketUrl`; unattended responses omit it +- Pending commands poll the same ID +- Not-started retries reuse the logical key +- Unknown outcomes never retry +- Timed-out writes never retry automatically +- Crashes after mint intent, Understudy creation, FlowSafe start, terminal cleanup, and compare-and-clear converge without losing or double-releasing a lease +- Cleanup handles `DELETE 202 → 204`, transient `5xx`, and terminal workflow states +- The proof-runner self-test covers both modes +- `/health.commit` matches the deployed full SHA +- Existing Metamind and Understudy tests remain green + +## Preserve rollout decisions + +These decisions remain in force until a reviewed change updates this runbook: + +- **Canonical runbook**: Keep execution details here so the technical plan remains a conceptual architecture document +- **Metamind proof**: Require a real governed consumer to prove FlowSafe, Breakwater, audit correlation, and cleanup together +- **Attended toggle**: Preserve it until rollout closes so consumer rollback does not depend on unattended recovery +- **Exact origin allowlists**: Keep Metamind and extension policy reviewable and identical +- **Durable D1 cleanup**: Use persisted leases and the existing maintenance cron so Worker crashes cannot lose cleanup work +- **Staged ramp**: Use `1 → 5 → all` to constrain blast radius and require evidence between stages +- **Source-controlled variables**: Make each production configuration reviewable and reproducible +- **Migration-`v2` baseline**: Establish a flags-off version because Cloudflare cannot roll back across the class lifecycle change + +The rollout rejected: + +- **Immediate unattended-only cutover**: It removes the attended recovery path before unattended proof exists +- **Best-effort cleanup**: Idle expiry cannot replace durable ownership and explicit release +- **Arbitrary record origins**: Dynamic policy would bypass source review and extension enrollment +- **Wildcard tenant enablement**: It expands the failure domain beyond the proven consumer +- **Dashboard-only configuration**: It breaks reproducibility and review +- **Direct rollback to migration `v1`**: Cloudflare blocks the incompatible Durable Object lifecycle rollback + +## Keep excluded work out of the rollout + +Do not add these features while executing this runbook: + +- Automated extension distribution or enrollment +- Dynamic fleet or origin-policy administration +- Cross-tenant profiles +- Storage isolation between tabs in one profile +- More than two controlled tabs per device +- New browser providers +- Local daemons +- Automatic URL restoration +- New package releases unless package code changes +- Changes to Metamind’s primary intake workflow +- Any Gmail send path + +These exclusions do not waive release gates or permit temporary unsafe behavior. + +## Appendix: current-state anchors + +These anchors pin the initial edit targets. Understudy snippets are from `e4b98e6824b2dbee078a7c57da37a11f389010b9`. Metamind snippets are from `ee94790ddf92b8fabebba10a502e76005a57e17d`. Re-read each file and symbol before editing because line numbers and surrounding code will drift. + +### Release flow is complete + +`README.md`, `Release published packages`: + +```markdown +1. Merge `Version Packages` into `dev`. +2. Verify the versioned `dev` commit. +3. Open and merge a promotion pull request from `dev` to `master`. +``` + +This supports the Phase 0 decision to avoid another package release unless package code changes. + +### Understudy distinguishes attended creation by an absent body + +`apps/backend/src/index.ts`, `POST /v1/sessions`: + +```typescript +if (c.req.raw.body === null) { +``` + +Metamind currently violates that contract in `packages/worker/src/intake/browser-connectors.ts`, `createBrowserSession`: + +```typescript +headers: { + "content-type": "application/json", + authorization: `Bearer ${config.UNDERSTUDY_TOKEN}`, + "idempotency-key": idempotencyKey, +}, +body: "{}", +``` + +Implement attended mode with no `body` property and no `content-type` header. Implement unattended mode with the JSON body and header. + +### Metamind uses older client packages + +`packages/worker/package.json`: + +```json + "@understudy/connector": "^0.4.0", + "@understudy/protocol": "^0.6.0", +``` + +Upgrade these dependencies to the Phase 1 versions and update the lockfile. + +### The enrichment route always exposes an attended socket + +`packages/worker/src/routes/intake.ts`, `POST /records/:id/enrich-website`: + +```typescript +return c.json( + { + workflowId: LEAD_ENRICH_BROWSER_WORKFLOW_ID, + ...started, + webSocketUrl: browserWebSocketUrl(c.env, started.sessionId), + }, + 202, +); +``` + +Replace this shape with the mode-discriminated response from Phase 1. + +### Durable recovery has a reusable pattern + +`packages/worker/src/intake/resume-recovery.ts`, resume sweep: + +```typescript + const parked = await listResumeRecoveries(env.DB); + for (const entry of parked) { + try { + await enqueueJobs(env, [ + { + kind: "resume", + recordId: entry.recordId, + record: entry.record as ApprovalRecord, + attempt: 1, + }, + ]); + await clearResumeRecovery(env.DB, entry.recordId, entry.recordJson); +``` + +Reuse its durable marker, retry containment, and compare-and-clear pattern for browser-session leases. + +### Metamind deployment provenance is not stamped + +Both Metamind Wrangler configurations currently contain: + +```json + "VERSION": "0.0.0-dev", + "COMMIT": "local", +``` + +The live Worker also reports `COMMIT=local`. Phase 1 must replace this deployment path before unattended proof evidence is accepted. + +### Understudy source and production infrastructure differ + +`apps/backend/wrangler.jsonc` declares: + +```json +{ "name": "DEVICE", "class_name": "DeviceAgent" }, +{ "name": "TENANT_CONTROL", "class_name": "TenantDeviceCoordinator" } +``` + +It also declares: + +```json +{ + "tag": "v2", + "new_sqlite_classes": ["DeviceAgent", "TenantDeviceCoordinator"] +} +``` + +The production baseline remains on migration `v1` with only `SessionAgent`. This difference makes Phase 2 the first infrastructure deployment and makes the `v2`, flags-off version the first safe rollback target. From f8b05bd24875f57c588da12c94d457400f244e07 Mon Sep 17 00:00:00 2001 From: gcharang <21151592+gcharang@users.noreply.github.com> Date: Tue, 28 Jul 2026 04:11:02 +0400 Subject: [PATCH 3/3] docs(rollout): record Phase 1 implementation and its deltas MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Splits Phase 1 into implementation (1a) and attended deployment (1b): the code exists and its local lane is green, but nothing is merged or deployed, and conflating the two would let a code commit read as a passed release gate — which this runbook exists to prevent. Records Phase 2's source preconditions as verified without a deployment (dry-run bindings, migration tag, flags-off variables), so the operator starts from what is already checked rather than rechecking it. Three implementation facts diverge from what this document specified, and are recorded rather than left to drift: - The lease state machine gained `abandoned`. `released` must keep meaning "confirmed 204" for the ramp gates and operator queries to be readable, so a lease that provably owns nothing needed its own terminal state instead of being folded in. - Deploy-time `--var COMMIT` is not sufficient on its own. Metamind auto-deploys its default branch with a bare `wrangler deploy`, which would ship the placeholder and silently un-stamp a stamped deployment; the commit is now baked at build time so both paths carry it. Added as a Phase 1b precondition. - The origin allowlist binds attended mode too. Spelled out because it has a real operational consequence once Phase 1b ships the canary origins while the mode is still attended. Co-Authored-By: Claude Opus 5 --- docs/unattended-production-rollout.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/docs/unattended-production-rollout.md b/docs/unattended-production-rollout.md index a669261..bf5599c 100644 --- a/docs/unattended-production-rollout.md +++ b/docs/unattended-production-rollout.md @@ -38,9 +38,10 @@ Never mark a gate `Passed` without its approved SHA, deployment disposition, evi | Gate | State | Approved SHA | Deployment | Evidence | Completed | Owner | |---|---|---|---|---|---|---| | Phase 0a: release-flow baseline | Passed | `4843b6bccd8e1028c8fb6dba7812d643a4106778` | Not applicable: package and branch-flow gate | [PR #20](https://github.com/ProofOfTechOrg/understudy/pull/20), [PR #21](https://github.com/ProofOfTechOrg/understudy/pull/21), [master CI](https://github.com/ProofOfTechOrg/understudy/actions/runs/30255565127), [Version](https://github.com/ProofOfTechOrg/understudy/actions/runs/30255262164), [Release](https://github.com/ProofOfTechOrg/understudy/actions/runs/30255565196) | `2026-07-27T09:51:30Z` | Release operator | -| Phase 0b: rollout runbook merged | In progress | Pending | Not applicable: documentation gate | This file; merge pull request pending | Pending | Engineering | -| Phase 1: Metamind compatible and recoverable in attended mode | Not started | Pending | Current baseline: deployment `f85a53c6-7b90-4b6f-a427-2e8cef7df637`, version `b7890ecc-b4c4-489f-a67f-3cafbca67b6a` | Pending | Pending | Engineering | -| Phase 2: Understudy `v2`, flags-off rollback baseline | Not started | Pending | Current baseline: deployment `b73220f0-8035-40d2-9987-243770d96306`, version `41434382-ecdd-4f95-a27c-811c4337b6bd` | Pending | Pending | Release operator | +| Phase 0b: rollout runbook merged | In progress | Pending | Not applicable: documentation gate | This file, committed to `dev@ee105ac7de0b0a8022e815f601fc991a0c8144d7`; `dev → master` promotion pull request pending | Pending | Engineering | +| Phase 1a: Metamind implementation | In progress | Metamind `e602b3b32426e35490d8f1df6bdc3f7ebebbd9de` (branch `feat/unattended-phase-1`, not yet merged) | Not applicable: implementation gate | Local lane green: typecheck, 716 tests, Biome (0 errors), proof-runner self-test both modes, `pnpm build` + `validate-build.sh`, `git diff --check`. Both migration copies apply to SQLite with identical schemas and the additive file is idempotent. Not merged, not deployed, no CI run yet | Pending | Engineering | +| Phase 1b: Metamind attended deployment and proof | Not started | Pending | Current baseline: deployment `f85a53c6-7b90-4b6f-a427-2e8cef7df637`, version `b7890ecc-b4c4-489f-a67f-3cafbca67b6a` | Pending: needs the D1 lease migration, a stamped deploy with `UNDERSTUDY_SESSION_MODE=attended`, `/health.commit` verification, and the attended production proof | Pending | Release operator | +| Phase 2: Understudy `v2`, flags-off rollback baseline | Not started | Pending | Current baseline: deployment `b73220f0-8035-40d2-9987-243770d96306`, version `41434382-ecdd-4f95-a27c-811c4337b6bd` | Source preconditions verified on `dev@ee105ac7de0b0a8022e815f601fc991a0c8144d7` (no deployment): `pnpm install --frozen-lockfile`, `build`, `typecheck`, `test` (194 backend) all pass, and `wrangler deploy --dry-run` confirms migration `v2`, the `SESSION`/`DEVICE`/`TENANT_CONTROL` bindings, `ANALYTICS`, `RATE_LIMITER`, `VAULT`, quota policy, and both rollout variables at `"[]"`. Still pending: `wrangler secret list`, the deploy itself, and the active-version inspection | Pending | Release operator | | Phase 3a: canary device acceptance | Not started | Pending | Pending | Pending | Pending | Canary operator | | Phase 3b: 24-hour read-only soak | Not started | Pending | Pending | Pending | Pending | Canary operator | | Phase 4: governed Metamind unattended proof | Not started | Pending | Pending | Pending | Pending | Canary operator | @@ -144,6 +145,8 @@ Set the synthetic canary origins to exactly: Reject a record origin that is absent from this list before creating an Understudy session. Expanding production scope requires one reviewed change that updates both Metamind’s source-controlled allowlist and the extension’s local origin policy. +The list binds BOTH modes whenever it is set, covers every origin the workflow visits (the record’s own and the portal it authenticates against), and is re-checked at each approved side-effect boundary so a run suspended across a policy narrowing cannot resume against origins the list no longer permits. Note the operational consequence: once Phase 1b deploys these canary origins while the mode is still `attended`, attended enrichment of any other origin is refused with `403 origin_not_allowed` until that origin is added by reviewed change. That is the intended fail-closed behavior, not a regression. + ### Create attended and unattended sessions correctly Attended creation must omit both `body` and `Content-Type`. Understudy treats any body, including `"{}"`, as an unattended request. @@ -219,9 +222,16 @@ Use this state machine: ```text minting → active → cleanup_pending → released + ↘ abandoned ``` -Persist the mint intent before calling Understudy. A crash must not lose the business idempotency key or request needed to recover the mint. +`released` means Understudy CONFIRMED cleanup with `204`. `abandoned` is a separate terminal state for a lease that provably owns nothing — Understudy refused the creation outright, or had already disposed of the session. The two are deliberately distinct: an operator auditing confirmed cleanups must not silently get rows where nothing was ever confirmed, and a lease left in `cleanup_pending` for a session that never existed would retry and alert forever, failing the ramp gate on what is ordinary, self-clearing device contention. + +Persist the mint intent before calling Understudy, and the session id before waiting for it to connect. A crash must not lose the business idempotency key or request needed to recover the mint; recording the id before the connect wait is what lets an interrupted creation be resolved with a `DELETE` instead of another creation. + +Classify a failed mint before cleaning up after it. A refusal (`409`, `429`, `503`) means nothing was created, so the lease is abandoned. Any other failure means the outcome is unknown and something may exist, so the lease is LEFT in `minting` for the sweep to replay and discover — moving it out disables the only path that can find that orphan. + +Never re-mint under a terminal lease that held a session. Flowsafe can forget a completed run once its retention window passes, so "no run found" is not proof that none ever ran; re-minting would silently re-execute the workflow's browser writes. Refuse with a conflict and require a fresh idempotency key. On workflow-start failure or a terminal FlowSafe status, move the lease to `cleanup_pending`. Terminal statuses are `success`, `failed`, `tripwire`, `canceled`, `bailed`, and `skipped`. @@ -265,6 +275,10 @@ Keep the existing attended path and extension-token handling for attended proof Replace the static `COMMIT=local` deployment path with an automated clean-tree deploy. The source-controlled deployment command must calculate the full SHA, reject a dirty tree, pass the SHA through Wrangler’s `COMMIT` variable, and include it in the deployment message. +A deploy-time `--var` alone is not sufficient, because it stamps only the path that passes it. Metamind also auto-deploys its default branch through Cloudflare Workers Builds, whose deploy command is a bare `wrangler deploy` — that path would ship the `COMMIT` placeholder from `wrangler.jsonc` and silently un-stamp a previously stamped deployment, invalidating recorded evidence after the fact and with no signal. The commit is therefore ALSO baked into the bundle at build time (`packages/worker/vite.config.ts`, from `WORKERS_CI_COMMIT_SHA` or `git rev-parse HEAD`) and preferred by `/health`. Both deploy paths then report a real SHA. + +Confirm before Phase 1b that a merge to Metamind’s default branch cannot land an unstamped deployment over a stamped one — either the auto-deploy is disabled for the rollout, or it is running a build that carries the SHA. + The automated command must implement this sequence: ```bash