docs: restore drill runbook and testnet cutover runbook

docker/docker-compose.dev.yml

@@ -10,16 +10,11 @@ services:
       POSTGRES_PASSWORD: password
       POSTGRES_DB: trident
     ports:
-      - \
-5432:5432\
+      - "5432:5432"
     volumes:
       - postgres_dev_data:/var/lib/postgresql/data
     healthcheck:
-      test: [\
-CMD-SHELL\, \pg_isready
--U
-trident
-trident\]
+      test: ["CMD-SHELL", "pg_isready -U trident -d trident"]
       interval: 5s
       timeout: 5s
       retries: 10
@@ -28,8 +23,7 @@ trident\]
   redis:
     image: redis:7-alpine
     ports:
-      - \
-6379:6379\
+      - "6379:6379"
     volumes:
       - redis_dev_data:/data
     command: redis-server --appendonly yes

docs/RESTORE_DRILL_RUNBOOK.md

@@ -0,0 +1,140 @@
+# Postgres Restore Drill Runbook
+
+**Status: partially executed.** This documents a real, locally-performed
+restore drill (schema/tooling fidelity — see "What this drill proves" below)
+with genuine measured numbers. It does **not** replace the production-shaped
+drill #501 ultimately asks for, which needs a real staging environment and a
+real automated backup to restore from — **neither exists yet** (see
+"What's still missing" below). Treat this as the tooling/procedure baseline
+the production drill will build on, not as a substitute for it.
+
+## Why #431 blocks the full drill
+
+#501 depends on #431 ("automated Postgres backups with a restore we have
+actually performed"), which is **still open**. `grep -rl "backup\|pg_dump\|restore" docs/ scripts/`
+confirms there is no backup automation, no cron/CI job producing scheduled
+dumps, and no `scripts/backup.sh`/`restore.sh` in this repo. A "production-shaped
+backup from staging" (#501's first scope item) cannot be taken because there
+is no defined backup artifact or schedule to draw one from yet.
+
+## What this drill proves
+
+Rather than skip #501 entirely while #431 is open, this drill answers the
+parts of it that don't require production infrastructure or an existing
+backup pipeline: **does `pg_dump`/`pg_restore` actually round-trip this
+schema correctly, especially the partitioned `soroban_events` table**, and
+**what does a dump/restore of a realistically-sized dataset actually cost in
+wall-clock time**. Both are real, useful facts independent of where the
+backup schedule eventually lives.
+
+### What was actually done
+
+1. A local PostgreSQL 15.15 instance was started (matching `docker/docker-compose.dev.yml`'s
+   pinned `postgres:15-alpine`, verified via `psql -c "select version()"`).
+2. The real schema was applied from `database/schema.sql` (the same file
+   used for local/dev bootstrap).
+3. **Found and fixed a real bug this drill surfaced**: `docker/docker-compose.dev.yml`
+   was checked into this repo with corrupted YAML — every double-quoted
+   string (`"5432:5432"`, the `healthcheck.test` array) had been mangled
+   into literal backslash-newline sequences at some point in its history
+   (introduced in `699fce4`, "chore(db): integrate sqlx-cli database
+   migration management" / `#93` — confirmed clean at the file's creation
+   in `ca79d07`, already corrupted by `699fce4`; `#195` later removed an
+   already-corrupted `version:` key without noticing the same corruption
+   elsewhere in the file). `python3 -c "import yaml;
+   yaml.safe_load(...)"` confirmed the committed file fails to parse at
+   all — `docker compose -f docker/docker-compose.dev.yml up` would have
+   failed outright. Fixed in this same change (see the diff on this PR) by
+   restoring proper YAML string quoting.
+4. **`create_soroban_partition(bigint, bigint)` is documented in migration
+   `0017` as callable but is not actually defined in `database/schema.sql`**
+   — the convenience snapshot has drifted from the migration chain here.
+   Worked around locally by applying migration `0017`'s function definition
+   directly; flagging the drift as a separate, small gap for whoever owns
+   `database/schema.sql`'s upkeep (its own header comment says it "must
+   mirror the end state of that chain").
+5. Two real partitions were created via `create_soroban_partition` —
+   `soroban_events_p0_1999999` and `soroban_events_p2000000_3999999`.
+6. 50,000 representative rows were inserted into `soroban_events`, landing
+   in both partitions (29,196 / 20,804 — confirming the partition routing
+   itself works, not just that the parent table accepts inserts).
+7. **`pg_dump -Fc`** (custom format, the same one `pg_restore` expects) —
+   **8.0s wall-clock**, producing a 3.9 MB archive.
+8. Simulated 500 further events written *after* the backup was taken, to
+   create a real, measurable "data loss window" for the restore to reveal.
+9. **`pg_restore --no-owner --no-privileges`** into a brand-new, empty
+   database — **8s wall-clock**, zero errors.
+10. Verified against the restored database, not assumed:
+    - Row count: exactly 50,000 (not 50,500) — confirms the restore
+      captured precisely what was in the backup, no more, no less.
+    - `soroban_events` came back as a **partitioned table**, `RANGE
+      (ledger_sequence)`, with both partitions (`soroban_events_p0_1999999`,
+      `soroban_events_p2000000_3999999`) present as real child tables, not
+      collapsed into a flat table.
+    - Per-partition row counts matched the pre-backup source exactly
+      (29,196 / 20,804) — partition routing survived the round-trip.
+    - All 5 indexes came back intact, including the partial index
+      (`idx_soroban_events_contract_topic0 ... WHERE topic_0 IS NOT NULL`)
+      and the composite primary key `(ledger_sequence, id)`.
+
+### Measured numbers (this drill's dataset: 50,000 rows, 3.9 MB compressed)
+
+| Metric | Value |
+|---|---|
+| Backup (`pg_dump -Fc`) duration | 8.0s |
+| Restore (`pg_restore`) duration | 8s |
+| Backup artifact size | 3.9 MB |
+| Data loss window in this drill | 500 rows / ~83s of simulated writes |
+
+**These numbers do not extrapolate linearly to a production-sized database.**
+`pg_dump`/`pg_restore` duration scales with data volume and index count, not
+row count alone; a production `soroban_events` table with years of mainnet
+history across many more partitions will take meaningfully longer for both
+directions. Re-run this same procedure against a real backup once #431
+lands, and replace this table with those numbers before this runbook is
+relied on for a real incident.
+
+## What's still missing (blocks calling #501 fully done)
+
+- **A real backup to restore** — #431 is open; there is no scheduled
+  production backup this drill could have used instead of a synthetic one.
+- **A production-shaped dataset** — this drill's 50,000 synthetic rows
+  exercise the partitioning and index behavior correctly, but say nothing
+  about restore time at production scale (more partitions, more indexes,
+  larger individual rows via `topics`/`data` JSONB payloads).
+- **RTO under real operational conditions** — this drill ran against a
+  local, otherwise-idle Postgres instance. A real RTO measurement needs to
+  account for provisioning a replacement database, network transfer of the
+  backup artifact, and bringing the indexer back up against the restored
+  database (reconciling its ingest cursor — see `docs/runbooks/incident-response.md`'s
+  `TridentIndexerLagCritical` alert for what "caught back up" means
+  operationally).
+
+## Reproducing this drill
+
+```bash
+# 1. Start a local Postgres 15 instance (after the docker-compose.dev.yml fix in this PR):
+docker compose -f docker/docker-compose.dev.yml up -d postgres
+
+# 2. Apply schema + the partition-creation function (until database/schema.sql
+#    is updated to include it — see "What was actually done" step 4 above):
+psql -h localhost -U trident -d trident -f database/schema.sql
+psql -h localhost -U trident -d trident -c "$(sed -n '/CREATE OR REPLACE FUNCTION create_soroban_partition/,/^\$\$;/p' database/migrations/0017_soroban_events_partitioning.sql)"
+
+# 3. Create partitions and seed representative data, then:
+pg_dump -h localhost -U trident -d trident -Fc -f backup.dump
+
+# 4. Restore into a fresh database and verify:
+createdb -h localhost -U trident trident_restored
+pg_restore -h localhost -U trident -d trident_restored --no-owner --no-privileges backup.dump
+psql -h localhost -U trident -d trident_restored -c "\d+ soroban_events"
+```
+
+## Next steps once #431 lands
+
+1. Point this same procedure at #431's real scheduled backup artifact
+   instead of a synthetic dump.
+2. Re-run against a copy of staging's actual data volume.
+3. Replace the "Measured numbers" table above with those real figures.
+4. Cross-reference the resulting RPO/RTO into `docs/runbooks/testnet-cutover.md`'s
+   preconditions (see that runbook, added alongside this one).

docs/runbooks/testnet-cutover.md

@@ -0,0 +1,184 @@
+# Testnet cutover runbook
+
+**Status: documented procedure, not yet walked through by someone who
+didn't write it.** Issue #502's "done when" is a real dry-run against
+staging with a second person driving from this document alone. That walkthrough
+needs staging access and a second engineer, neither available in this pass.
+What follows is the ordered procedure, grounded in what this repo's
+deployment tooling, CI gates, and prior runbooks actually do — not generic
+advice.
+
+## Preconditions
+
+All of the following must be true before cutover starts. None of these are
+new checks invented for this runbook — they're the existing gates this repo
+already has, gathered into one list.
+
+### CI / code
+
+- [ ] The commit being deployed passed CI's `docker`, `e2e`, and
+      `e2e-contract-events` jobs (see [`docs/CI.md`](../CI.md)) on `dev`.
+- [ ] Coverage floors are met (`docs/CI.md`'s "Coverage: collection and
+      enforced floors" section) — a coverage regression here is a signal
+      the change wasn't tested as thoroughly as the rest of the codebase.
+- [ ] `scripts/lint-migrations.sh` and `scripts/check-schema-drift.sh` are
+      green if the release includes a migration. **Known gap, found while
+      writing this runbook**: `check-schema-drift.sh`'s own documentation
+      states it compares tables/columns/indexes/constraints — it does
+      **not** compare functions. `create_soroban_partition` (added in
+      migration `0017`) is missing from `database/schema.sql` today and
+      this check does not catch it (see
+      [`RESTORE_DRILL_RUNBOOK.md`](../RESTORE_DRILL_RUNBOOK.md)'s
+      "What was actually done" step 4). Don't rely on this check alone to
+      certify a migration that adds or changes a function.
+
+### Which issues must be closed
+
+- [ ] [#431](https://github.com/Telocel-Labs/Trident/issues/431) —
+      automated backups with a real restore performed. **Currently open.**
+      Cutting over to testnet without this means the first real incident
+      that needs a restore will be the first time anyone has ever
+      performed one. See
+      [`RESTORE_DRILL_RUNBOOK.md`](../RESTORE_DRILL_RUNBOOK.md) for what's
+      been verified so far (schema/partition round-trip, real dump/restore
+      timing) and what's still missing (a real backup to restore, at
+      production-shaped scale).
+- [ ] [#460](https://github.com/Telocel-Labs/Trident/issues/460) — rollback
+      rehearsed. **Closed**, but its own runbook
+      ([`ROLLBACK_RUNBOOK.md`](../ROLLBACK_RUNBOOK.md)) is explicitly
+      marked "template — not yet rehearsed" and documents a real, load-bearing
+      finding: **there are zero `.down.sql` files for any of the 25
+      migrations** — schema rollback is not automated today. Read that
+      runbook's "Finding: migrations here are forward-only" section before
+      cutover, especially if the release being cut over includes a
+      migration.
+- [ ] [#445](https://github.com/Telocel-Labs/Trident/issues/445) —
+      incident response process with a named on-call owner. **Closed** —
+      [`incident-response.md`](incident-response.md) exists with severity
+      levels, escalation path, and a communication channel. Its "On-call
+      owner — launch week" section is a `[FILL IN: ...]` placeholder as of
+      this writing — confirm it has real names/contacts filled in before
+      cutover, not just that the section exists.
+- [ ] Row 1 of [`LAUNCH_CHECKLIST.md`](../LAUNCH_CHECKLIST.md) (alerts
+      verified firing) and row 9 (rollback rehearsed within the last 30
+      days) are checked off with evidence, per that checklist's own no-go
+      criteria.
+
+## Ordered cutover steps
+
+Each step names an owner role (not a person — fill in the actual name in
+the walkthrough) and a verification to run immediately after, not deferred
+to the end.
+
+### 1. Freeze `dev` and cut the release branch — *release owner*
+
+```bash
+git checkout dev && git pull
+git checkout -b release/testnet-cutover-<date>
+```
+
+**Verify:** CI is green on the release branch (same jobs as the preconditions
+above, re-run on the exact commit being cut over — not assumed still-green
+from when it merged to `dev`).
+
+### 2. Confirm environment configuration for testnet — *deploy owner*
+
+Per [`docs/ENVIRONMENT.md`](../ENVIRONMENT.md), `NETWORK` defaults to
+`testnet` and needs no override for a testnet deployment; confirm the
+target `.env` does not have a stale `NETWORK=mainnet` or
+`NETWORK=futurenet` left over from a previous environment's config.
+
+**Verify:**
+```bash
+grep -E '^NETWORK=' .env   # expect: NETWORK=testnet, or absent (defaults to testnet)
+```
+
+### 3. Take a pre-cutover database snapshot — *database owner*
+
+Even without #431's automated backups yet, take a manual one immediately
+before cutover so there is at least one restore point:
+
+```bash
+pg_dump -Fc -h <host> -U trident -d trident -f pre-cutover-$(date -u +%Y%m%dT%H%M%SZ).dump
+```
+
+**Verify:** the dump file exists and is non-empty; spot-check its size is in
+the expected range for the current database (compare against the previous
+manual snapshot, if any — a dump an order of magnitude smaller than
+expected usually means a connection/permission problem, not an empty
+database).
+
+### 4. Run migrations — *database owner*
+
+Per [`docs/deployment.md`](../deployment.md#5-start-postgresql-and-run-database-migrations)'s
+existing migration procedure.
+
+**Verify:** `sqlx migrate info` (or the project's equivalent) shows every
+migration applied, none pending. Re-run `scripts/check-schema-drift.sh`
+against the now-migrated database to confirm no drift was introduced by
+this release — subject to the function-comparison gap noted in
+Preconditions above.
+
+### 5. Deploy the indexer and API — *deploy owner*
+
+Per [`docs/deployment.md`](../deployment.md#6-start-all-services)'s existing
+deploy steps.
+
+**Verify:** immediately after, per
+[`docs/deployment.md`](../deployment.md#7-verify-health):
+```bash
+curl -sf https://<host>/v1/health
+```
+Confirm the indexer's ingest cursor is advancing (not stuck at whatever it
+was pre-cutover) — a health check passing does not by itself confirm the
+indexer resumed consuming testnet ledgers.
+
+### 6. Watch for the first full ingest cycle — *on-call owner*
+
+Per [`incident-response.md`](incident-response.md)'s alert catalog,
+specifically `TridentIndexerHeartbeatStale` and `TridentIndexerLagCritical`
+— these are exactly the signals that would fire if cutover left the indexer
+unable to reach testnet RPC or resume its cursor correctly.
+
+**Verify:** no SEV-1/SEV-2 alert fires within one full expected ingest
+cycle after cutover. Manually query recent `soroban_events` rows and
+confirm `ledger_timestamp` values are current, not frozen at the pre-cutover
+watermark.
+
+### 7. Announce cutover complete — *release owner*
+
+Per [`incident-response.md`](incident-response.md#user-communication-channel)'s
+existing communication channel — post confirmation there, not ad hoc.
+
+## Rollback trigger and procedure
+
+**Trigger**: any SEV-1 per [`incident-response.md`](incident-response.md#sev-1-service-down-or-data-incorrect)'s
+definition within the first ingest cycle after cutover, or a health check
+that never turns green within 15 minutes of step 5.
+
+**Who calls it**: per [`LAUNCH_CHECKLIST.md`](../LAUNCH_CHECKLIST.md#rollback-decision-procedure) —
+the on-call engineer, or the release owner if reachable within 5 minutes.
+Do not wait for a quorum.
+
+**Procedure**: follow [`ROLLBACK_RUNBOOK.md`](../ROLLBACK_RUNBOOK.md) in
+full. If this release included a migration, read that runbook's
+"Rollback across a migration boundary" section first — **a
+backward-incompatible migration currently has no automated reverse path**,
+which may mean the correct rollback for this specific release is
+"roll back the application only, leave the schema" (if the migration was
+additive) rather than a full schema rollback. Decide which case applies
+*before* cutover, not while already mid-incident.
+
+## What this runbook still needs (not done here)
+
+- [ ] A real walkthrough by someone who did not write this document,
+      driving from this file alone against a real staging environment —
+      this is #502's literal "done when" criterion and the single most
+      important gap.
+- [ ] #431 landed, so precondition checkbox 1 above can actually be
+      checked rather than flagged as open.
+- [ ] `incident-response.md`'s on-call section filled in with real names.
+- [ ] The exact commands in each step re-verified against whatever the
+      target testnet environment's actual hostnames/credentials are (the
+      commands above use placeholders consistent with `deployment.md`'s
+      existing style).