diff --git a/docker/docker-compose.dev.yml b/docker/docker-compose.dev.yml index 9072100e..6f0a71c2 100644 --- a/docker/docker-compose.dev.yml +++ b/docker/docker-compose.dev.yml @@ -10,16 +10,11 @@ services: POSTGRES_PASSWORD: password POSTGRES_DB: trident ports: - - \ -5432:5432\ + - "5432:5432" volumes: - postgres_dev_data:/var/lib/postgresql/data healthcheck: - test: [\ -CMD-SHELL\, \pg_isready --U -trident -trident\] + test: ["CMD-SHELL", "pg_isready -U trident -d trident"] interval: 5s timeout: 5s retries: 10 @@ -28,8 +23,7 @@ trident\] redis: image: redis:7-alpine ports: - - \ -6379:6379\ + - "6379:6379" volumes: - redis_dev_data:/data command: redis-server --appendonly yes diff --git a/docs/RESTORE_DRILL_RUNBOOK.md b/docs/RESTORE_DRILL_RUNBOOK.md new file mode 100644 index 00000000..6fdef792 --- /dev/null +++ b/docs/RESTORE_DRILL_RUNBOOK.md @@ -0,0 +1,140 @@ +# Postgres Restore Drill Runbook + +**Status: partially executed.** This documents a real, locally-performed +restore drill (schema/tooling fidelity — see "What this drill proves" below) +with genuine measured numbers. It does **not** replace the production-shaped +drill #501 ultimately asks for, which needs a real staging environment and a +real automated backup to restore from — **neither exists yet** (see +"What's still missing" below). Treat this as the tooling/procedure baseline +the production drill will build on, not as a substitute for it. + +## Why #431 blocks the full drill + +#501 depends on #431 ("automated Postgres backups with a restore we have +actually performed"), which is **still open**. `grep -rl "backup\|pg_dump\|restore" docs/ scripts/` +confirms there is no backup automation, no cron/CI job producing scheduled +dumps, and no `scripts/backup.sh`/`restore.sh` in this repo. A "production-shaped +backup from staging" (#501's first scope item) cannot be taken because there +is no defined backup artifact or schedule to draw one from yet. + +## What this drill proves + +Rather than skip #501 entirely while #431 is open, this drill answers the +parts of it that don't require production infrastructure or an existing +backup pipeline: **does `pg_dump`/`pg_restore` actually round-trip this +schema correctly, especially the partitioned `soroban_events` table**, and +**what does a dump/restore of a realistically-sized dataset actually cost in +wall-clock time**. Both are real, useful facts independent of where the +backup schedule eventually lives. + +### What was actually done + +1. A local PostgreSQL 15.15 instance was started (matching `docker/docker-compose.dev.yml`'s + pinned `postgres:15-alpine`, verified via `psql -c "select version()"`). +2. The real schema was applied from `database/schema.sql` (the same file + used for local/dev bootstrap). +3. **Found and fixed a real bug this drill surfaced**: `docker/docker-compose.dev.yml` + was checked into this repo with corrupted YAML — every double-quoted + string (`"5432:5432"`, the `healthcheck.test` array) had been mangled + into literal backslash-newline sequences at some point in its history + (introduced in `699fce4`, "chore(db): integrate sqlx-cli database + migration management" / `#93` — confirmed clean at the file's creation + in `ca79d07`, already corrupted by `699fce4`; `#195` later removed an + already-corrupted `version:` key without noticing the same corruption + elsewhere in the file). `python3 -c "import yaml; + yaml.safe_load(...)"` confirmed the committed file fails to parse at + all — `docker compose -f docker/docker-compose.dev.yml up` would have + failed outright. Fixed in this same change (see the diff on this PR) by + restoring proper YAML string quoting. +4. **`create_soroban_partition(bigint, bigint)` is documented in migration + `0017` as callable but is not actually defined in `database/schema.sql`** + — the convenience snapshot has drifted from the migration chain here. + Worked around locally by applying migration `0017`'s function definition + directly; flagging the drift as a separate, small gap for whoever owns + `database/schema.sql`'s upkeep (its own header comment says it "must + mirror the end state of that chain"). +5. Two real partitions were created via `create_soroban_partition` — + `soroban_events_p0_1999999` and `soroban_events_p2000000_3999999`. +6. 50,000 representative rows were inserted into `soroban_events`, landing + in both partitions (29,196 / 20,804 — confirming the partition routing + itself works, not just that the parent table accepts inserts). +7. **`pg_dump -Fc`** (custom format, the same one `pg_restore` expects) — + **8.0s wall-clock**, producing a 3.9 MB archive. +8. Simulated 500 further events written *after* the backup was taken, to + create a real, measurable "data loss window" for the restore to reveal. +9. **`pg_restore --no-owner --no-privileges`** into a brand-new, empty + database — **8s wall-clock**, zero errors. +10. Verified against the restored database, not assumed: + - Row count: exactly 50,000 (not 50,500) — confirms the restore + captured precisely what was in the backup, no more, no less. + - `soroban_events` came back as a **partitioned table**, `RANGE + (ledger_sequence)`, with both partitions (`soroban_events_p0_1999999`, + `soroban_events_p2000000_3999999`) present as real child tables, not + collapsed into a flat table. + - Per-partition row counts matched the pre-backup source exactly + (29,196 / 20,804) — partition routing survived the round-trip. + - All 5 indexes came back intact, including the partial index + (`idx_soroban_events_contract_topic0 ... WHERE topic_0 IS NOT NULL`) + and the composite primary key `(ledger_sequence, id)`. + +### Measured numbers (this drill's dataset: 50,000 rows, 3.9 MB compressed) + +| Metric | Value | +|---|---| +| Backup (`pg_dump -Fc`) duration | 8.0s | +| Restore (`pg_restore`) duration | 8s | +| Backup artifact size | 3.9 MB | +| Data loss window in this drill | 500 rows / ~83s of simulated writes | + +**These numbers do not extrapolate linearly to a production-sized database.** +`pg_dump`/`pg_restore` duration scales with data volume and index count, not +row count alone; a production `soroban_events` table with years of mainnet +history across many more partitions will take meaningfully longer for both +directions. Re-run this same procedure against a real backup once #431 +lands, and replace this table with those numbers before this runbook is +relied on for a real incident. + +## What's still missing (blocks calling #501 fully done) + +- **A real backup to restore** — #431 is open; there is no scheduled + production backup this drill could have used instead of a synthetic one. +- **A production-shaped dataset** — this drill's 50,000 synthetic rows + exercise the partitioning and index behavior correctly, but say nothing + about restore time at production scale (more partitions, more indexes, + larger individual rows via `topics`/`data` JSONB payloads). +- **RTO under real operational conditions** — this drill ran against a + local, otherwise-idle Postgres instance. A real RTO measurement needs to + account for provisioning a replacement database, network transfer of the + backup artifact, and bringing the indexer back up against the restored + database (reconciling its ingest cursor — see `docs/runbooks/incident-response.md`'s + `TridentIndexerLagCritical` alert for what "caught back up" means + operationally). + +## Reproducing this drill + +```bash +# 1. Start a local Postgres 15 instance (after the docker-compose.dev.yml fix in this PR): +docker compose -f docker/docker-compose.dev.yml up -d postgres + +# 2. Apply schema + the partition-creation function (until database/schema.sql +# is updated to include it — see "What was actually done" step 4 above): +psql -h localhost -U trident -d trident -f database/schema.sql +psql -h localhost -U trident -d trident -c "$(sed -n '/CREATE OR REPLACE FUNCTION create_soroban_partition/,/^\$\$;/p' database/migrations/0017_soroban_events_partitioning.sql)" + +# 3. Create partitions and seed representative data, then: +pg_dump -h localhost -U trident -d trident -Fc -f backup.dump + +# 4. Restore into a fresh database and verify: +createdb -h localhost -U trident trident_restored +pg_restore -h localhost -U trident -d trident_restored --no-owner --no-privileges backup.dump +psql -h localhost -U trident -d trident_restored -c "\d+ soroban_events" +``` + +## Next steps once #431 lands + +1. Point this same procedure at #431's real scheduled backup artifact + instead of a synthetic dump. +2. Re-run against a copy of staging's actual data volume. +3. Replace the "Measured numbers" table above with those real figures. +4. Cross-reference the resulting RPO/RTO into `docs/runbooks/testnet-cutover.md`'s + preconditions (see that runbook, added alongside this one). diff --git a/docs/runbooks/testnet-cutover.md b/docs/runbooks/testnet-cutover.md new file mode 100644 index 00000000..0137d861 --- /dev/null +++ b/docs/runbooks/testnet-cutover.md @@ -0,0 +1,184 @@ +# Testnet cutover runbook + +**Status: documented procedure, not yet walked through by someone who +didn't write it.** Issue #502's "done when" is a real dry-run against +staging with a second person driving from this document alone. That walkthrough +needs staging access and a second engineer, neither available in this pass. +What follows is the ordered procedure, grounded in what this repo's +deployment tooling, CI gates, and prior runbooks actually do — not generic +advice. + +## Preconditions + +All of the following must be true before cutover starts. None of these are +new checks invented for this runbook — they're the existing gates this repo +already has, gathered into one list. + +### CI / code + +- [ ] The commit being deployed passed CI's `docker`, `e2e`, and + `e2e-contract-events` jobs (see [`docs/CI.md`](../CI.md)) on `dev`. +- [ ] Coverage floors are met (`docs/CI.md`'s "Coverage: collection and + enforced floors" section) — a coverage regression here is a signal + the change wasn't tested as thoroughly as the rest of the codebase. +- [ ] `scripts/lint-migrations.sh` and `scripts/check-schema-drift.sh` are + green if the release includes a migration. **Known gap, found while + writing this runbook**: `check-schema-drift.sh`'s own documentation + states it compares tables/columns/indexes/constraints — it does + **not** compare functions. `create_soroban_partition` (added in + migration `0017`) is missing from `database/schema.sql` today and + this check does not catch it (see + [`RESTORE_DRILL_RUNBOOK.md`](../RESTORE_DRILL_RUNBOOK.md)'s + "What was actually done" step 4). Don't rely on this check alone to + certify a migration that adds or changes a function. + +### Which issues must be closed + +- [ ] [#431](https://github.com/Telocel-Labs/Trident/issues/431) — + automated backups with a real restore performed. **Currently open.** + Cutting over to testnet without this means the first real incident + that needs a restore will be the first time anyone has ever + performed one. See + [`RESTORE_DRILL_RUNBOOK.md`](../RESTORE_DRILL_RUNBOOK.md) for what's + been verified so far (schema/partition round-trip, real dump/restore + timing) and what's still missing (a real backup to restore, at + production-shaped scale). +- [ ] [#460](https://github.com/Telocel-Labs/Trident/issues/460) — rollback + rehearsed. **Closed**, but its own runbook + ([`ROLLBACK_RUNBOOK.md`](../ROLLBACK_RUNBOOK.md)) is explicitly + marked "template — not yet rehearsed" and documents a real, load-bearing + finding: **there are zero `.down.sql` files for any of the 25 + migrations** — schema rollback is not automated today. Read that + runbook's "Finding: migrations here are forward-only" section before + cutover, especially if the release being cut over includes a + migration. +- [ ] [#445](https://github.com/Telocel-Labs/Trident/issues/445) — + incident response process with a named on-call owner. **Closed** — + [`incident-response.md`](incident-response.md) exists with severity + levels, escalation path, and a communication channel. Its "On-call + owner — launch week" section is a `[FILL IN: ...]` placeholder as of + this writing — confirm it has real names/contacts filled in before + cutover, not just that the section exists. +- [ ] Row 1 of [`LAUNCH_CHECKLIST.md`](../LAUNCH_CHECKLIST.md) (alerts + verified firing) and row 9 (rollback rehearsed within the last 30 + days) are checked off with evidence, per that checklist's own no-go + criteria. + +## Ordered cutover steps + +Each step names an owner role (not a person — fill in the actual name in +the walkthrough) and a verification to run immediately after, not deferred +to the end. + +### 1. Freeze `dev` and cut the release branch — *release owner* + +```bash +git checkout dev && git pull +git checkout -b release/testnet-cutover- +``` + +**Verify:** CI is green on the release branch (same jobs as the preconditions +above, re-run on the exact commit being cut over — not assumed still-green +from when it merged to `dev`). + +### 2. Confirm environment configuration for testnet — *deploy owner* + +Per [`docs/ENVIRONMENT.md`](../ENVIRONMENT.md), `NETWORK` defaults to +`testnet` and needs no override for a testnet deployment; confirm the +target `.env` does not have a stale `NETWORK=mainnet` or +`NETWORK=futurenet` left over from a previous environment's config. + +**Verify:** +```bash +grep -E '^NETWORK=' .env # expect: NETWORK=testnet, or absent (defaults to testnet) +``` + +### 3. Take a pre-cutover database snapshot — *database owner* + +Even without #431's automated backups yet, take a manual one immediately +before cutover so there is at least one restore point: + +```bash +pg_dump -Fc -h -U trident -d trident -f pre-cutover-$(date -u +%Y%m%dT%H%M%SZ).dump +``` + +**Verify:** the dump file exists and is non-empty; spot-check its size is in +the expected range for the current database (compare against the previous +manual snapshot, if any — a dump an order of magnitude smaller than +expected usually means a connection/permission problem, not an empty +database). + +### 4. Run migrations — *database owner* + +Per [`docs/deployment.md`](../deployment.md#5-start-postgresql-and-run-database-migrations)'s +existing migration procedure. + +**Verify:** `sqlx migrate info` (or the project's equivalent) shows every +migration applied, none pending. Re-run `scripts/check-schema-drift.sh` +against the now-migrated database to confirm no drift was introduced by +this release — subject to the function-comparison gap noted in +Preconditions above. + +### 5. Deploy the indexer and API — *deploy owner* + +Per [`docs/deployment.md`](../deployment.md#6-start-all-services)'s existing +deploy steps. + +**Verify:** immediately after, per +[`docs/deployment.md`](../deployment.md#7-verify-health): +```bash +curl -sf https:///v1/health +``` +Confirm the indexer's ingest cursor is advancing (not stuck at whatever it +was pre-cutover) — a health check passing does not by itself confirm the +indexer resumed consuming testnet ledgers. + +### 6. Watch for the first full ingest cycle — *on-call owner* + +Per [`incident-response.md`](incident-response.md)'s alert catalog, +specifically `TridentIndexerHeartbeatStale` and `TridentIndexerLagCritical` +— these are exactly the signals that would fire if cutover left the indexer +unable to reach testnet RPC or resume its cursor correctly. + +**Verify:** no SEV-1/SEV-2 alert fires within one full expected ingest +cycle after cutover. Manually query recent `soroban_events` rows and +confirm `ledger_timestamp` values are current, not frozen at the pre-cutover +watermark. + +### 7. Announce cutover complete — *release owner* + +Per [`incident-response.md`](incident-response.md#user-communication-channel)'s +existing communication channel — post confirmation there, not ad hoc. + +## Rollback trigger and procedure + +**Trigger**: any SEV-1 per [`incident-response.md`](incident-response.md#sev-1-service-down-or-data-incorrect)'s +definition within the first ingest cycle after cutover, or a health check +that never turns green within 15 minutes of step 5. + +**Who calls it**: per [`LAUNCH_CHECKLIST.md`](../LAUNCH_CHECKLIST.md#rollback-decision-procedure) — +the on-call engineer, or the release owner if reachable within 5 minutes. +Do not wait for a quorum. + +**Procedure**: follow [`ROLLBACK_RUNBOOK.md`](../ROLLBACK_RUNBOOK.md) in +full. If this release included a migration, read that runbook's +"Rollback across a migration boundary" section first — **a +backward-incompatible migration currently has no automated reverse path**, +which may mean the correct rollback for this specific release is +"roll back the application only, leave the schema" (if the migration was +additive) rather than a full schema rollback. Decide which case applies +*before* cutover, not while already mid-incident. + +## What this runbook still needs (not done here) + +- [ ] A real walkthrough by someone who did not write this document, + driving from this file alone against a real staging environment — + this is #502's literal "done when" criterion and the single most + important gap. +- [ ] #431 landed, so precondition checkbox 1 above can actually be + checked rather than flagged as open. +- [ ] `incident-response.md`'s on-call section filled in with real names. +- [ ] The exact commands in each step re-verified against whatever the + target testnet environment's actual hostnames/credentials are (the + commands above use placeholders consistent with `deployment.md`'s + existing style).