docs: restore drill runbook and testnet cutover runbook

[onuibeblessing2019-hash]

Closes #501
Closes #502

Both issues ask for real operational work (a production restore drill, a runbook walked through by someone else) that needs staging access and team availability this pass doesn't have. Followed the precedent already established in this repo by docs/ROLLBACK_RUNBOOK.md/docs/LAUNCH_CHECKLIST.md (both explicitly marked "template — not yet rehearsed/executed"): do the real, honest subset of the work that's actually possible without that access, document it with real numbers/findings rather than fabricated ones, and disclose exactly what's still missing.

#501 — restore drill (docs/RESTORE_DRILL_RUNBOOK.md)

#431 (automated backups) is still open — there is no backup schedule or artifact to restore from yet, so the literal "take a production-shaped backup from staging" scope item can't be done. Instead, performed a real local drill answering the parts that don't depend on #431: does pg_dump/pg_restore actually round-trip this schema correctly (especially the partitioned soroban_events table), and what does it cost in wall-clock time.

What was actually run, not simulated:

• Local PostgreSQL 15.15 (matching the repo's pinned version), real schema from database/schema.sql.
• Two real partitions created via create_soroban_partition, 50,000 rows seeded and confirmed distributed across both (29,196 / 20,804).
• pg_dump -Fc: 8.0s, 3.9 MB. pg_restore into a fresh DB: 8s, zero errors.
• Verified against the restored DB, not assumed: exact row count (50,000, correctly excluding 500 rows written after the backup — a real, measured RPO gap), partitioned structure preserved, per-partition counts matched, all 5 indexes intact including the partial index.

Two real, unrelated bugs found and fixed/flagged along the way:

1. Fixed: docker/docker-compose.dev.yml was checked into this repo with corrupted YAML — every double-quoted string had been mangled into literal backslash-newline sequences (confirmed via python3 -c "import yaml; yaml.safe_load(...)" failing to parse the committed file at all). Traced to 699fce4 (chore(db): integrate sqlx-cli database migration management #93) — confirmed clean at the file's creation (ca79d07), already broken by 699fce4. docker compose -f docker/docker-compose.dev.yml config now resolves cleanly after the fix.
2. Flagged, not fixed (out of scope for this PR): create_soroban_partition is documented as callable in migration 0017 but is missing from database/schema.sql, which its own header says "must mirror the end state of that chain." scripts/check-schema-drift.sh doesn't catch this — its own docs say it compares tables/columns/indexes/constraints, not functions. Worked around locally by applying 0017's function definition directly; noted as a real gap in the runbook for whoever owns schema.sql's upkeep.

Disclosed as still missing: a real backup to restore (blocked on #431), a production-shaped dataset, and RTO under real operational conditions (network transfer, indexer cursor reconciliation). See the runbook's "What's still missing" section.

#502 — testnet cutover runbook (docs/runbooks/testnet-cutover.md)

Built from what this repo's tooling and existing runbooks actually do, not generic advice:

• Preconditions cross-reference docs/CI.md's real gates (coverage floors, schema-drift/migration-lint scripts — including the function-comparison gap found above), and the actual state of launch: automated Postgres backups with a restore we have actually performed #431/launch: rehearse the rollback before we need it #460/launch: incident response process with a named on-call owner #445 (launch: automated Postgres backups with a restore we have actually performed #431 open, launch: rehearse the rollback before we need it #460/launch: incident response process with a named on-call owner #445 closed but their own runbooks disclose real gaps worth reading before cutover — e.g. ROLLBACK_RUNBOOK.md's finding that zero of the 25 migrations have a .down.sql, so schema rollback isn't automated today).
• Ordered steps link directly into docs/deployment.md's existing deploy/migration/health-check procedure (verified the anchor links resolve to the correct headings) rather than restating it, with a verification after each step.
• Rollback trigger cross-references incident-response.md's real SEV-1 definition and LAUNCH_CHECKLIST.md's real rollback-decision procedure, and calls out that a migration-boundary rollback decision (app-only vs. full schema) needs to be made before cutover given the .down.sql gap above.

Disclosed as still missing: the actual walkthrough by someone who didn't write it (#502's literal "done when"), and the on-call section in incident-response.md still being a [FILL IN: ...] placeholder.

Test plan

• python3 -c "import yaml; yaml.safe_load(open('docker/docker-compose.dev.yml'))" — passes after the fix (failed before).
• docker compose -f docker/docker-compose.dev.yml config — resolves cleanly, confirming semantic validity beyond just YAML syntax (daemon not required for config).
• Restore drill actually executed end-to-end locally with real timing and verification queries (see above and the runbook itself for full commands/output).
• Every internal markdown link in the new runbook (to deployment.md, incident-response.md, LAUNCH_CHECKLIST.md, ROLLBACK_RUNBOOK.md, and RESTORE_DRILL_RUNBOOK.md) checked against the actual heading text in each target file, including GitHub's anchor-slug conversion for the em-dash in "SEV-1 — service down or data incorrect".

[drips-wave]

@onuibeblessing2019-hash Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits