epic: deploy pipeline confidence — from reactive fixes to stable platform

[manavgup]

Context

The last 20 PRs (#550–#583) were almost entirely deploy pipeline stability fixes, discovered reactively after breakage. This epic tracked the structural changes needed to stop the pattern of "discover invariant, ship fix."

Status: COMPLETE (16 of 18 issues resolved)

All P0 deploy blockers resolved. Pipeline is functional and deploying.

Issues (final status)

P0 — Deploy blockers ✅ (5/5)

• 🚨 Deploy pipeline blocked — Docker workflow failed on main #578 — Docker workflow failing on main → Closed as stale (auto-resolved)
• ci(release): semantic-release push to main blocked by branch ruleset (GH013) #540 — semantic-release push blocked → Fixed (switched to release-please, PR chore(main): release 0.1.0 #608)
• fix: pin deploys to commit-specific image digest #584 — Pin deploys to commit-specific image digest → Merged (PR fix: deploy workflow hardening (#584, #590, #591, #595, #598) #605)
• fix: move migrations out of web/worker startup into release command #586 — Move migrations to release command + advisory lock → Merged (PR fix: move migrations to Fly release command with advisory lock (#586) #606)
• fix: set production min_machines_running = 1 to avoid cold-start timeouts #594 — Set production min_machines_running = 1 → Merged (PR fix: Fly config hardening (#594, #597) #601)

P1 — Pipeline integrity ✅ (4/5)

• fix: add post-rollback health verification in deploy workflow #595 — Add post-rollback health verification → Merged (PR fix: deploy workflow hardening (#584, #590, #591, #595, #598) #605)
• fix: pin Docker base image and Docling image to specific digests #596 — Pin Docker base image to 3.11.12 → Merged (PR fix: Docker + CI hardening (#596, #599, #600) #602)
• fix: resolve WEB_CONCURRENCY mismatch between fly.toml and gunicorn.conf.py #597 — Fix WEB_CONCURRENCY mismatch → Merged (PR fix: Fly config hardening (#594, #597) #601)
• test: add Alembic downgrade CI test before trusting schema-aware rollback #599 — Add Alembic downgrade CI test → Merged (PR fix: Docker + CI hardening (#596, #599, #600) #602)
• fix: make rollback schema-aware (not image-only) #589 — Make rollback schema-aware → Deferred (depends on test: add Alembic downgrade CI test before trusting schema-aware rollback #599 being validated in prod first)

P2 — Environment isolation ✅ (2/2)

• fix: isolate staging Docling sidecar from production #585 — Isolate staging Docling sidecar → Merged (PR fix: isolate staging Docling sidecar from production (#585) #603)
• fix: use separate verification tokens for staging and production #591 — Use separate verification tokens → Merged (PR fix: deploy workflow hardening (#584, #590, #591, #595, #598) #605)

P3 — Runtime observability ✅ (3/4)

• fix: add real Redis/worker health checks to /health/deep #587 — Add Redis/worker health checks to /health/deep → Merged (PR fix: add Redis connectivity and queue depth to /health/deep (#587) #604)
• fix: add deploy success/failure notifications #598 — Add deploy failure notifications → Merged (PR fix: deploy workflow hardening (#584, #590, #591, #595, #598) #605)
• fix: add external uptime monitoring (outside Fly.io) #600 — Add external uptime monitoring → Merged (PR fix: Docker + CI hardening (#596, #599, #600) #602)
• fix: add real background job E2E verification for production deploys #588 — Real E2E job verification for prod → Deferred (needs staging validation first)

P4 — Operational hygiene ✅ (1/2)

• fix: replace brittle flyctl CLI scraping with JSON output #590 — Replace brittle flyctl CLI scraping → Merged (PR fix: deploy workflow hardening (#584, #590, #591, #595, #598) #605)
• infra: evaluate multi-region or HA deployment for Fly.io #592 — Evaluate multi-region HA → Evaluation only (no code change needed)

PRs (all merged)

PR	Title	Issues
#583	Migration race + Redis TLS + ARQ worker	#582
#601	Fly config hardening	#594, #597
#602	Docker + CI hardening	#596, #599, #600
#603	Staging Docling isolation	#585
#604	Redis/queue health checks	#587
#605	Deploy workflow hardening	#584, #590, #591, #595, #598
#606	Migration release command	#586

Evidence

Full evidence page with test plan execution, code review results, and investigation details:
docs/evidence/epic-593/evidence.html