fix(database): unblock Postgres boot after edited migrations (WI-919) - #170
Merged
Conversation
Every Postgres install stamped before 2026-05-17 refused to boot on main. Live was left in a 10-minute crash-rollback loop: the update timer pulled main, the container was recreated, boot failed, and update.sh rolled back. Two commits months apart combined into this. ac1e4f1 (17 May) switched TIMESTAMP to TIMESTAMPTZ inside ten migrations that had already shipped and been applied. That was silent, because those bodies are CREATE TABLE IF NOT EXISTS and no-op against an existing table, so only fresh databases ever got the timezone-aware type. 54cff11 (4 Aug, unreleased) then added checksum validation and made a mismatch fatal, turning the latent drift into a hard boot failure. Reverting the ten bodies is not a fix: it would restore databases stamped between 14 and 17 May while breaking every database stamped after 17 May, which would then compute the old checksum and mismatch in the other direction. Instead, record the pre-edit checksum on each affected migration as Superseded. A stored checksum listed there is accepted and re-stamped to the current value, so the entry matters only once per database, and an unrecognized mismatch stays fatal. Superseded absorbs history; it does not make edits safe. The schema drift is converged for real by 20260804_pg_timestamptz_convergence, which alters the 27 affected columns across 11 tables to TIMESTAMPTZ. It is Postgres-only, self-guarding on data_type so it no-ops on already-converged databases and tolerates an absent table, and uses an explicit USING ... AT TIME ZONE 'UTC' so stored values are read as UTC rather than reinterpreted through the session TimeZone. Verified against a throwaway Postgres by reverting the 27 columns and stamping the ten pre-edit checksums: boot completes, all 27 columns convert, and no stale checksums remain. A deliberately unknown checksum still aborts startup with exit 1.
|
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every Postgres install stamped before 2026-05-17 refused to boot on main. Live was left in a 10-minute crash-rollback loop: the update timer pulled main, the container was recreated, boot failed, and update.sh rolled back.
Two commits months apart combined into this. ac1e4f1 (17 May) switched TIMESTAMP to TIMESTAMPTZ inside ten migrations that had already shipped and been applied. That was silent, because those bodies are CREATE TABLE IF NOT EXISTS and no-op against an existing table, so only fresh databases ever got the timezone-aware type. 54cff11 (4 Aug, unreleased) then added checksum validation and made a mismatch fatal, turning the latent drift into a hard boot failure.
Reverting the ten bodies is not a fix: it would restore databases stamped between 14 and 17 May while breaking every database stamped after 17 May, which would then compute the old checksum and mismatch in the other direction. Instead, record the pre-edit checksum on each affected migration as Superseded. A stored checksum listed there is accepted and re-stamped to the current value, so the entry matters only once per database, and an unrecognized mismatch stays fatal.
Superseded absorbs history; it does not make edits safe. The schema drift is converged for real by 20260804_pg_timestamptz_convergence, which alters the 27 affected columns across 11 tables to TIMESTAMPTZ. It is Postgres-only, self-guarding on data_type so it no-ops on already-converged databases and tolerates an absent table, and uses an explicit USING ... AT TIME ZONE 'UTC' so stored values are read as UTC rather than reinterpreted through the session TimeZone.
Verified against a throwaway Postgres by reverting the 27 columns and stamping the ten pre-edit checksums: boot completes, all 27 columns convert, and no stale checksums remain. A deliberately unknown checksum still aborts startup with exit 1.