Duplicate migration version 49 can stop the relay from starting

[0w3n-d]

What's wrong

Two migrations share version 49: V49__merged_block_split_fields.sql and V49__validator_prefs_api_key.sql. refinery_schema_history has a primary key on version, so only one of them can ever be recorded, and refinery matches applied migrations against the filesystem by version alone. Which file wins depends on directory read order, which is baked into the binary at compile time and is unsorted.

The consequence is that a rebuild can stop the relay from starting. verify_migrations compares the recorded name and checksum for version 49 against whichever V49 the binary lists first. If they differ it returns DivergentVersion, abort_divergent defaults to true and is never overridden, and init_forever retries every 5 seconds forever — the relay never finishes init.

This already happened on mainnet FR on 2026-09-01, from 17:48:04 to 17:50:40:

ERROR helix_database::postgres::postgres_db_service: failed to run migrations! retrying..
  err=applied migration V49__merged_block_split_fields is different than filesystem one V49__validator_prefs_api_key

It was cleared by hand by deleting rows 49 and 50 from refinery_schema_history and restarting. That run then failed once more on error applying migration V49__merged_block_split_fields, db error — the plain ADD COLUMN against columns that already existed — and only succeeded because the retry skipped version 49 by version match before applying V50. The two history rows are 6.03 s apart, which is that retry sleep.

The landmine is still armed, with the polarity reversed: version 49 now records validator_prefs_api_key, which happens to match the current binary's order.

Repro

cargo test -p helix-database --lib migration_versions_are_unique fails on develop:

migration versions must be unique, found [(49, ["merged_block_split_fields", "validator_prefs_api_key"])]

Two runtime paths reproduce the failure modes:

1. Empty history — both V49 bodies execute, the first records at 49, the second's INSERT ... VALUES (49, ...) violates the primary key, the run errors, and the retry skips it. Converges, but logs an error and delays startup by the retry interval.
2. Version 49 recorded under one name while the binary lists the other first — DivergentVersion on every attempt, forever.

Suspected cause

refinery 0.8.16:

• traits/mod.rs:23 — divergence check finds the filesystem migration by version only, then compares version, name and checksum.
• traits/mod.rs:68 — a filesystem migration counts as applied if any applied row shares its version.
• util.rs:80 — discovery is WalkDir with no sort, and migrations.sort() is stable on version, so relative order of same-version files is whatever the build machine's readdir gave.
• runner.rs:237 — abort_divergent and abort_missing both default to true.

V49__merged_block_split_fields.sql also lacks IF NOT EXISTS, so it cannot be safely re-run.

Affected surface

crates/database/src/postgres/migrations/, postgres_db_init.rs, and init_forever at postgres_db_service.rs:246.

Steps (each becomes one PR)

• Step 1: assert unique migration versions in a test, renumber the merged-block migration above the current maximum, and guard its ADD COLUMNs with IF NOT EXISTS (tests: migration_versions_are_unique) (PR: Fix duplicate migration version 49, add merged_blocks.region_id (#572 step 1) #573)

Open questions

Any database whose version 49 row names merged_block_split_fields still diverges after the renumber and needs rows 49 and 50 deleted once — both, because abort_missing rejects a gap. Mainnet FR is not in that state; devnet and staging databases should be checked. After the renumber that recovery is clean rather than lucky, since 49, 50 and 51 all become guarded no-ops.