NEO-61: apply migrations on deploy and gate traffic on health - #100
Merged
lucasvienna merged 3 commits intoAug 25, 2026
Merged
Conversation
No service set preDeployCommand, so pending migrations reached production only when someone ran them by hand, and a deploy carrying one came up against the old schema. uppity-server now runs ./entrypoint.sh migrate between build and deploy. A non-zero exit aborts the deploy and leaves the running version serving, which is what both entrypoint.sh and migrate.ts already exit for. Only the web tier runs it. drizzle-orm's migrator takes no lock -- its PgDialect.migrate reads the newest journal row, then applies every later file in one transaction -- so a second copy racing it would apply the same DDL twice. The workers deploy alongside and run against the old schema for the length of the migration. That window is not free: worker-monitor calls initializeMaintenanceJobs() before its poll loop and outside any try, so a migration touching maintenance_job kills it, and restartPolicyMaxRetries is 3. It is still strictly better than the status quo, where the schema was never migrated at all and new worker code met the old schema indefinitely. Tracked in NEO-63. Also corrects the comment above the migrator's connection, which claimed an advisory lock that drizzle-orm does not take. That claim is what the server-only decision turns on. Refs NEO-61 Claude-Session: https://claude.ai/code/session_01AqHppaB4Px72dTuAvpAEBD
Railway does not read the Dockerfile's HEALTHCHECK -- that instruction serves docker run and Compose. Railway uses deploy.healthcheckPath, and uppity-server set none, so a deploy counted as healthy the moment the process started. Railway then switched traffic over and tore down the previous version, whether or not the new container could serve. /api/health runs SELECT 1 and answers 503 while Postgres is unreachable, so it fails for the reason worth catching rather than answering 200 from a process that has started and nothing more. This matters more now that deploys carry migrations: the pre-deploy step proves the schema applied, and the healthcheck proves the container that meets that schema can actually serve before it takes traffic. The workers keep no healthcheck. They serve no HTTP, and the probe would fail for them by design. Refs NEO-61 Claude-Session: https://claude.ai/code/session_01AqHppaB4Px72dTuAvpAEBD
🦋 Changeset detectedLatest commit: 867b3fb The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
The healthcheck comment claimed /api/health answering 503 was "the state
worth catching here." Railway does not catch 503. It queries the endpoint
until it gets a 200 and fails the deploy at timeout, so every non-200 and
the no-answer case behave alike. The sentence described a specificity the
mechanism does not have, and narrated a route in another directory that
it could not keep up with. What remains is the part that earns its place:
Railway ignores the Dockerfile's HEALTHCHECK, which is a trap this repo
sets for itself by declaring one on the same path.
The max: 1 comment stated its own stakes and they were nothing -- "a pool
would add nothing here but idle sockets." A one-shot script opening one
connection surprises no one, and "so max: 1" above { max: 1 } is the
config reading itself aloud.
The lock sentence stays. drizzle-orm 0.45.2 has no advisory lock anywhere
and a transaction is not mutual exclusion, so two migrators both read the
last applied row before either commits. That is a foreign dependency
behaving surprisingly on a live path, and the railway.ts comment cites it.
Refs NEO-61
Claude-Session: https://claude.ai/code/session_01AqHppaB4Px72dTuAvpAEBD
lucasvienna
deleted the
lucas/neo-61-migrations-are-never-applied-on-railway
branch
August 25, 2026 08:22
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
No Railway service set
preDeployCommand, so pending migrations reached production only when someone remembered to run them by hand, and a deploy carrying one came up against the old schema.uppity-servernow runs./entrypoint.sh migratebetween build and deploy. A non-zero exit aborts the deploy and leaves the running version serving, which is whatentrypoint.shandmigrate.tsalready exit for.Deployment
84c30a66shows it working:The journal was already current, so no new DDL ran. The path itself is proven: the pre-deploy container connects, runs the migrator, exits 0, and only then does the app container start.
Why only the web tier
drizzle-orm'sPgDialect.migratereads the newest journal row, then applies every later file in one transaction. It takes no lock, so a second copy racing it would apply the same DDL twice. Running the migration on all three services would need an advisory lock added first.The comment above the migrator's connection claimed the library takes an advisory lock. It does not, and that claim is what this decision turns on, so it is corrected here.
The cost is that the workers deploy alongside and run against the old schema for the length of the migration. That window is not free —
worker-monitorcallsinitializeMaintenanceJobs()before its poll loop and outside anytry, so a migration touchingmaintenance_jobkills it, andrestartPolicyMaxRetriesis 3. Filed as NEO-63 rather than fixed here. It is still strictly better than the previous state, where the schema was never migrated at all and new worker code met the old schema indefinitely.Health check
Found while applying. Railway does not read the Dockerfile's
HEALTHCHECK— that instruction servesdocker runand Compose. Railway usesdeploy.healthcheckPath, anduppity-serverset none, so a deploy counted as healthy the moment the process started. Railway then switched traffic over and tore down the previous version, whether or not the new container could serve./api/healthrunsSELECT 1and answers 503 while Postgres is unreachable, so it fails for the reason worth catching rather than answering 200 from a process that has started and nothing more. This matters more now that deploys carry migrations. The workers keep no health check; they serve no HTTP and the probe would fail for them by design.Related Issues
Closes NEO-61. Files NEO-63 as follow-up. Depends on nothing; #99 is already merged.
Type of Change
Checklist
aubr changeset, oraubr changeset --emptyif no release is needed)aubr test:unit run) — 308/308aubr lint:ci)aubr check)On tests: no new ones. The change is three declarative lines of deploy config; the behaviour they invoke is already covered.
scripts/entrypoint.spec.tshas 24 tests over themigratetarget, including its permission set and its network allowlist. A test assertingrailway.tscontains the literal it contains would restate the config, not verify it. The real verification is the deploy log above.On docs:
docs/runtime-permissions.mddescribes what each process may access, not when each runs, so nothing there goes stale.Additional Notes
While applying this I found that the workers were never actually running. #99 fixed their
startCommandin the repo and merged, but nobody ranrailway config apply, so it never reached Railway.worker-monitorwas loggingListening on http://0.0.0.0:8080/— running the web server, not the scheduler. Monitor checks and notification delivery had not been running on the hosted deployment.That is fixed now, by the same apply. Checks against Uppity, DSC, MBOS, wforacle and Apple are landing, and the maintenance jobs ran for the first time, including a cleanup that deleted 3421 old check rows. No code change in this PR does that — it was #99's fix, finally applied.
The lesson generalises: merged IaC is not applied IaC. Worth checking
railway config planreads clean after any PR touching.railway/railway.tsmerges.One wart for whoever applies next:
~ Update uppity-server networkingreappears in every plan and never converges.privateNetworkEndpoint: "uppity-server"matches the name Railway derives by default, so Railway appears not to record it and the diff persists. Harmless and graded safe, but it means the plan never reads clean — which is exactly the signal that would have caught #99 sitting unapplied.https://claude.ai/code/session_01AqHppaB4Px72dTuAvpAEBD