fix(deploy): run the bundled Keycloak on Postgres instead of dev-mode H2 - #1204
fix(deploy): run the bundled Keycloak on Postgres instead of dev-mode H2#1204njbrake wants to merge 2 commits into
Conversation
_Note: this PR description was drafted by Claude via back-and-forth with @njbrake. The reasoning and decisions are his; the prose is Claude's._ Runs the bundled Keycloak in production mode against Postgres instead of `start-dev` with file-backed H2. `start-dev` implies H2, which Keycloak documents as development only. Combined with an orchestrator that mounts no volume for the service, the realm becomes ephemeral: every redeploy re-imports the realm JSON and discards everything created since. Observed on a live deployment: - users added in the admin console vanish - master-realm settings such as brute-force protection reset to off - there is no login or admin event history to inspect - `KC_BOOTSTRAP_ADMIN_*` becomes load-bearing on every boot, when it exists for the first boot only Two-stage build so the image can run `start --optimized`: the database vendor is baked in by `kc.sh build`, while hostname, DB URL and credentials stay runtime options. Keycloak gets its own database rather than a schema in the app's. It owns roughly 95 tables managed by Liquibase, which has no business sharing a namespace with the app's Drizzle migrations. `postgres-init` creates it alongside `powersync_storage`, so fresh stacks need nothing extra. One behaviour change worth calling out: `--import-realm` only applies to an empty database, so realm JSON edits no longer take effect on an existing stack. That is the point, since console changes now survive redeploys, but it moves realm configuration from the file to the console. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1e15fbf to
99ff17b
Compare
|
Preview environment deployed 🚀
Stack: Auto-destroys on PR close/merge. Login via the bundled Keycloak realm — |
ital0
left a comment
ital0
left a comment
There was a problem hiding this comment.
I think the goal is right: move Keycloak state out of the container and into Postgres so users, settings, and audit history survive a redeploy. The optimized image itself follows Keycloak's documented build pattern.
The integration is not complete yet, so I would not merge this version:
- Every checked-in deployment still bypasses or overrides the new production command with
start-dev. - None supplies the Postgres URL and credentials. The current HTTP topology also needs
KC_HTTP_ENABLED=truefor edge TLS termination. - The new database is created only on first Postgres initialization, so existing volumes never get it. Kubernetes also mounts a separate stale init script.
- No dedicated Keycloak database role or secret is provisioned.
- The bootstrap admin password is also used by Pulumi as a permanent admin credential. Once the database persists, rotating that value will break preview client provisioning.
- CI builds the image but does not start this path or test persistence across a restart.
I reproduced the first two startup failures with the image published by this PR. With the deployment override it rebuilt and started in dev mode on H2. With the new default command it exited first for missing HTTPS/HTTP configuration, then for a connection attempt to localhost:5432.
Once the runtime wiring, upgrade path, credential lifecycle, and smoke coverage are included, this should deliver the persistence the PR describes.
|
|
||
| ENTRYPOINT ["/opt/keycloak/bin/kc.sh"] | ||
| CMD ["start-dev", "--import-realm"] | ||
| CMD ["start", "--optimized", "--import-realm"] |
There was a problem hiding this comment.
Blocker: every checked-in deployment bypasses this command. Compose and Helm still set start-dev --import-realm, and both Pulumi task definitions override the image CMD with the same command (deploy/docker-compose.yml:108, deploy/k8s/templates/keycloak.yaml:34, deploy/pulumi/src/services.ts:315, deploy/pulumi/src/shared.ts:543). I ran the published PR image with that override; it rebuilt itself for the dev profile and started on H2. Can we update all consumers in this PR so the production command actually runs?
|
|
||
| ENTRYPOINT ["/opt/keycloak/bin/kc.sh"] | ||
| CMD ["start-dev", "--import-realm"] | ||
| CMD ["start", "--optimized", "--import-realm"] |
There was a problem hiding this comment.
Blocker: once this command is used, the current network setup still cannot start it. All stacks send plain HTTP to port 8080 behind ALB or Ingress, but none sets KC_HTTP_ENABLED=true or provides Keycloak TLS key material. The published image exits with Key material not provided to setup HTTPS. Please wire edge TLS termination explicitly.
| FROM quay.io/keycloak/keycloak:26.7 AS builder | ||
|
|
||
| # Build-time option: selects the JDBC driver and dialect compiled into the image. | ||
| # KC_DB_URL / KC_DB_USERNAME / KC_DB_PASSWORD remain runtime options. |
There was a problem hiding this comment.
Blocker: none of the Compose, Helm, or Pulumi Keycloak definitions sets these values. The published image therefore tries localhost:5432/keycloak even though Postgres runs as another service, and it exits. Please add the URL and credentials for every deployment target that uses this image.
| -- than a schema in this one: Keycloak owns ~95 tables and manages them with | ||
| -- Liquibase, which has no business sharing a namespace with the app's Drizzle | ||
| -- migrations. Harmless when unused -- an empty database costs nothing. | ||
| CREATE DATABASE keycloak OWNER postgres; |
There was a problem hiding this comment.
Blocker: /docker-entrypoint-initdb.d runs only when PGDATA is empty, so existing Compose volumes, Helm PVCs, and Fargate EFS volumes will never create this database. Helm also mounts a separate init ConfigMap that still stops after powersync_storage, so fresh Kubernetes installs miss it too. We need an upgrade-safe provisioning step and the Kubernetes initializer updated in this PR.
| -- than a schema in this one: Keycloak owns ~95 tables and manages them with | ||
| -- Liquibase, which has no business sharing a namespace with the app's Drizzle | ||
| -- migrations. Harmless when unused -- an empty database costs nothing. | ||
| CREATE DATABASE keycloak OWNER postgres; |
There was a problem hiding this comment.
Security blocker: which login will Keycloak use? This creates the database under postgres, but no Keycloak role or password exists in the deployment config. Reusing the current Postgres secret would give the IdP superuser access to every database. Please provision a dedicated Keycloak role and secret.
| # `--import-realm` reads this on the first boot against an EMPTY database. Once | ||
| # the realm exists the import is skipped, which is the whole point of moving off | ||
| # H2: changes made in the admin console now survive a redeploy. The flip side is | ||
| # that edits to this file (or to KC_SEED_*) no longer take effect on a database | ||
| # that already holds the realm — change those in the console, or drop the schema. |
There was a problem hiding this comment.
Non-blocking, but this comment is not accurate. Startup import runs on every boot and skips only a realm that already exists; the whole database does not need to be empty. KC_SEED_* is also not used in this repo, and dropping the schema would erase the users and event history this change is trying to preserve. Could we document the exact behavior and remove the destructive advice?
| # whole realm ephemeral: every redeploy re-imports the realm JSON and discards | ||
| # everything created since — users added in the admin console, master-realm | ||
| # settings such as brute-force protection, and the login/admin event history. It | ||
| # also leaves `KC_BOOTSTRAP_ADMIN_*` load-bearing on every boot, when it is meant |
There was a problem hiding this comment.
Blocker for shared previews: this value is also used as the durable password by per-pr-stack.ts (deploy/pulumi/src/shared.ts:604 and deploy/pulumi/src/per-pr-stack.ts:217-218). Once the master realm persists, changing args.keycloakAdminPassword updates Pulumi but KC_BOOTSTRAP_ADMIN_PASSWORD no longer changes the real account. New preview client provisioning will then fail authentication. Can we separate bootstrap credentials from the ongoing admin credential lifecycle?
- Restore start-dev/H2 as the default command so local and ephemeral stacks work without extra config; prod deployments opt into start --optimized on Postgres. - Add a dedicated keycloak_role (least privilege) owning the keycloak database instead of the postgres superuser, with a dev-default password overridable via KEYCLOAK_DB_PASSWORD.
ital0
left a comment
ital0
left a comment
There was a problem hiding this comment.
Thanks for the follow-up.
The remaining questions are limited to self-hosting. KEYCLOAK_DB_PASSWORD still falls back silently to keycloak, existing Postgres volumes will not run the new initialization, and the Helm chart uses a separate initializer that does not create the new role or database. If some of those paths are intentionally unsupported, documenting the scope and upgrade steps would be enough.
| # Dev default so a fresh local stack initializes without extra config; a | ||
| # deployment that runs Keycloak on Postgres overrides this and sets the same | ||
| # value as KC_DB_PASSWORD. | ||
| : "${KEYCLOAK_DB_PASSWORD:=keycloak}" |
There was a problem hiding this comment.
Since this script is for self-hosting, would it make sense to add KEYCLOAK_DB_PASSWORD to .env.example and pass it from Compose instead of silently falling back to keycloak?
The default is convenient for a disposable local stack, but someone enabling Postgres in a real self-hosted installation could keep it by accident.
| -- a namespace with the app's Drizzle migrations. Harmless when unused -- an | ||
| -- empty database costs nothing. Opt-in prod deployments set KC_DB_USERNAME to | ||
| -- this role and KC_DB_PASSWORD to the value above. | ||
| CREATE ROLE keycloak_role WITH LOGIN PASSWORD :'keycloak_pass'; |
There was a problem hiding this comment.
How should existing self-hosted installations pick up this change?
/docker-entrypoint-initdb.d only runs when PGDATA is empty, so upgrading an existing volume will not create the new role or database. If Postgres-backed Keycloak is only supported for fresh self-hosted stacks, a short upgrade note would be enough. Otherwise, we will need a provisioning step that also runs for existing volumes.
| -- empty database costs nothing. Opt-in prod deployments set KC_DB_USERNAME to | ||
| -- this role and KC_DB_PASSWORD to the value above. | ||
| CREATE ROLE keycloak_role WITH LOGIN PASSWORD :'keycloak_pass'; | ||
| CREATE DATABASE keycloak OWNER keycloak_role; |
There was a problem hiding this comment.
The Helm chart mounts a separate postgres-init ConfigMap over /docker-entrypoint-initdb.d, and that version still stops after creating powersync_storage.
Could we update that initializer as well, or document that Helm does not support this opt-in yet? As written, a fresh Helm installation will not get the role or database added here.
2 participants
Note: this PR description was drafted by Claude via back-and-forth with @njbrake. The reasoning and decisions are his; the prose is Claude's.
Runs the bundled Keycloak in production mode against Postgres instead of
start-devwith file-backed H2.start-devimplies H2, which Keycloak documents as development only. Combined with an orchestrator that mounts no volume for the service, the realm becomes ephemeral: every redeploy re-imports the realm JSON and discards everything created since. Observed on a live deployment:KC_BOOTSTRAP_ADMIN_*becomes load-bearing on every boot, when it exists for the first boot onlyTwo-stage build so the image can run
start --optimized: the database vendor is baked in bykc.sh build, while hostname, DB URL and credentials stay runtime options.Keycloak gets its own database rather than a schema in the app's. It owns roughly 95 tables managed by Liquibase, which has no business sharing a namespace with the app's Drizzle migrations.
postgres-initcreates it alongsidepowersync_storage, so fresh stacks need nothing extra.One behaviour change worth calling out:
--import-realmonly applies to an empty database, so realm JSON edits no longer take effect on an existing stack. That is the point, since console changes now survive redeploys, but it moves realm configuration from the file to the console.