Skip to content

fix(helm): make fresh installs work — valid db tag, working init SQL, no core-schema pre-creation - #1

Draft
dnplkndll wants to merge 3 commits into
mainfrom
fix/helm-fresh-install
Draft

fix(helm): make fresh installs work — valid db tag, working init SQL, no core-schema pre-creation#1
dnplkndll wants to merge 3 commits into
mainfrom
fix/helm-fresh-install

Conversation

@dnplkndll

@dnplkndll dnplkndll commented Jul 26, 2026

Copy link
Copy Markdown

PRACTICE PR on fork — for quality review before filing upstream. Do not merge.

Title: fix(helm): make fresh installs work — valid db tag, working init SQL, no core-schema pre-creation

Fixes #TBD (upstream issue to be filed on approval).

What was broken

  1. Dead image tagtwenty-postgres-spilo:3.3-p2 is gone from Docker Hub
    → db pod ErrImagePull with default values.
  2. Init SQL never executed — psql interpolates :'var' only in plain stdin
    statements, not in -c strings or dollar-quoted DO bodies; the init used
    both, so all of its SQL errored. Installs relied on spilo pre-creating the
    database and app user with an image-baked password that doesn't match the
    chart secret → fresh installs crash-loop on password authentication failed.
  3. Migration skip — pre-creating core defeats the entrypoint's
    fresh-database check, so base migrations were skipped and the server booted
    "healthy" against an empty database (signUprelation "core.user" does not exist).

Changes

  • db image tag → pinned v0.43.5 (newest published tag; a floating latest
    into a deprecated repo can drift silently). Replacing spilo with postgres:16
    per [MISC] DeprecatesSpilo twentyhq/twenty#10883 is proposed separately; the rewritten init works with either image.
  • Init SQL → one stdin heredoc, ON_ERROR_STOP=1, format(...) \gexec for the
    conditional CREATEs, unconditional ALTER USER … PASSWORD to re-sync the
    secret, and database/public ownership transfer instead of schema
    pre-creation + grants (migrations create schemas themselves, correctly owned).
  • Tests assert the new contract, including notMatchRegex: CREATE SCHEMA so
    the migration-skip bug cannot return. 54/54 pass.

Verification (kind v0.32, transcripts below)

  • Before: default values → ErrImagePull; live tag → CrashLoopBackOff on
    app-user auth.
  • After, default values only: server 1/1, core populated (53 tables), signUp
    returns a token — no manual intervention.

Transcripts

Before — default values (dead tag)

$ kubectl -n t20 get pods   # chart installed with DEFAULT values
NAME                                 READY   STATUS         RESTARTS   AGE
t20-twenty-db-794c4b664-sxppg        0/1     ErrImagePull   0          30s
t20-twenty-redis-6d89c94b97-znnzc    1/1     Running        0          30s
t20-twenty-server-864b49b65c-zbl8k   0/1     Init:0/2       0          30s
t20-twenty-worker-5999544fb6-jjwmr   0/1     Init:0/1       0          30s

$ kubectl -n t20 describe pod -l app=t20-db | grep -A2 Failed
Error: ErrImagePull
Error: ImagePullBackOff
Failed to pull image "twentycrm/twenty-postgres-spilo:3.3-p2": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/twentycrm/twenty-postgres-spilo:3.3-p2": failed to resolve reference "docker.io/twentycrm/twenty-postgres-spilo:3.3-p2": docker.io/twentycrm/twenty-postgres-spilo:3.3-p2: not found

Before — live tag, stock init (auth crash-loop)

$ kubectl -n t20 get pods   # stock chart @ main, only db tag overridden to a live one
NAME                                 READY   STATUS             RESTARTS      AGE
t20-twenty-db-6694fc4684-8f5ff       1/1     Running            0             2m30s
t20-twenty-redis-6d89c94b97-cdkcf    1/1     Running            0             2m30s
t20-twenty-server-864b49b65c-nxk9d   0/1     CrashLoopBackOff   3 (55s ago)   2m30s
t20-twenty-worker-5999544fb6-cdncf   1/1     Running            2 (35s ago)   2m30s

$ kubectl -n t20 logs deploy/t20-twenty-server --tail=6
Running database setup and migrations...
psql: error: connection to server at "t20-twenty-db.t20.svc.cluster.local" (10.96.108.29), port 5432 failed: FATAL:  password authentication failed for user "twenty_app_user"
connection to server at "t20-twenty-db.t20.svc.cluster.local" (10.96.108.29), port 5432 failed: FATAL:  password authentication failed for user "twenty_app_user"

After — this branch, default values

$ kubectl -n t20 get pods   # fix/helm-fresh-install (review-final, spilo v0.43.5), DEFAULT values
NAME                                 READY   STATUS    RESTARTS   AGE
t20-twenty-db-74f5584799-6cgc8       1/1     Running   0          2m15s
t20-twenty-redis-6d89c94b97-vwcft    1/1     Running   0          2m15s
t20-twenty-server-66fb986974-cwbzw   1/1     Running   0          2m15s
t20-twenty-worker-5999544fb6-q72rs   1/1     Running   0          2m15s

$ init container log
ALTER ROLE
ALTER DATABASE
ALTER SCHEMA
Database twenty is ready.

$ core table count
53

$ signUp mutation
{"data":{"signUp":{"tokens":{"accessOrWorkspaceAgnosticToken":{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6

@dnplkndll
dnplkndll force-pushed the fix/helm-fresh-install branch 2 times, most recently from 5d926e0 to e97409e Compare July 26, 2026 17:10
twenty-postgres-spilo:3.3-p2 was removed from Docker Hub; default installs
fail at ErrImagePull before anything else runs. Pin the newest published
tag. (Replacing spilo with postgres:16 per twentyhq#10883 is proposed separately.)
…re-creating core schema

psql interpolates :'var' only in plain stdin statements — not in -c
strings and not inside dollar-quoted DO bodies — so none of the previous
init SQL ever ran. Installs only appeared to work because spilo images
pre-create the database and app user with an image-baked password, which
does not match the chart-generated secret: fresh installs crash-loop on
"password authentication failed".

The rewrite uses one heredoc with \gexec conditionals, always re-syncs the
app-user password, and replaces schema pre-creation with database/public
ownership: the entrypoint treats an existing "core" schema as "already
migrated", so creating it here made every fresh install skip base
migrations and boot against an empty database.
@dnplkndll
dnplkndll force-pushed the fix/helm-fresh-install branch from e97409e to 61ddc9e Compare July 26, 2026 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant