docker compose up fails consistently with a sqlalchemy error during db initialization

[collectoss-issue-migrator]

Note

Migrated from augurlabs/augur#3634
Originally opened by @dcermak on 2026-01-29

---

Please help us help you by filling out the following sections as thoroughly as you can.

Description:

docker compose up doesn't result in a running system on Fedora 43 consistently on both the release and main branches. The augur container dies consistently with a database error.

The first error appears to be

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "contributor_repo" does not exist

which then results also:

augur-1  | sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "augur_operations.config" does not exist
augur-1  | LINE 2: FROM augur_operations.config 
augur-1  |              ^
augur-1  | 
augur-1  | [SQL: SELECT augur_operations.config.id AS augur_operations_config_id, augur_operations.config.section_name AS augur_operations_config_section_name, augur_operations.config.setting_name AS augur_operations_config_setting_name, augur_operations.config.value AS augur_operations_config_value, augur_operations.config.type AS augur_operations_config_type 
augur-1  | FROM augur_operations.config 
augur-1  | WHERE augur_operations.config.section_name = %(section_name_1)s AND augur_operations.config.setting_name = %(setting_name_1)s]
augur-1  | [parameters: {'section_name_1': 'Redis', 'setting_name_1': 'cache_group'}]

This appeared multiple times on different systems during ChaossCon

How to reproduce:

1. run docker compose up

Log files

Software versions:

• Augur: main or release branch
• OS: Fedora 43
• Browser: n/a

[collectoss-issue-migrator]

Note

Migrated comment from original
Originally posted by @deepanshu-singh22 on 2026-02-02

---

Hi! I’m interested in working on this issue as part of GSoC. Can I take this up?

[collectoss-issue-migrator]

Note

Migrated comment from original
Originally posted by @MoralCode on 2026-02-02

---

Hi! I’m interested in working on this issue as part of GSoC. Can I take this up?

CHAOSS is not participating in GSoC this year. I'd be happy to chat about this in more detail over on the CHAOSS Slack in the #wg-augur-8knot channel if you'd like.

[collectoss-issue-migrator]

Note

Migrated comment from original
Originally posted by @MoralCode on 2026-02-03

---

OP reported this when using Fedora 43.
I saw this when using Fedora 42.
(both of these MIGHT explain why the unit tests, which run on ubuntu images, didnt catch this.

I also have logs from someone who attended CHAOSScon who experienced this on an Arm64 Mac from a fresh start after deleting the augur_augurpostgres docker volume:

(trimmed) Mac Arm logs

augur-db-1  | The files belonging to this database system will be owned by user "postgres".
augur-db-1  | This user must also own the server process.
augur-db-1  | 
augur-db-1  | The database cluster will be initialized with locale "en_US.utf8".
augur-db-1  | The default database encoding has accordingly been set to "UTF8".
augur-db-1  | The default text search configuration will be set to "english".
augur-db-1  | 
augur-db-1  | Data page checksums are disabled.
augur-db-1  | 
augur-db-1  | fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
augur-db-1  | creating subdirectories ... ok
augur-db-1  | selecting dynamic shared memory implementation ... posix
augur-db-1  | selecting default max_connections ... 100
augur-db-1  | selecting default shared_buffers ... 128MB
augur-db-1  | selecting default time zone ... Etc/UTC
augur-db-1  | creating configuration files ... ok
augur-db-1  | running bootstrap script ... ok
augur-keyman-1  | 2026-01-29 16:35:15,233 - KeyOrchestrator - INFO - Ready
augur-db-1      | performing post-bootstrap initialization ... ok
augur-db-1      | syncing data to disk ... ok
augur-db-1      | 
augur-db-1      | 
augur-db-1      | Success. You can now start the database server using:
augur-db-1      | 
augur-db-1      |     pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start
augur-db-1      | 
augur-db-1      | initdb: warning: enabling "trust" authentication for local connections
augur-db-1      | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
augur-db-1      | waiting for server to start....2026-01-29 16:35:15.378 UTC [50] LOG:  starting PostgreSQL 16.11 (Debian 16.11-1.pgdg13+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
augur-db-1      | 2026-01-29 16:35:15.379 UTC [50] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
augur-db-1      | 2026-01-29 16:35:15.380 UTC [53] LOG:  database system was shut down at 2026-01-29 16:35:15 UTC
augur-db-1      | 2026-01-29 16:35:15.382 UTC [50] LOG:  database system is ready to accept connections
augur-db-1      |  done
augur-db-1      | server started
augur-db-1      | CREATE DATABASE
augur-db-1      | 
augur-db-1      | 
augur-db-1      | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
augur-db-1      | 
augur-db-1      | waiting for server to shut down....2026-01-29 16:35:15.516 UTC [50] LOG:  received fast shutdown request
augur-db-1      | 2026-01-29 16:35:15.517 UTC [50] LOG:  aborting any active transactions
augur-db-1      | 2026-01-29 16:35:15.518 UTC [50] LOG:  background worker "logical replication launcher" (PID 56) exited with exit code 1
augur-db-1      | 2026-01-29 16:35:15.520 UTC [51] LOG:  shutting down
augur-db-1      | 2026-01-29 16:35:15.520 UTC [51] LOG:  checkpoint starting: shutdown immediate
augur-db-1      | 2026-01-29 16:35:15.538 UTC [51] LOG:  checkpoint complete: wrote 926 buffers (5.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.006 s, sync=0.011 s, total=0.018 s; sync files=301, longest=0.003 s, average=0.001 s; distance=4273 kB, estimate=4273 kB; lsn=0/191F080, redo lsn=0/191F080
augur-db-1      | 2026-01-29 16:35:15.540 UTC [50] LOG:  database system is shut down
augur-db-1      |  done
augur-db-1      | server stopped
augur-db-1      | 
augur-db-1      | PostgreSQL init process complete; ready for start up.
augur-db-1      | 
augur-db-1      | 2026-01-29 16:35:15.651 UTC [1] LOG:  starting PostgreSQL 16.11 (Debian 16.11-1.pgdg13+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
augur-db-1      | 2026-01-29 16:35:15.651 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
augur-db-1      | 2026-01-29 16:35:15.651 UTC [1] LOG:  listening on IPv6 address "::", port 5432
augur-db-1      | 2026-01-29 16:35:15.652 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
augur-db-1      | 2026-01-29 16:35:15.654 UTC [66] LOG:  database system was shut down at 2026-01-29 16:35:15 UTC
augur-db-1      | 2026-01-29 16:35:15.655 UTC [1] LOG:  database system is ready to accept connections
augur-1         | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
augur-1         | INFO  [alembic.runtime.migration] Will assume transactional DDL.
augur-1         | INFO  [alembic.runtime.migration] Running upgrade  -> 0, Goodbye, .sql migrations
augur-db-1      | 2026-01-29 16:35:18.542 UTC [71] ERROR:  relation "augur_settings" does not exist at character 26
augur-db-1      | 2026-01-29 16:35:18.542 UTC [71] STATEMENT:  SELECT "value"::int FROM augur_settings WHERE setting = 'augur_data_version'
augur-1         | INFO  [alembic.runtime.migration] New database, will run all legacy migrations
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 80.1-create-schema.sql
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 80.2-create-tables.sql
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 80.3-sample-data.sql
augur-db-1      | 2026-01-29 16:35:18.743 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.743 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.744 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.744 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.744 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.745 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.745 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.747 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.747 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.747 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.747 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 83.sql
augur-db-1      | 2026-01-29 16:35:18.753 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.754 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 84.sql
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 85.sql
augur-db-1      | 2026-01-29 16:35:18.758 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.758 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.760 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 86.sql
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 87.sql
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 88.sql
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 89.sql
augur-db-1      | 2026-01-29 16:35:18.762 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.763 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.763 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 90.sql
augur-db-1      | 2026-01-29 16:35:18.765 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 91.sql
augur-db-1      | 2026-01-29 16:35:18.766 UTC [71] WARNING:  there is already a transaction in progress
augur-db-1      | 2026-01-29 16:35:18.766 UTC [71] WARNING:  there is already a transaction in progress


augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 92.sql
augur-db-1      | 2026-01-29 16:35:18.766 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 94.sql
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 95.sql
augur-db-1      | 2026-01-29 16:35:18.768 UTC [71] WARNING:  there is already a transaction in progress


augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 96.sql
augur-db-1      | 2026-01-29 16:35:18.770 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 97.sql
augur-db-1      | 2026-01-29 16:35:18.772 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 98.sql
augur-db-1      | 2026-01-29 16:35:18.776 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 99.sql
augur-db-1      | 2026-01-29 16:35:18.779 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running legacy migration 100.sql
augur-db-1      | 2026-01-29 16:35:18.781 UTC [71] WARNING:  there is already a transaction in progress
augur-1         | INFO  [alembic.runtime.migration] Running upgrade 0 -> 1, Augur New Changes
augur-db-1      | 2026-01-29 16:35:18.809 UTC [71] ERROR:  relation "contributor_repo" does not exist
augur-db-1      | 2026-01-29 16:35:18.809 UTC [71] STATEMENT:  ALTER TABLE contributor_repo DROP COLUMN cntrb_id
augur-db-1      | 2026-01-29 16:35:18.809 UTC [71] WARNING:  there is no transaction in progress
augur-1         | Traceback (most recent call last):
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
augur-1         |     self.dialect.do_execute(
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
augur-1         |     cursor.execute(statement, parameters)
augur-1         | psycopg2.errors.UndefinedTable: relation "contributor_repo" does not exist
augur-1         | 
augur-1         | 
augur-1         | The above exception was the direct cause of the following exception:
augur-1         | 
augur-1         | Traceback (most recent call last):
augur-1         |   File "/augur/.venv/bin/alembic", line 10, in <module>
augur-1         |     sys.exit(main())
augur-1         |              ^^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/config.py", line 1033, in main
augur-1         |     CommandLine(prog=prog).main(argv=argv)
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/config.py", line 1023, in main
augur-1         |     self.run_cmd(cfg, options)
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/config.py", line 957, in run_cmd
augur-1         |     fn(
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/command.py", line 483, in upgrade
augur-1         |     script.run_env()
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/script/base.py", line 545, in run_env
augur-1         |     util.load_python_file(self.dir, "env.py")
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 116, in load_python_file
augur-1         |     module = load_module_py(module_id, path)
augur-1         |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 136, in load_module_py
augur-1         |     spec.loader.exec_module(module)  # type: ignore
augur-1         |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
augur-1         |   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
augur-1         |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
augur-1         |   File "/augur/augur/application/schema/alembic/env.py", line 113, in <module>
augur-1         |     run_migrations_online()
augur-1         |   File "/augur/augur/application/schema/alembic/env.py", line 107, in run_migrations_online
augur-1         |     context.run_migrations()
augur-1         |   File "<string>", line 8, in run_migrations
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
augur-1         |     self.get_context().run_migrations(**kw)
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
augur-1         |     step.migration_fn(**kw)
augur-1         |   File "/augur/augur/application/schema/alembic/versions/1_augur_new_changes.py", line 28, in upgrade
augur-1         |     change_cntrb_id_to_uuid_5()
augur-1         |   File "/augur/augur/application/schema/alembic/versions/1_augur_new_changes.py", line 217, in change_cntrb_id_to_uuid_5
augur-1         |     op.drop_column('contributor_repo', 'cntrb_id')              
augur-1         |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
augur-1         |   File "<string>", line 8, in drop_column
augur-1         |   File "<string>", line 3, in drop_column
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/operations/ops.py", line 2341, in drop_column
augur-1         |     return operations.invoke(op)
augur-1         |            ^^^^^^^^^^^^^^^^^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/operations/base.py", line 441, in invoke
augur-1         |     return fn(self, operation)
augur-1         |            ^^^^^^^^^^^^^^^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/operations/toimpl.py", line 95, in drop_column
augur-1         |     operations.impl.drop_column(
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/ddl/impl.py", line 396, in drop_column
augur-1         |     self._exec(
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/alembic/ddl/impl.py", line 246, in _exec
augur-1         |     return conn.execute(construct, params)
augur-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
augur-1         |     return meth(
augur-1         |            ^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/sql/ddl.py", line 181, in _execute_on_connection
augur-1         |     return connection._execute_ddl(
augur-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1528, in _execute_ddl
augur-1         |     ret = self._execute_context(
augur-1         |           ^^^^^^^^^^^^^^^^^^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
augur-1         |     return self._exec_single_context(
augur-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
augur-1         |     self._handle_dbapi_exception(
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
augur-1         |     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
augur-1         |     self.dialect.do_execute(
augur-1         |   File "/augur/.venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
augur-1         |     cursor.execute(statement, parameters)
augur-1         | sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "contributor_repo" does not exist
augur-1         | 
augur-1         | [SQL: ALTER TABLE contributor_repo DROP COLUMN cntrb_id]
augur-1         | (Background on this error at: https://sqlalche.me/e/20/f405)

To me it looks like the chain of events here is:

1. database starts, detects no existing database and creates a new one (see CREATE DATABASE in the logs)
2. database restarts - not sure why. maybe this is normal
3. Migration 0 begins (Running upgrade -> 0, Goodbye, .sql migrations)
4. Migration 0 fails (RROR: relation "augur_settings" does not exist at character 26)
5. other sub components of that migration 0 (which itself contains more migrations not versioned with alembic) keep going.
6. WARNING: there is already a transaction in progress - probably due to the failure in step 4
7. repeat 5 and 6 a few times
8. migration 1 starts running (Running upgrade 0 -> 1, Augur New Changes)
9. that fails with ERROR: relation "contributor_repo" does not exist and various similar errors
10. the augur backend starts restart looping, possibly due to configured docker restart behavior that is intentional and doesnt need changing.

This issue is likely the same as augurlabs/augur#3548.

While augurlabs/augur#3559 was tried during the conference as a fix attempt, it didnt work.

Given the above, i would strongly suspect that augurlabs/augur#3551 may be a viable fix. Needs testing though.

Edit: augurlabs/augur#3351 also explains why the CI doesnt see this: becuause the CI is already effectively using the patch in that PR...

Must have just been a little too scatterbrained during CHAOSScon to have thought about that PR.

[collectoss-issue-migrator]

Note

Migrated comment from original
Originally posted by @MoralCode on 2026-02-03

---

my testing (cherry picking the change from 3351 onto the chaosscon branch to combine the two fixes), seems to have resolved the issue on Fedora 42.

Will wait maybe 24-48 hrs for people to test, but im inclined to merge this after that window even with no additional feedback.

[collectoss-issue-migrator]

Note

Migrated comment from original
Originally posted by @choeppler on 2026-02-03

---

I've run into this issue 5 days ago on a Mac Arm as described above. Just tried augurlabs/augur#3551, which resolves the issue for me as well.

[collectoss-issue-migrator]

Note

Migrated comment from original
Originally posted by @MoralCode on 2026-02-03

---

Sounds good! Going to mark this issue as a duplicate of the other issue that was filed. Also hoping to merge the fix soon (next day or so)

(it also looks like Sean made augurlabs/augur#3643 based on that CHAOSScon branch i mentioned too - i think that may be most likely what I merge)