Skip to content

Flaky parallel tests: PG deadlock burst (50 DRb::DRbRemoteError / PG::TRDeadlockDetected in one run) #349

Description

@kylefritz

Priority: low — test reliability. Observed once while verifying #342's fix (2026-06-11): one of six consecutive full parallel suite runs failed with 50 errors, all DRb::DRbRemoteError: PG::TRDeadlockDetected: ERROR: deadlock detected. The other five runs were green; an immediate rerun did not reproduce, so I couldn't capture the full deadlock detail (which relations/locks were in the cycle).

This is a different failure mode from #342 (RecordNotFound from the stale RailsSettings cache) and is not addressed by that fix.

Hypothesis worth checking first

OrdersController#create takes pg_advisory_xact_lock(Zlib.crc32("order:<user_id>:<menu_id>")). Postgres advisory locks are cluster-wide, not per-database — parallel test workers use separate databases (motzi_test-N) but share one cluster, and since every worker loads identical fixtures, they compute identical lock keys. Cross-worker contention on the same advisory key, combined with each worker's own row locks, could form a deadlock cycle under unlucky scheduling. CI shares one Postgres service container, so it's exposed to the same mechanism.

If that's the cause, candidate fixes:

  • include something worker-unique in the lock key under test (e.g. mix Rails.env + database name into the crc32 input), or
  • move the duplicate-check lock to a row-level lock (SELECT ... FOR UPDATE on the user row), which is naturally per-database

Next time it fires

Grab the full error output — Postgres reports both lock holders in the deadlock message (Process X waits for ...; Process Y waits for ...), which will confirm or kill the advisory-lock hypothesis.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrubyPull requests that update Ruby code

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions