fix(ci): bound queue DB connections to stop E2E 1040 stalls - #700
Merged
Conversation
behinddwalls
marked this pull request as ready for review
September 11, 2026 04:52
## Summary ### Why? Tenant-scoped reconciliation fans out one goroutine per tenant per subscription tick. With `database/sql`'s unlimited open-connection default, the E2E stack opened more sockets than MySQL's ~151-connection ceiling, returned Error 1040, and stalled pipeline progress until Bazel timed out. Gateway integration also published logs under a tenant that was not in `MQ_TENANTS`, so the consumer never saw them. ### What? - Apply `QUEUE_MYSQL_MAX_OPEN_CONNECTIONS` (default 16) on each service-owned queue `*sql.DB`, and keep that many connections idle so the cap does not churn sockets. - Point the gateway log-consumer integration at the configured `test-queue` tenant and scope the land-message lookup by tenant. - Upload Bazel `test.log` files on failure for every CI test job. Uploads authenticate with the Actions runtime token, so the workflow keeps its `contents: read`-only `GITHUB_TOKEN`. ## Test Plan ✅ `./tool/bazel test //service/messagequeue:go_default_test` ✅ `./tool/bazel test //test/integration/submitqueue/gateway:go_default_test --sandbox_writable_path="$HOME/.docker/buildx"` ✅ `./tool/bazel test //test/e2e/submitqueue:go_default_test --sandbox_writable_path="$HOME/.docker/buildx"` (167s, then 161s) ✅ `./tool/bazel test //test/e2e/submitqueue:go_default_test --test_filter='TestGitMergeE2E' --sandbox_writable_path="$HOME/.docker/buildx"` ✅ `docker run --rm -v "$PWD:/src:ro" -w /src ghcr.io/zizmorcore/zizmor:1.25.2 --persona regular .` — no findings ✅ `make fmt && make gazelle && make tidy` Co-authored-by: Cursor <cursoragent@cursor.com>
behinddwalls
force-pushed
the
preetam/fix-main-ci-test-boundaries
branch
from
September 11, 2026 04:57
b828467 to
e15dc9c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
Tenant-scoped reconciliation fans out one goroutine per tenant per subscription tick. With
database/sql's unlimited open-connection default, the E2E stack opened more sockets than MySQL's ~151-connection ceiling, returned Error 1040, and stalled pipeline progress until Bazel timed out. Gateway integration also published logs under a tenant that was not inMQ_TENANTS, so the consumer never saw them.What?
QUEUE_MYSQL_MAX_OPEN_CONNECTIONS(default 16) on each service-owned queue*sql.DB, and keep that many connections idle so the cap does not churn sockets.test-queuetenant and scope the land-message lookup by tenant.test.logfiles on failure for every CI test job. Uploads authenticate with the Actions runtime token, so the workflow keeps itscontents: read-onlyGITHUB_TOKEN.Test Plan
✅
./tool/bazel test //service/messagequeue:go_default_test✅
./tool/bazel test //test/integration/submitqueue/gateway:go_default_test --sandbox_writable_path="$HOME/.docker/buildx"✅
./tool/bazel test //test/e2e/submitqueue:go_default_test --sandbox_writable_path="$HOME/.docker/buildx"(167s, then 161s)✅
./tool/bazel test //test/e2e/submitqueue:go_default_test --test_filter='TestGitMergeE2E' --sandbox_writable_path="$HOME/.docker/buildx"✅
docker run --rm -v "$PWD:/src:ro" -w /src ghcr.io/zizmorcore/zizmor:1.25.2 --persona regular .— no findings✅
make fmt && make gazelle && make tidy