Skip to content

fix(db): bound connection pools and back off edge heartbeats - #486

Merged
youzi-1122 merged 1 commit into
mainfrom
fix/db-connection-pressure
Oct 9, 2026
Merged

youzi-1122 merged 1 commit into
mainfrom
fix/db-connection-pressure

Conversation

@youzi-1122

@youzi-1122 youzi-1122 commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Summary

Concurrent device heartbeats could open an unbounded number of MySQL connections. Each failed heartbeat then triggered registration writes, and repeated failures could restart the Edge process, adding more load during a database outage.

  • Bound each Manager's MySQL pool to 50 open / 10 idle connections by default. Validate configurable pool budgets, retain reusable connections, and retire idle/old connections. Wire both Compose stacks and document multi-instance budgeting, rollout, and rollback.
  • Retry failed heartbeats and initial registrations with capped exponential backoff and jitter. Keep stale-route recovery in the existing tunnel layer; successful reconnect registration also writes the upgrade health marker.
  • Add regression coverage for database outages, recovery, cancellation, reconnects, and 220 concurrent devices against a disposable MySQL instance.

Refs #473. This addresses the verified connection-pressure amplification mechanism; the initiating customer trigger and the contribution of the slow alert query remain unconfirmed.

Validation

  • Passed affected-package go test -race and go vet for config, dbx, Edge biz, and tunnel. Re-ran Edge race/vet after the final reconnect health-marker change.
  • Passed Linux go test -p 2 ./... in an isolated source copy as a non-root user. macOS cannot run all Linux-only packages; initial Linux container failures from a stale read-only module cache, a host worktree path, and root permission semantics were resolved by correcting the test environment.
  • Passed ONGRID_TEST_DB_POOL=1 TESTCONTAINERS_RYUK_DISABLED=true go test -race -tags=integration ./tests/integration -run '^TestMySQLPoolHeartbeatBurst$' -count=1 -v: 220 devices, 660 heartbeats, zero heartbeat failures and zero MySQL connection-limit rejections at server max_connections=151. Verified pool saturation honors request deadlines, releases connections, recovers, and accepts a custom limit. The test container was removed.
  • Passed go test -tags=e2e ./tests/e2e -run '^TestAuth_LoginAndSelf_B1$' -count=1 with a disposable MySQL and actual Manager process.
  • Verified default and overridden pool limits in rendered development and installation Compose configuration; git diff --check passed.
  • make arch-lint could not run because go-arch-lint is not installed. No customer-environment validation or release has been performed.

Risk and rollback

The total budget must cover every Manager replica and other database clients, with operational headroom. An undersized pool can increase queueing/timeouts; slow queries or database resource problems still require investigation. Upgrade Manager first, then Edge; old Edge versions retain the immediate-registration retry behavior.

No schema, API, or dependency changes. Adjust pool environment variables and recreate Manager to change the budget. Reverting the images restores the old unbounded pool/retry behavior and therefore the original failure risk. See docs/install/mysql-connection-pool.md for configuration and acceptance steps.

Author confirmation

Limit MySQL connections per Manager, retain idle connections, and expose pool budgets in both Compose stacks. Back off failed heartbeats without immediate registration writes or process restarts while preserving tunnel recovery and upgrade health markers.

Refs #473
@youzi-1122
youzi-1122 marked this pull request as ready for review October 9, 2026 07:55
@youzi-1122
youzi-1122 requested a review from singchia as a code owner October 9, 2026 07:55
@youzi-1122
youzi-1122 merged commit 147e9c1 into main Oct 9, 2026
13 of 14 checks passed
@youzi-1122
youzi-1122 deleted the fix/db-connection-pressure branch October 9, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant