Stabilize MySQL testcontainer startup in CI failpoint tests

[mashenjun]

Background

The code-ci / ci workflow has repeatedly failed in the Failpoint test step while starting the MySQL testcontainer used by internal/testmysql.Start. This is a CI/test-infrastructure failure, not a business assertion failure.

The failure was observed while validating PR #677, whose diff does not touch internal/testmysql, scripts/run_failpoint_tests.py, Makefile, or .github/workflows/code-ci.yml; the PR currently changes e2e smoke scripts and scripts/verify_description.sh.

Reproduction / Observed Runs

Run: https://github.com/mem9-ai/drive9/actions/runs/28854834904?pr=677

All attempts failed in Failpoint test after Lint and the normal Test step had passed:

• Attempt 1, job 85578689663, runner region westus3: failed starting the first failpoint MySQL container for pkg/backend.
• Attempt 2, job 85580414141, runner region westus: failed starting the first failpoint MySQL container for pkg/backend.
• Attempt 3, job 85585205164, runner region eastus: pkg/backend failpoint tests passed and terminated their MySQL container, then pkg/datastore failed starting the next MySQL container.

A previous PR #677 run also showed the same class of failure: https://github.com/mem9-ai/drive9/actions/runs/28853035608

Evidence

Representative failure line:

setup mysql test instance: start mysql container: run mysql: generic container: start container: container start: Error response from daemon: failed to set up container networking: driver failed programming external connectivity ... failed to bind host port for 0.0.0.0::172.18.0.2:3306/tcp: address already in use

Affected command path:

make test-failpoint
./scripts/run_failpoint_tests.py
go test -v -tags failpoint <package> -run <failpoint tests>
internal/testmysql.Start -> tcmysql.Run(ctx, "mysql:8.0.36", ...)

Impact

CI can fail before running the actual failpoint test assertions. This blocks otherwise unrelated PRs and makes reruns unreliable. Since the regular Test step creates/stops many MySQL testcontainers before Failpoint test, the failpoint step can hit Docker port/network cleanup timing when it immediately starts another MySQL container.

Root Cause Analysis / Suspected Path

The current internal/testmysql.Start calls tcmysql.Run once and returns an error immediately on container start failure. The observed Docker error is a transient host port/network binding failure from testcontainers/Docker, not a deterministic application-level failure.

The issue appears when MySQL containers are created and terminated in quick succession across normal tests and failpoint packages. Docker may not have fully released port/network state when the next mysql:8.0.36 container starts.

Suggested Fix

Add targeted retry/backoff around tcmysql.Run in internal/testmysql.Start:

• Retry only for transient Docker bind errors containing failed to bind host port or address already in use.
• Use a small bounded retry count, for example 3 attempts with 1s/2s backoff.
• If tcmysql.Run returns a partially created container with an error, terminate it before retrying.
• Do not retry unrelated errors, so real setup failures remain visible.

A larger alternative is to update scripts/run_failpoint_tests.py so all failpoint packages reuse one explicit DRIVE9_TEST_MYSQL_DSN, but that requires managing a longer-lived MySQL container and is a broader change.

Validation Plan

• gofmt internal/testmysql/testmysql.go
• go test ./internal/testmysql
• Rerun PR code-ci / ci
• Confirm Failpoint test no longer fails on transient MySQL testcontainer bind errors