Skip to content

[#13529] improvement(test): Run core database lanes with bounded adaptive forks - #13530

Open
nevzheng wants to merge 8 commits into
apache:mainfrom
nevzheng:codex/13529-core-database-adaptive-forks
Open

nevzheng wants to merge 8 commits into
apache:mainfrom
nevzheng:codex/13529-core-database-adaptive-forks

Conversation

@nevzheng

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

This patch lets the explicit core H2, MySQL, and PostgreSQL lanes use up to two isolated Gradle test workers while keeping JUnit execution serial inside each worker.

It:

  • derives maxParallelForks from CPU, JVM-visible memory, Gradle worker capacity, concurrently selected database lanes, and a conservative rollout cap;
  • accepts strict property and environment overrides, enforces every known bound, and falls back to one worker when automatic resource signals are incomplete;
  • keeps Docker lanes at one worker on macOS fixed-network connector configurations;
  • adds exact-type DatabaseTestContext injection while preserving [#6505] improvement(test): Reuse core database fixtures by test class #13518's class-scoped cleanup, poisoning, rebuild, and fresh-fixture behavior;
  • records per-worker and per-class [CORE-DB-SHARD] telemetry and requires it in each database lane's CI evidence artifact; and
  • adds focused policy, injection, and telemetry regression tests.

Adaptive forks apply only to the explicit database lane tasks. The unit lane and legacy :core:test behavior remain unchanged, and JUnit method-level parallelism remains disabled.

Why are the changes needed?

#13518 gives each test class an explicit fixture lifecycle, but each database lane still runs in one JVM and leaves available CI capacity unused. Bounded JVM-level concurrency reduces wall time while preserving worker-owned database servers and exact test identities.

This PR is stacked on #13518. It does not introduce connection pooling, shared database servers, namespace sharding, or method-level parallelism.

Fix: #13529

Depends on: #13518

Related to: #13515 and #13517

Does this PR introduce any user-facing change?

No production or runtime user-facing change. Public APIs, runtime configuration, and storage formats are unchanged.

The only new controls are test/CI-only Gradle properties and matching environment variables:

  • coreDatabaseForks / CORE_DATABASE_FORKS
  • coreDatabaseBuildMemoryReserveMiB / CORE_DATABASE_BUILD_MEMORY_RESERVE_MIB
  • coreDatabaseForkMemoryOverheadMiB / CORE_DATABASE_FORK_MEMORY_OVERHEAD_MIB

How was this patch tested?

Matched local runs compared #13518 at 095f7d7430f4a8c441ba7c095210fed72045b7c2 with this patch at 5a51aa804b834b7f198c472ee1f045ebea778b65. The environment was macOS 27.0 arm64, OrbStack, JDK 17, Gradle 8.2, 14 JVM-visible processors, 48 GiB JVM-visible memory, and --max-workers=2; every run used --rerun-tasks.

Lane Baseline task wall Candidate task wall Improvement Result parity Fixtures Containers Candidate workers
H2 63.696s 53.455s 10.241s / 16.1% / 1.19x 519 passed; exact identities 48 → 48 n/a 2
MySQL 1326.146s 932.091s 394.055s / 29.7% / 1.42x 519 passed; exact identities 48 → 48 1 → 2 2
PostgreSQL 485.720s 307.759s 177.961s / 36.6% / 1.58x 517 passed, 2 skipped; exact identities 50 → 50 1 → 2 2

Additional completed checks:

  • ./gradlew -p buildSrc test --rerun-tasks: 36 policy tests passed.
  • :core:coreUnitTest: 1,972 passed and 2 skipped.
  • Four-lane reconciliation: 3,531 tests total, identical database-lane identity multisets, and no unit/database overlap.
  • Candidate telemetry reconciled with all 40 JUnit classes and all 519 tests in each database lane.
  • Docker containers were absent after each lane completed.
  • actionlint .github/workflows/build.yml, Spotless, Kotlin Gradle formatting, RAT, and :core:compileTestJava passed.
  • python3 -B -m unittest discover -s dev/ci/tests -p 'test_core_test_identity.py': 9 passed.

yuqi1129 and others added 8 commits September 23, 2026 11:16
…the aggregate result jobs

- Fold the `client` Backend IT shard into `others`. Both are short, and the
  merged shard still finishes before the slowest build shard, so PR feedback
  time is unchanged while each backend runs one job less.
- Remove `build-result` and `BackendIT-result`. No branch protection rule
  requires them, so they only added checks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Subtask] Run core database test lanes with bounded adaptive forks

2 participants