From dae5a2497bfe8f90e435d2b618fdc3217de9eb8b Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 15:41:02 +0100 Subject: [PATCH 1/5] docs(plan): bound Backend CI cache optimization --- .../changes/backend-ci-cache-and-timing.md | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 .commitrail/changes/backend-ci-cache-and-timing.md diff --git a/.commitrail/changes/backend-ci-cache-and-timing.md b/.commitrail/changes/backend-ci-cache-and-timing.md new file mode 100644 index 000000000..00a3cac99 --- /dev/null +++ b/.commitrail/changes/backend-ci-cache-and-timing.md @@ -0,0 +1,132 @@ +# Backend CI Cache And Timing Diagnostics + +- Initiative: `None` +- Durable disposition: `Planned` +- Intended merge outcome: Backend CI reuses an exact content-addressed MinIO + source image and pip download cache while retaining every semantic test, + fail-closed aggregate, and useful per-test duration evidence. + +## Intent + +Reduce repeated setup work in the required Backend workflow and retain enough +timing evidence to rebalance slow lanes from measurements. This change targets +only deterministic downloads/build inputs and diagnostics; it does not trade +test or evidence coverage for a faster green result. + +## Current behavior + +`.github/workflows/backend.yml` builds the source-pinned MinIO image once per +commit because its cache key includes `github.sha`. Backend jobs create fresh +Python environments and repeatedly download the same dependency inputs. The +semantic runner retains each lane log but reports only the 25 slowest pytest +phases, limiting later balancing evidence. + +GitHub Backend run `37784941972` on PR #507 is the measurement source for this +slice. Its first MinIO job spent 145 seconds building after a cache miss. Lane +jobs also repeatedly spent about 25--28 seconds installing backend dependencies, +and queued for materially different periods. These measurements describe that +run only; this record makes no hosted performance claim before a candidate run. + +## Bounded change + +### Allowed + +- `.github/workflows/backend.yml`: make the MinIO tar cache key depend on the + runner OS/architecture and exact `docker/minio/**` content, retaining exact + per-run artifact identity, checksums and live provider verification. +- `.github/workflows/backend.yml`: enable only setup-python's pip download cache + for the exact backend dependency/workflow inputs; every job still performs a + fresh install into its fresh hosted Python environment. +- `backend/scripts/run_test_lanes.py`: retain all pytest duration diagnostics in + the already authenticated per-lane logs without changing selection or timeout. +- `scripts/test_lightweight_agent_gates.py` and + `backend/tests/test_ci_test_lanes.py`: focused positive and adversarial + regressions for cache identity, fresh installs, runtime verification, artifact + identity, and lane duration options. +- `CONTRIBUTING.md`: concise operator guidance for failed-job reruns, same-head + evidence reuse, repeated timeout diagnosis, and fresh CI after source/base + changes, with explicit limits on what the cache optimization can improve. +- This standalone Commitrail record. + +### Not allowed + +- No product source, migrations, dependencies, test catalogue/partition, lane + count, test selection, skips, coverage behavior, timeout, retry, runner + permission, concurrency, service image or branch-protection change. +- No installed virtual-environment cache, broad restore prefix, commit-SHA MinIO + cache key, unverified image reuse, affected-only tests or hidden failures. +- No lane/DAG redesign or performance/capacity claim before hosted measurement. + +## Design and decisions + +The reusable MinIO tar is a deterministic product of the pinned Docker context +and runner platform. Its cache key therefore uses only a version, runner OS and +architecture, and `hashFiles('docker/minio/**')`. The cache has no restore +prefix. Every workflow attempt still publishes a SHA-and-attempt-named artifact +after loading the tar, checking the image version, starting it, probing health, +and producing a checksum that every consumer verifies. + +`actions/setup-python` owns a pip download cache keyed by Python/platform and +the hashes of `backend/pyproject.toml` plus this workflow, which includes the +separate exact Ruff pin. It does not cache an installed environment. Existing +`pip install` commands remain mandatory in preflight, every lane and aggregation. + +The lane runner changes `--durations=25` to `--durations=0`; pytest retains all +nontrivial setup/call/teardown timings in the existing authenticated log. A lane +terminated by the unchanged deadline may not emit the final duration summary, +so completion evidence and interruption metadata remain the authority. + +The operator guidance treats rerun as evidence recovery, never a green-result +loop. Removing the observed 145-second rebuild when its exact context is +unchanged does not solve 17-minute queue waits or 20-minute lanes and does not +promise an eight-minute Backend completion time. + +## Acceptance criteria + +- [ ] Unchanged MinIO Docker context on the same runner OS/architecture uses one + exact cache key across commits; any context or platform change misses it. +- [ ] MinIO cache restore has no broad prefix and cannot bypass image load, + version execution, live health probe, per-run artifact name or checksum. +- [ ] Backend jobs cache pip downloads only and still run the existing fresh + package installation commands from exact dependency inputs. +- [ ] Ordinary and schema-admin lane commands request complete duration + diagnostics while retaining the same exact node list, coverage, isolation, + timeout, log and evidence contracts. +- [ ] The nine lanes, full inventory, fan-in failure propagation, CLI dependency, + aggregate validation, permissions and timeouts are unchanged. +- [ ] Contributor guidance permits only failed-job reruns for a diagnosed + same-head transient, explains aggregate revalidation of successful lane + evidence, and requires diagnosis or fresh current-tree CI in the other cases. +- [ ] Focused workflow/runner tests and mutation probes fail when cache identity, + verification, fresh installation, duration output or required gates weaken. + +## Risk and review routing + +- Risk class: `L1` +- Required reviewers: `ci_integrity`, `qa`, `test_delta` +- Human review focus: Cross-commit cache trust and invalidation, fresh-install + preservation, exact artifact/runtime verification, unchanged gate propagation, + and whether duration output remains diagnostic rather than authoritative. + +## Evidence + +| Claim | Command or proof | Result | Remaining uncertainty | +|---|---|---|---| +| Measured bottlenecks | Inspect PR #507 Backend run `37784941972` job/step timestamps from `/tmp/ws-ci-507-jobs.json` | MinIO build took 145 seconds; repeated lane installs took about 25--28 seconds | Hosted queue and runner variability are uncontrolled | +| Cache custody | Focused lightweight workflow tests plus cache-key/runtime-verification mutation probes | Planned | A real cross-commit hit requires hosted runs | +| Lane diagnostics | Focused lane-command tests plus duration-option mutation probe | Planned | A timed-out pytest process cannot print its final duration table | +| Full gate preservation | Workflow inventory/fan-in tests and exact diff inspection | Planned | Root owns hosted aggregate validation | + +## Review findings + +None yet. + +## Reconciliation + +- Current-source reconciliation: Based on `main` at + `72b83ffc2f0fcb29efdc68f6babaee884ab7d4dd`; no product or migration owner is + affected. +- Next usable boundary: Measure fresh and cross-commit hosted runs before any + lane rebalance or further DAG change. +- Remaining risks: GitHub-hosted cache availability and queue time vary outside + repository control; no improvement is claimed until measured. From 7e4ed8864258019fc6a10d7db2acc2249b4d7158 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 15:52:47 +0100 Subject: [PATCH 2/5] ci: cache trusted Backend build inputs --- .../changes/backend-ci-cache-and-timing.md | 88 +++++++++++-------- .github/workflows/backend.yml | 42 ++++++++- CONTRIBUTING.md | 3 + backend/scripts/run_test_lanes.py | 4 +- backend/tests/test_ci_test_lanes.py | 6 ++ docs/operations_backend_testing.md | 20 +++-- scripts/test_lightweight_agent_gates.py | 77 +++++++++++++++- 7 files changed, 193 insertions(+), 47 deletions(-) diff --git a/.commitrail/changes/backend-ci-cache-and-timing.md b/.commitrail/changes/backend-ci-cache-and-timing.md index 00a3cac99..48ee2da11 100644 --- a/.commitrail/changes/backend-ci-cache-and-timing.md +++ b/.commitrail/changes/backend-ci-cache-and-timing.md @@ -1,7 +1,7 @@ # Backend CI Cache And Timing Diagnostics - Initiative: `None` -- Durable disposition: `Planned` +- Durable disposition: `Complete` - Intended merge outcome: Backend CI reuses an exact content-addressed MinIO source image and pip download cache while retaining every semantic test, fail-closed aggregate, and useful per-test duration evidence. @@ -10,7 +10,7 @@ Reduce repeated setup work in the required Backend workflow and retain enough timing evidence to rebalance slow lanes from measurements. This change targets -only deterministic downloads/build inputs and diagnostics; it does not trade +only exact cache-identity inputs and diagnostics; it does not trade test or evidence coverage for a faster green result. ## Current behavior @@ -32,20 +32,25 @@ run only; this record makes no hosted performance claim before a candidate run. ### Allowed - `.github/workflows/backend.yml`: make the MinIO tar cache key depend on the - runner OS/architecture and exact `docker/minio/**` content, retaining exact - per-run artifact identity, checksums and live provider verification. -- `.github/workflows/backend.yml`: enable only setup-python's pip download cache - for the exact backend dependency/workflow inputs; every job still performs a - fresh install into its fresh hosted Python environment. + runner OS/architecture, exact `docker/minio/**` content and workflow recipe, + retaining exact per-run artifact identity, checksums and live provider + verification. +- `.github/workflows/backend.yml`: restore a pip download cache for the exact + backend dependency/workflow inputs and save it only after trusted-main + validation; every job still performs a fresh install into its fresh hosted + Python environment. - `backend/scripts/run_test_lanes.py`: retain all pytest duration diagnostics in - the already authenticated per-lane logs without changing selection or timeout. + the existing redacted per-lane diagnostic logs without changing selection or + timeout or treating those logs as fan-in evidence. - `scripts/test_lightweight_agent_gates.py` and `backend/tests/test_ci_test_lanes.py`: focused positive and adversarial regressions for cache identity, fresh installs, runtime verification, artifact identity, and lane duration options. -- `CONTRIBUTING.md`: concise operator guidance for failed-job reruns, same-head - evidence reuse, repeated timeout diagnosis, and fresh CI after source/base - changes, with explicit limits on what the cache optimization can improve. +- `docs/operations_backend_testing.md`: extend the canonical failure/rerun owner + with the exact failed-job command, repeated-timeout diagnosis, fresh-current- + tree rule and explicit limits on what the cache optimization can improve. +- `CONTRIBUTING.md`: link contributors to that canonical Backend rerun guidance + without copying its operational contract. - This standalone Commitrail record. ### Not allowed @@ -59,20 +64,25 @@ run only; this record makes no hosted performance claim before a candidate run. ## Design and decisions -The reusable MinIO tar is a deterministic product of the pinned Docker context -and runner platform. Its cache key therefore uses only a version, runner OS and -architecture, and `hashFiles('docker/minio/**')`. The cache has no restore -prefix. Every workflow attempt still publishes a SHA-and-attempt-named artifact -after loading the tar, checking the image version, starting it, probing health, -and producing a checksum that every consumer verifies. - -`actions/setup-python` owns a pip download cache keyed by Python/platform and -the hashes of `backend/pyproject.toml` plus this workflow, which includes the -separate exact Ruff pin. It does not cache an installed environment. Existing -`pip install` commands remain mandatory in preflight, every lane and aggregation. +The reusable MinIO tar has an exact recipe/platform cache identity: a version, +runner OS and architecture, and a hash of `docker/minio/**` plus the workflow +that owns the build command. Rare workflow-only edits intentionally invalidate +the cache. A cold rebuild is not guaranteed byte-identical because the Docker +recipe consumes live apt repositories. Pull requests can restore only; a cache +miss is saved only after a trusted `main` push verifies the built image. The +cache has no restore prefix. Every workflow attempt still publishes a +SHA-and-attempt-named artifact after loading the tar, checking the image version, +starting it, probing health, and producing a checksum that every consumer verifies. + +Pinned cache restore/save actions manage a pip download directory keyed by +Python/platform and the hashes of `backend/pyproject.toml` plus this workflow, +which includes the separate exact Ruff pin. Pull requests restore only; the +validated preflight on a trusted `main` push may save a miss. No installed +environment is cached. Existing `pip install` commands remain mandatory in +preflight, every lane and aggregation. The lane runner changes `--durations=25` to `--durations=0`; pytest retains all -nontrivial setup/call/teardown timings in the existing authenticated log. A lane +nontrivial setup/call/teardown timings in the existing diagnostic log. A lane terminated by the unchanged deadline may not emit the final duration summary, so completion evidence and interruption metadata remain the authority. @@ -83,21 +93,24 @@ promise an eight-minute Backend completion time. ## Acceptance criteria -- [ ] Unchanged MinIO Docker context on the same runner OS/architecture uses one - exact cache key across commits; any context or platform change misses it. -- [ ] MinIO cache restore has no broad prefix and cannot bypass image load, +- [x] Unchanged MinIO Docker context and Backend build workflow on the same + runner OS/architecture use one exact cache key across commits; any context, + workflow or platform change misses it. +- [x] MinIO cache restore has no broad prefix and cannot bypass image load, version execution, live health probe, per-run artifact name or checksum. -- [ ] Backend jobs cache pip downloads only and still run the existing fresh +- [x] Pull requests never save MinIO or pip caches; only a verified trusted-main + miss can populate a key that later pull requests restore. +- [x] Backend jobs cache pip downloads only and still run the existing fresh package installation commands from exact dependency inputs. -- [ ] Ordinary and schema-admin lane commands request complete duration +- [x] Ordinary and schema-admin lane commands request complete duration diagnostics while retaining the same exact node list, coverage, isolation, timeout, log and evidence contracts. -- [ ] The nine lanes, full inventory, fan-in failure propagation, CLI dependency, +- [x] The nine lanes, full inventory, fan-in failure propagation, CLI dependency, aggregate validation, permissions and timeouts are unchanged. -- [ ] Contributor guidance permits only failed-job reruns for a diagnosed +- [x] Contributor guidance permits only failed-job reruns for a diagnosed same-head transient, explains aggregate revalidation of successful lane evidence, and requires diagnosis or fresh current-tree CI in the other cases. -- [ ] Focused workflow/runner tests and mutation probes fail when cache identity, +- [x] Focused workflow/runner tests and mutation probes fail when cache identity, verification, fresh installation, duration output or required gates weaken. ## Risk and review routing @@ -113,13 +126,18 @@ promise an eight-minute Backend completion time. | Claim | Command or proof | Result | Remaining uncertainty | |---|---|---|---| | Measured bottlenecks | Inspect PR #507 Backend run `37784941972` job/step timestamps from `/tmp/ws-ci-507-jobs.json` | MinIO build took 145 seconds; repeated lane installs took about 25--28 seconds | Hosted queue and runner variability are uncontrolled | -| Cache custody | Focused lightweight workflow tests plus cache-key/runtime-verification mutation probes | Planned | A real cross-commit hit requires hosted runs | -| Lane diagnostics | Focused lane-command tests plus duration-option mutation probe | Planned | A timed-out pytest process cannot print its final duration table | -| Full gate preservation | Workflow inventory/fan-in tests and exact diff inspection | Planned | Root owns hosted aggregate validation | +| Cache custody | `python -m unittest -v scripts.test_lightweight_agent_gates`; checksum-verified actionlint 1.7.7; cache-key and invalid-context mutants | 19 workflow gates passed; candidate linted cleanly; both mutants were rejected | A real cross-commit hit requires a trusted-main seed and later hosted run | +| Lane diagnostics | Focused lane/evidence pytest batch plus duration-option mutant | 123 tests passed; reverting both commands to `--durations=25` failed the exact lane-command regression | A timed-out pytest process cannot print its final duration table | +| Full gate preservation | Workflow inventory/fan-in tests, exact diff inspection, Commitrail and Markdown gates | Existing lane count, selection, timeouts, permissions, fan-in and fresh installs remain; repository gates passed | Root owns hosted aggregate validation | ## Review findings -None yet. +Initial workflow review found that `runner.temp` is not an allowed workflow-level +`env` context. Each existing identity step now exports `PIP_CACHE_DIR` through +`GITHUB_ENV` before installation; actionlint and an invalid-context mutant prove +that regression is rejected. Cache trust review also changed both caches to +restore-only on pull requests and save-on-miss only after trusted-main +verification. ## Reconciliation diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index fe9c656e2..1acacc19c 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -36,12 +36,12 @@ jobs: test "$(git rev-parse HEAD)" = "${GITHUB_SHA}" echo "artifact=minio-source-${GITHUB_SHA}-${GITHUB_RUN_ATTEMPT}" >> "${GITHUB_OUTPUT}" - - name: Cache exact source image + - name: Restore exact source image from trusted cache id: cache - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 with: path: ${{ runner.temp }}/minio-image/minio.tar - key: minio-source-v1-${{ github.sha }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('docker/minio/**') }} + key: minio-source-v2-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('docker/minio/**', '.github/workflows/backend.yml') }} - name: Build source-pinned MinIO once if: steps.cache.outputs.cache-hit != 'true' @@ -74,6 +74,13 @@ jobs: done exit 1 + - name: Save verified source image to trusted cache + if: github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.cache.outputs.cache-hit != 'true' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/minio-image/minio.tar + key: minio-source-v2-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('docker/minio/**', '.github/workflows/backend.yml') }} + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 with: name: ${{ steps.identity.outputs.artifact }} @@ -96,6 +103,13 @@ jobs: with: python-version: "3.12" + - name: Restore exact backend pip downloads from trusted cache + id: pip-cache + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/backend-pip-cache + key: backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }} + - name: Bind exact checked-out tree shell: bash run: | @@ -103,6 +117,7 @@ jobs: test "$(git rev-parse HEAD)" = "${GITHUB_SHA}" test -z "$(git status --porcelain)" test "$(git rev-parse "${GITHUB_SHA}^{tree}")" = "$(git rev-parse HEAD^{tree})" + echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}" - name: Install backend architecture dependencies working-directory: backend @@ -139,6 +154,13 @@ jobs: --ledger ../.ci/auth-boundaries/TEST_STRUCTURE_DEBT.json python -m scripts.behavior_ownership validate + - name: Save verified backend pip downloads to trusted cache + if: github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.pip-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/backend-pip-cache + key: backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }} + lanes: needs: minio-image runs-on: ubuntu-latest @@ -192,6 +214,12 @@ jobs: with: python-version: "3.12" + - name: Restore exact backend pip downloads from trusted cache + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/backend-pip-cache + key: backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }} + - id: identity name: Bind exact checked-out tree shell: bash @@ -204,6 +232,7 @@ jobs: test "$(git rev-parse "${tree_sha}^{tree}")" = "$(git rev-parse HEAD^{tree})" echo "job_start_epoch=${job_start_epoch}" >> "${GITHUB_OUTPUT}" echo "tree_sha=${tree_sha}" >> "${GITHUB_OUTPUT}" + echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}" - name: Install backend and exact Ruff working-directory: backend @@ -358,6 +387,12 @@ jobs: with: python-version: "3.12" + - name: Restore exact backend pip downloads from trusted cache + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/backend-pip-cache + key: backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }} + - id: identity name: Bind exact checked-out tree shell: bash @@ -369,6 +404,7 @@ jobs: test -z "$(git status --porcelain)" echo "job_start_epoch=${job_start_epoch}" >> "${GITHUB_OUTPUT}" echo "tree_sha=${tree_sha}" >> "${GITHUB_OUTPUT}" + echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}" - name: Install backend working-directory: backend diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f237e24cc..85341f0a7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -149,6 +149,9 @@ Different initiatives may proceed concurrently in separate branches or worktrees. If another pull request changes the base, inspect the new delta and rerun affected checks; unchanged evidence does not need ceremonial repetition. +For Backend failure diagnosis and safe partial reruns, follow the canonical +[Backend testing operations guide](docs/operations_backend_testing.md#failure-diagnosis-and-reruns). + ## Behavior Ownership Catalogue The hosted behavior-mutation check is temporarily retired because its diff --git a/backend/scripts/run_test_lanes.py b/backend/scripts/run_test_lanes.py index 644416e2a..79b394e1b 100644 --- a/backend/scripts/run_test_lanes.py +++ b/backend/scripts/run_test_lanes.py @@ -496,7 +496,7 @@ def lane_command( *_plugin_args(), "--cov=app", "--cov-report=", - "--durations=25", + "--durations=0", *nodes, ] return [ @@ -525,7 +525,7 @@ def admin_runner_command(nodes: list[str]) -> list[str]: *_plugin_args(), "--cov=app", "--cov-report=", - "--durations=25", + "--durations=0", *nodes, ] return [sys.executable, "-c", ADMIN_REDACTING_WRAPPER, *pytest_command] diff --git a/backend/tests/test_ci_test_lanes.py b/backend/tests/test_ci_test_lanes.py index 18726aa0f..65cfb1e3b 100644 --- a/backend/tests/test_ci_test_lanes.py +++ b/backend/tests/test_ci_test_lanes.py @@ -98,6 +98,12 @@ def test_lane_command_uses_exact_nodes_and_isolation_contract(tmp_path: Path) -> assert command[-1] == nodes[0] assert "--cov=app" in command assert "--cov-report=" in command + assert "--durations=0" in command + assert "--durations=25" not in command + + admin = runner.admin_runner_command([f"{runner.ADMIN_RUNNER_MODULE}::test_one"]) + assert "--durations=0" in admin + assert "--durations=25" not in admin def test_lane_environment_uses_private_evidence_and_coverage(tmp_path: Path) -> None: diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index ab85af6f4..663905209 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -220,11 +220,14 @@ coverage tampering before coverage combination. - API contract or evidence-integrity failure: the required job remains failed; lane completion cannot compensate. A lower coverage percentage is not a failure. -On the same exact head, rerun failed lanes (and their dependent final job), or -rerun only the final job when the lane evidence already passed. Successful lanes -not rerun retain their previous attempt's evidence; a rerun lane's newest bundle -must independently pass all existing checks. A failed, cancelled or skipped -required job still blocks fan-in. Never edit or upload evidence manually. +After diagnosing a transient failure on the same exact head, rerun only failed +jobs and their dependents with `gh run rerun RUN_ID --failed`; do not use +`--all`. Successful lanes not rerun retain their previous attempt's evidence, +while a rerun lane's newest bundle must independently pass all existing checks. +The required aggregate job selects those bundles and revalidates the complete +exact-head union. A failed, cancelled or skipped required job still blocks +fan-in. Never edit or upload evidence manually. A repeated timeout requires +diagnosis rather than retries until green. Review submission or dismissal does not rerun Backend because it does not change the tested tree. A new PR commit starts a new run and cancels the superseded same-PR run. Every new commit requires complete evidence because @@ -240,6 +243,13 @@ does not override otherwise passing correctness, custody, service-contract, API, and complete-execution gates. Coverage is diagnostic only. Never skip nodes or add a silent fallback to meet the target. +A source push or base-branch change creates a different current tree and +requires fresh CI; prior same-head retry evidence cannot be carried forward. +The content-addressed MinIO cache can remove the measured 145-second image +rebuild when its exact Docker context, Backend workflow and runner platform are +unchanged. It does not solve observed 17-minute queue waits or 20-minute lanes +and does not promise an eight-minute Backend completion time. + ## Retired changed-scope behavior mutation The hosted `Behavior Mutation Gate` is temporarily removed. Its callable-wide diff --git a/scripts/test_lightweight_agent_gates.py b/scripts/test_lightweight_agent_gates.py index 493abbfe5..c594ee6b4 100644 --- a/scripts/test_lightweight_agent_gates.py +++ b/scripts/test_lightweight_agent_gates.py @@ -228,12 +228,39 @@ def test_minio_source_image_is_built_once_and_shared_without_bypassing_lanes(sel image_job = workflow.split("\n minio-image:\n", 1)[1].split("\n auth-boundary-preflight:\n", 1)[0] self.assertEqual(workflow.count('docker build --tag "${MINIO_IMAGE}" docker/minio'), 1) self.assertNotIn("quay.io/minio", workflow) - self.assertIn("hashFiles('docker/minio/**')", image_job) + self.assertEqual( + image_job.count("hashFiles('docker/minio/**', '.github/workflows/backend.yml')"), + 2, + ) self.assertIn( - "key: minio-source-v1-${{ github.sha }}-${{ runner.os }}-${{ runner.arch }}-", + "key: minio-source-v2-${{ runner.os }}-${{ runner.arch }}-", image_job, ) + cache_step = image_job.split( + " - name: Restore exact source image from trusted cache\n", 1 + )[1].split("\n - name:", 1)[0] + self.assertIn( + "uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830", + cache_step, + ) + self.assertNotIn("github.sha", cache_step) self.assertNotIn("restore-keys:", image_job) + save_step = image_job.split( + " - name: Save verified source image to trusted cache\n", 1 + )[1].split("\n - uses:", 1)[0] + self.assertIn( + "if: github.event_name == 'push' && github.ref == 'refs/heads/main' " + "&& steps.cache.outputs.cache-hit != 'true'", + save_step, + ) + self.assertIn( + "uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830", + save_step, + ) + self.assertLess( + image_job.index(" - name: Verify the cached or freshly built provider"), + image_job.index(" - name: Save verified source image to trusted cache"), + ) self.assertIn("minio-source-${GITHUB_SHA}-${GITHUB_RUN_ATTEMPT}", image_job) self.assertIn("artifact: ${{ steps.identity.outputs.artifact }}", image_job) self.assertIn("sha256sum minio.tar > minio.tar.sha256", image_job) @@ -249,6 +276,52 @@ def test_minio_source_image_is_built_once_and_shared_without_bypassing_lanes(sel self.assertIn('docker load --input "${RUNNER_TEMP}/minio-image/minio.tar"', job) self.assertIn('"${MINIO_IMAGE}" server /data --address :9000', job) + def test_backend_python_cache_keeps_fresh_installs_and_exact_inputs(self) -> None: + workflow = Path(".github/workflows/backend.yml").read_text(encoding="utf-8") + + key = ( + "backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-" + "${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }}" + ) + self.assertEqual(workflow.count("Restore exact backend pip downloads from trusted cache"), 3) + self.assertEqual(workflow.count(f"key: {key}"), 4) + self.assertEqual( + workflow.count( + "uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830" + ), + 4, + ) + self.assertEqual( + workflow.count("uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830"), + 2, + ) + self.assertEqual(workflow.count('python -m pip install -e ".[dev,agents]"'), 3) + self.assertEqual( + workflow.count( + 'echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}"' + ), + 3, + ) + root_env = workflow.split("\nenv:\n", 1)[1].split("\njobs:\n", 1)[0] + self.assertNotIn("runner.", root_env) + pip_save = workflow.split( + " - name: Save verified backend pip downloads to trusted cache\n", 1 + )[1].split("\n lanes:\n", 1)[0] + self.assertIn( + "if: github.event_name == 'push' && github.ref == 'refs/heads/main' " + "&& steps.pip-cache.outputs.cache-hit != 'true'", + pip_save, + ) + preflight = workflow.split("\n auth-boundary-preflight:\n", 1)[1].split( + "\n lanes:\n", 1 + )[0] + self.assertLess( + preflight.index(" - name: Validate module, AUTH, and test boundaries"), + preflight.index(" - name: Save verified backend pip downloads to trusted cache"), + ) + self.assertNotIn("restore-keys:", workflow) + self.assertNotIn(".venv", workflow) + def test_parallel_preflight_and_lanes_fail_closed_at_fan_in(self) -> None: workflow = Path(".github/workflows/backend.yml").read_text(encoding="utf-8") lanes = workflow.split("\n lanes:\n", 1)[1].split("\n test:\n", 1)[0] From cba95bbe7ae04b34a638f316c6abc1c285f5b6de Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 15:58:39 +0100 Subject: [PATCH 3/5] test: tighten CI cache trust checks --- .../changes/backend-ci-cache-and-timing.md | 15 +++++++++++-- docker/minio/README.md | 21 +++++++++++------- docs/operations_backend_testing.md | 14 ++++++------ scripts/test_lightweight_agent_gates.py | 22 +++++++++++++------ 4 files changed, 48 insertions(+), 24 deletions(-) diff --git a/.commitrail/changes/backend-ci-cache-and-timing.md b/.commitrail/changes/backend-ci-cache-and-timing.md index 48ee2da11..effe3891d 100644 --- a/.commitrail/changes/backend-ci-cache-and-timing.md +++ b/.commitrail/changes/backend-ci-cache-and-timing.md @@ -49,6 +49,8 @@ run only; this record makes no hosted performance claim before a candidate run. - `docs/operations_backend_testing.md`: extend the canonical failure/rerun owner with the exact failed-job command, repeated-timeout diagnosis, fresh-current- tree rule and explicit limits on what the cache optimization can improve. +- `docker/minio/README.md`: describe trusted-main cache publication, exact + content/workflow/platform reuse and the separate per-run artifact binding. - `CONTRIBUTING.md`: link contributors to that canonical Backend rerun guidance without copying its operational contract. - This standalone Commitrail record. @@ -116,7 +118,7 @@ promise an eight-minute Backend completion time. ## Risk and review routing - Risk class: `L1` -- Required reviewers: `ci_integrity`, `qa`, `test_delta` +- Required reviewers: `ci_integrity`, `security`, `qa`, `test_delta` - Human review focus: Cross-commit cache trust and invalidation, fresh-install preservation, exact artifact/runtime verification, unchanged gate propagation, and whether duration output remains diagnostic rather than authoritative. @@ -126,7 +128,7 @@ promise an eight-minute Backend completion time. | Claim | Command or proof | Result | Remaining uncertainty | |---|---|---|---| | Measured bottlenecks | Inspect PR #507 Backend run `37784941972` job/step timestamps from `/tmp/ws-ci-507-jobs.json` | MinIO build took 145 seconds; repeated lane installs took about 25--28 seconds | Hosted queue and runner variability are uncontrolled | -| Cache custody | `python -m unittest -v scripts.test_lightweight_agent_gates`; checksum-verified actionlint 1.7.7; cache-key and invalid-context mutants | 19 workflow gates passed; candidate linted cleanly; both mutants were rejected | A real cross-commit hit requires a trusted-main seed and later hosted run | +| Cache custody | `python -m unittest -v scripts.test_lightweight_agent_gates`; checksum-verified actionlint 1.7.7; cache-key, invalid-context and per-job-install relocation mutants | 19 workflow gates passed; candidate linted cleanly; all three cache/install mutants were rejected | A real cross-commit hit requires a trusted-main seed and later hosted run | | Lane diagnostics | Focused lane/evidence pytest batch plus duration-option mutant | 123 tests passed; reverting both commands to `--durations=25` failed the exact lane-command regression | A timed-out pytest process cannot print its final duration table | | Full gate preservation | Workflow inventory/fan-in tests, exact diff inspection, Commitrail and Markdown gates | Existing lane count, selection, timeouts, permissions, fan-in and fresh installs remain; repository gates passed | Root owns hosted aggregate validation | @@ -139,6 +141,15 @@ that regression is rejected. Cache trust review also changed both caches to restore-only on pull requests and save-on-miss only after trusted-main verification. +Documentation review also corrected the `gh` command description: `--failed` +is the selective form, while the bare command is the whole-workflow rerun; there +is no `--all` option. The MinIO owner documentation now distinguishes the +trusted cross-commit cache from each SHA-and-attempt-bound artifact. +QA review found that a repository-wide install count could miss moving the +aggregate install into preflight. The regression now inspects preflight, lane +and aggregate blocks independently and requires exactly one cache restore, +environment export and fresh install in order; the relocation mutant fails it. + ## Reconciliation - Current-source reconciliation: Based on `main` at diff --git a/docker/minio/README.md b/docker/minio/README.md index fc2dad15e..6ad9fabf7 100644 --- a/docker/minio/README.md +++ b/docker/minio/README.md @@ -28,14 +28,19 @@ requires network access and Go compilation resources; do not launch it on an already memory-constrained workstation. Subsequent builds reuse Docker layers. -Backend CI builds or restores one image cache keyed by the exact Git commit, -this directory's contents and runner platform. Older PR commits cannot supply a -cached executable to a new commit; retries of the same commit can reuse its -image. CI verifies server startup, then supplies a checksummed image -artifact to the existing lanes and aggregate job. Jobs never substitute a mock -storage provider. A missing build, artifact or health check fails verification. -The source-image artifact is independent of test/coverage evidence and cannot -make a failed test lane pass. +Backend CI builds or restores one image tar cache keyed by a cache version, the +runner OS/architecture, this directory's contents and the Backend workflow that +owns the build command. Pull requests restore only. A trusted `main` push saves +a miss after the image passes its version command and live health probe. The +same exact cache identity can therefore be reused across commits; a Docker +context, workflow or platform change misses it. + +Every run still loads and probes the image, then supplies a newly checksummed, +Git-SHA-and-attempt-named artifact to the existing lanes and aggregate job. +Those consumers verify the checksum before loading it. Jobs never substitute a +mock storage provider. A missing build, artifact or health check fails +verification. The source-image artifact is independent of test/coverage +evidence and cannot make a failed test lane pass. When updating upstream source, update the commit, archive checksum, provenance and relevant build pins together. Require a fresh image build, health check and diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index 663905209..9d9b51d0e 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -221,13 +221,13 @@ coverage tampering before coverage combination. lane completion cannot compensate. A lower coverage percentage is not a failure. After diagnosing a transient failure on the same exact head, rerun only failed -jobs and their dependents with `gh run rerun RUN_ID --failed`; do not use -`--all`. Successful lanes not rerun retain their previous attempt's evidence, -while a rerun lane's newest bundle must independently pass all existing checks. -The required aggregate job selects those bundles and revalidates the complete -exact-head union. A failed, cancelled or skipped required job still blocks -fan-in. Never edit or upload evidence manually. A repeated timeout requires -diagnosis rather than retries until green. +jobs and their dependents with `gh run rerun RUN_ID --failed`; avoid the bare +whole-workflow rerun command for that case. Successful lanes not rerun retain +their previous attempt's evidence, while a rerun lane's newest bundle must +independently pass all existing checks. The required aggregate job selects those +bundles and revalidates the complete exact-head union. A failed, cancelled or +skipped required job still blocks fan-in. Never edit or upload evidence +manually. A repeated timeout requires diagnosis rather than retries until green. Review submission or dismissal does not rerun Backend because it does not change the tested tree. A new PR commit starts a new run and cancels the superseded same-PR run. Every new commit requires complete evidence because diff --git a/scripts/test_lightweight_agent_gates.py b/scripts/test_lightweight_agent_gates.py index c594ee6b4..6c0b672e1 100644 --- a/scripts/test_lightweight_agent_gates.py +++ b/scripts/test_lightweight_agent_gates.py @@ -295,13 +295,21 @@ def test_backend_python_cache_keeps_fresh_installs_and_exact_inputs(self) -> Non workflow.count("uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830"), 2, ) - self.assertEqual(workflow.count('python -m pip install -e ".[dev,agents]"'), 3) - self.assertEqual( - workflow.count( - 'echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}"' - ), - 3, - ) + install = 'python -m pip install -e ".[dev,agents]"' + cache_env = 'echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}"' + job_blocks = ( + workflow.split("\n auth-boundary-preflight:\n", 1)[1].split("\n lanes:\n", 1)[0], + workflow.split("\n lanes:\n", 1)[1].split("\n cli-public-contract:\n", 1)[0], + workflow.split("\n test:\n", 1)[1], + ) + for job in job_blocks: + self.assertEqual(job.count("Restore exact backend pip downloads from trusted cache"), 1) + self.assertEqual(job.count(install), 1) + self.assertEqual(job.count(cache_env), 1) + self.assertLess(job.index("Restore exact backend pip downloads"), job.index(cache_env)) + self.assertLess(job.index(cache_env), job.index(install)) + self.assertEqual(workflow.count(install), 3) + self.assertEqual(workflow.count(cache_env), 3) root_env = workflow.split("\nenv:\n", 1)[1].split("\njobs:\n", 1)[0] self.assertNotIn("runner.", root_env) pip_save = workflow.split( From c175ea93b2f3ba61a01228c455f2394cae120b83 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 17:10:07 +0100 Subject: [PATCH 4/5] fix(ci): rebalance measured checker delivery workloads --- .../changes/backend-ci-cache-and-timing.md | 52 +++++++++--- backend/scripts/test_lane_catalogue.py | 55 ++++++------ backend/tests/test_ci_lane_catalogue.py | 85 +++++++++++++------ docs/operations_backend_testing.md | 8 +- 4 files changed, 133 insertions(+), 67 deletions(-) diff --git a/.commitrail/changes/backend-ci-cache-and-timing.md b/.commitrail/changes/backend-ci-cache-and-timing.md index effe3891d..0da957738 100644 --- a/.commitrail/changes/backend-ci-cache-and-timing.md +++ b/.commitrail/changes/backend-ci-cache-and-timing.md @@ -1,17 +1,16 @@ -# Backend CI Cache And Timing Diagnostics +# Backend CI Caches And Measured Lane Balance - Initiative: `None` - Durable disposition: `Complete` - Intended merge outcome: Backend CI reuses an exact content-addressed MinIO - source image and pip download cache while retaining every semantic test, - fail-closed aggregate, and useful per-test duration evidence. + source image and pip download cache, moves measured workloads into existing + lane capacity, and retains every test and the fail-closed aggregate. ## Intent -Reduce repeated setup work in the required Backend workflow and retain enough -timing evidence to rebalance slow lanes from measurements. This change targets -only exact cache-identity inputs and diagnostics; it does not trade -test or evidence coverage for a faster green result. +Reduce repeated setup work and repair measured task-lane overload in the required +Backend workflow. Exact cache inputs, retained duration diagnostics and explicit +workload placement preserve complete test and evidence coverage. ## Current behavior @@ -42,6 +41,10 @@ run only; this record makes no hosted performance claim before a candidate run. - `backend/scripts/run_test_lanes.py`: retain all pytest duration diagnostics in the existing redacted per-lane diagnostic logs without changing selection or timeout or treating those logs as fan-in evidence. +- `backend/scripts/test_lane_catalogue.py` and + `backend/tests/test_ci_lane_catalogue.py`: rebalance measured checker-delivery + and routing-preparation workloads into existing project-lane capacity; + preserve every test node, deterministic assignment and all nine lanes. - `scripts/test_lightweight_agent_gates.py` and `backend/tests/test_ci_test_lanes.py`: focused positive and adversarial regressions for cache identity, fresh installs, runtime verification, artifact @@ -57,12 +60,30 @@ run only; this record makes no hosted performance claim before a candidate run. ### Not allowed -- No product source, migrations, dependencies, test catalogue/partition, lane +- No product source, migrations, dependencies, lane count, test selection, skips, coverage behavior, timeout, retry, runner permission, concurrency, service image or branch-protection change. - No installed virtual-environment cache, broad restore prefix, commit-SHA MinIO cache key, unverified image reuse, affected-only tests or hidden failures. -- No lane/DAG redesign or performance/capacity claim before hosted measurement. +- No lane-count/DAG redesign or performance claim before hosted measurement. + +### Timeout repair plan + +Backend run `37799550116` tested merge `9ea0fc9a2f9b0f9927ae98f06f9ea0a487daa050`. +TASK B and C exhausted 1200 seconds with 22 and 30 nodes unfinished. Their +databases and MinIO resources were cleaned up; the aggregate correctly failed. +PROJECT A completed in 529 seconds, PROJECT B in 959 and PROJECT C in 758; +TASK A completed in 969. This demonstrates available capacity in the existing +jobs, not that a same-head retry would solve the imbalance. + +Move checker execution, post-submit materialization/selection, evaluation +capacity and output-custody/storage proof together to PROJECT A, beside the +existing initial-dispatch and evaluation-delivery owners. Move routing AUTH +PREP proof from TASK C to PROJECT C. Keep the original per-node hash mechanism, +UUID seed, node IDs, isolation, 1200-second limits and exact aggregate custody. +Use catalogue/inventory regressions and a dropped/duplicated-owner mutant +before independent CI-integrity and QA review. The recorded durations guide +this allocation; only a fresh hosted run can establish runtime success. ## Design and decisions @@ -108,7 +129,8 @@ promise an eight-minute Backend completion time. diagnostics while retaining the same exact node list, coverage, isolation, timeout, log and evidence contracts. - [x] The nine lanes, full inventory, fan-in failure propagation, CLI dependency, - aggregate validation, permissions and timeouts are unchanged. + aggregate validation, permissions and timeouts are unchanged; only the named + checker-delivery and routing-preparation workloads change lane ownership. - [x] Contributor guidance permits only failed-job reruns for a diagnosed same-head transient, explains aggregate revalidation of successful lane evidence, and requires diagnosis or fresh current-tree CI in the other cases. @@ -131,6 +153,8 @@ promise an eight-minute Backend completion time. | Cache custody | `python -m unittest -v scripts.test_lightweight_agent_gates`; checksum-verified actionlint 1.7.7; cache-key, invalid-context and per-job-install relocation mutants | 19 workflow gates passed; candidate linted cleanly; all three cache/install mutants were rejected | A real cross-commit hit requires a trusted-main seed and later hosted run | | Lane diagnostics | Focused lane/evidence pytest batch plus duration-option mutant | 123 tests passed; reverting both commands to `--durations=25` failed the exact lane-command regression | A timed-out pytest process cannot print its final duration table | | Full gate preservation | Workflow inventory/fan-in tests, exact diff inspection, Commitrail and Markdown gates | Existing lane count, selection, timeouts, permissions, fan-in and fresh installs remain; repository gates passed | Root owns hosted aggregate validation | +| Timeout diagnosis and allocation | Retained summaries and complete TASK A duration phases from Backend run `37799550116` | TASK B/C interrupted at 1200 seconds; delivery-group transfer forecasts about 152/161 seconds of relief and about 1036 seconds for PROJECT A | Forecast extrapolates TASK A module costs; fresh hosted execution is required | +| Rebalance integrity | Catalogue/runner/evidence/merge focused pytest batch; dropped and duplicated delivery-owner mutations | 124 tests passed; recursive inventory rejected both mutations; all node IDs and existing hash/UUID-seed mechanisms remain | This proves assignment and evidence contracts, not hosted speed | ## Review findings @@ -152,10 +176,10 @@ environment export and fresh install in order; the relocation mutant fails it. ## Reconciliation -- Current-source reconciliation: Based on `main` at - `72b83ffc2f0fcb29efdc68f6babaee884ab7d4dd`; no product or migration owner is +- Current-source reconciliation: Reconciled with `main` at + `b169e83f816bba417fc0618a6e2d419acb2f94ed`; no product or migration owner is affected. -- Next usable boundary: Measure fresh and cross-commit hosted runs before any - lane rebalance or further DAG change. +- Next usable boundary: Validate the measured allocation in fresh hosted CI and + measure cross-commit cache reuse before further allocation or DAG changes. - Remaining risks: GitHub-hosted cache availability and queue time vary outside repository control; no improvement is claimed until measured. diff --git a/backend/scripts/test_lane_catalogue.py b/backend/scripts/test_lane_catalogue.py index 61df134d7..838faab6b 100644 --- a/backend/scripts/test_lane_catalogue.py +++ b/backend/scripts/test_lane_catalogue.py @@ -29,7 +29,6 @@ class TestLane: "tests/authorization/post_submit/test_live_authority.py", "tests/authorization/post_submit/test_principals.py", "tests/authorization/post_submit/test_timeout.py", - "tests/authorization/project_roles/test_cancellation_postgresql.py", "tests/authorization/project_roles/test_constraint_postgresql.py", "tests/authorization/project_roles/test_lifecycle_postgresql.py", @@ -395,13 +394,31 @@ class TestLane: "tests/test_projects.py", ) +# Checker delivery shares the project dispatch owner and uses measured +# project-A capacity after task B/C exhausted their unchanged execution cap. +CHECKER_DELIVERY_MODULES = ( + "tests/checkers/execution/test_results.py", + "tests/checkers/execution/test_execution.py", + "tests/checkers/execution/test_coordination.py", + "tests/checkers/execution/test_concurrency.py", + "tests/checkers/execution/test_storage.py", + "tests/checkers/execution/test_migration.py", + "tests/checkers/execution/test_material_lineage.py", + "tests/checkers/execution/test_material_migration.py", + "tests/test_post_submit_materialization.py", + "tests/test_post_submit_selection.py", + "tests/test_submission_evaluation_capacity.py", + "tests/test_checker_output_custody.py", + "tests/test_checker_output_storage.py", +) + + TASK_MODULES = ( "tests/reviews/decision/test_contracts.py", "tests/reviews/decision/test_storage.py", "tests/reviews/decision/test_migration.py", "tests/reviews/packet/test_repository.py", "tests/reviews/packet/test_migration.py", - "tests/tasks/post_submit_routing/test_contracts.py", "tests/tasks/post_submit_routing/test_storage.py", "tests/tasks/post_submit_routing/test_migration.py", @@ -420,7 +437,6 @@ class TestLane: "tests/authorization/task_audit_evidence/test_authority.py", "tests/authorization/task_audit_evidence/test_history.py", "tests/authorization/task_audit_evidence/test_transactions_concurrency.py", - "tests/authorization/task_queues/test_authority.py", "tests/authorization/task_queues/test_contracts.py", "tests/authorization/task_queues/test_transactions.py", @@ -445,14 +461,6 @@ class TestLane: "tests/tasks/test_assignment_invalidation_races.py", "tests/tasks/test_contribution_claim_races.py", "tests/tasks/test_submission_lineage.py", - "tests/checkers/execution/test_results.py", - "tests/checkers/execution/test_execution.py", - "tests/checkers/execution/test_coordination.py", - "tests/checkers/execution/test_concurrency.py", - "tests/checkers/execution/test_storage.py", - "tests/checkers/execution/test_migration.py", - "tests/checkers/execution/test_material_lineage.py", - "tests/checkers/execution/test_material_migration.py", "tests/checkers/post_submit/test_catalogue.py", "tests/checkers/post_submit/test_compiled_policy.py", "tests/checkers/post_submit/test_configuration.py", @@ -467,11 +475,6 @@ class TestLane: "tests/checkers/test_effective_intake_rules.py", "tests/test_default_pre_submit_execution.py", "tests/test_approved_guide_intake.py", - "tests/test_submission_evaluation_capacity.py", - "tests/test_post_submit_materialization.py", - "tests/test_post_submit_selection.py", - "tests/test_checker_output_custody.py", - "tests/test_checker_output_storage.py", "tests/test_pre_submit_attempt_recovery.py", "tests/test_pre_submit_attempt_contracts.py", "tests/test_pre_submit_attempt_authority_integration.py", @@ -509,8 +512,8 @@ class TestLane: "tests/test_celery_observability.py", ) -# Routing PREP proofs stay together on task C, which has measured headroom -# after task A reached the unchanged execution cap. +# Routing PREP proofs stay together beside project-policy preparation, using +# measured project-C capacity after task C exhausted its execution cap. ROUTING_AUTH_PREPARATION_MODULES = ( "tests/authorization/post_submit_routing/test_contracts.py", "tests/authorization/post_submit_routing/test_prepared.py", @@ -557,10 +560,14 @@ class TestLane: ADMIN_RUNNER_MODULE, ), ), - *(TestLane(name, PROJECT_MODULES) for name in PARTITIONED_PROJECT_LANES), - *(TestLane( - name, - TASK_MODULES - + (ROUTING_AUTH_PREPARATION_MODULES if name == "task_lifecycle_c" else ()), - ) for name in PARTITIONED_TASK_LANES), + *( + TestLane( + name, + PROJECT_MODULES + + (CHECKER_DELIVERY_MODULES if name == "project_lifecycle_a" else ()) + + (ROUTING_AUTH_PREPARATION_MODULES if name == "project_lifecycle_c" else ()), + ) + for name in PARTITIONED_PROJECT_LANES + ), + *(TestLane(name, TASK_MODULES) for name in PARTITIONED_TASK_LANES), ) diff --git a/backend/tests/test_ci_lane_catalogue.py b/backend/tests/test_ci_lane_catalogue.py index 97703dae8..b94a72f7a 100644 --- a/backend/tests/test_ci_lane_catalogue.py +++ b/backend/tests/test_ci_lane_catalogue.py @@ -42,14 +42,47 @@ def test_committed_lanes_cover_recursive_inventory_exactly_once() -> None: ) +def test_checker_delivery_and_routing_preparation_keep_exact_exclusive_owners() -> None: + delivery = { + "tests/checkers/execution/test_results.py", + "tests/checkers/execution/test_execution.py", + "tests/checkers/execution/test_coordination.py", + "tests/checkers/execution/test_concurrency.py", + "tests/checkers/execution/test_storage.py", + "tests/checkers/execution/test_migration.py", + "tests/checkers/execution/test_material_lineage.py", + "tests/checkers/execution/test_material_migration.py", + "tests/test_post_submit_materialization.py", + "tests/test_post_submit_selection.py", + "tests/test_submission_evaluation_capacity.py", + "tests/test_checker_output_custody.py", + "tests/test_checker_output_storage.py", + } + assert set(catalogue.CHECKER_DELIVERY_MODULES) == delivery + groups = ( + (delivery, "project_lifecycle_a"), + (set(catalogue.ROUTING_AUTH_PREPARATION_MODULES), "project_lifecycle_c"), + ) + for modules, owner in groups: + assert not modules & set(catalogue.TASK_MODULES) + assert not modules & set(catalogue.PARTITION_LANES_BY_MODULE) + for module in modules: + assert [lane.name for lane in LANES if module in lane.modules] == [owner] + nodes = [f"{module}::test_custody[{index}]" for module in modules for index in range(3)] + manifest = runner.build_manifest("a" * 40, list(reversed(nodes))) + assert len(manifest["nodes"]) == len(nodes) + assert {row["nodeid"] for row in manifest["nodes"]} == set(nodes) + assert {row["lane"] for row in manifest["nodes"]} == {owner} + + def test_measured_hotspots_have_explicit_semantic_owners() -> None: """Keep lane balance tied to subsystem ownership and measured schema cost.""" modules_by_lane = {lane.name: set(lane.modules) for lane in LANES} assert ( - modules_by_lane["project_lifecycle_a"] + modules_by_lane["project_lifecycle_a"] - set(catalogue.CHECKER_DELIVERY_MODULES) == modules_by_lane["project_lifecycle_b"] - == modules_by_lane["project_lifecycle_c"] + == modules_by_lane["project_lifecycle_c"] - set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) == { "tests/tasks/evaluation_delivery/test_custody.py", "tests/tasks/evaluation_delivery/test_delivery.py", @@ -193,7 +226,7 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: assert ( modules_by_lane["task_lifecycle_a"] == modules_by_lane["task_lifecycle_b"] - == modules_by_lane["task_lifecycle_c"] - set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) + == modules_by_lane["task_lifecycle_c"] == { "tests/authorization/submission_history/test_reads.py", "tests/authorization/submission_history/test_privacy.py", @@ -232,7 +265,6 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/authorization/task_audit_evidence/test_authority.py", "tests/authorization/task_audit_evidence/test_history.py", "tests/authorization/task_audit_evidence/test_transactions_concurrency.py", - "tests/tasks/test_management_queue.py", "tests/tasks/test_task_detail.py", "tests/tasks/test_work_context.py", @@ -250,14 +282,6 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/tasks/test_payment_policy_migration.py", "tests/tasks/test_contribution_claim_races.py", "tests/tasks/test_submission_lineage.py", - "tests/checkers/execution/test_results.py", - "tests/checkers/execution/test_execution.py", - "tests/checkers/execution/test_coordination.py", - "tests/checkers/execution/test_concurrency.py", - "tests/checkers/execution/test_storage.py", - "tests/checkers/execution/test_migration.py", - "tests/checkers/execution/test_material_lineage.py", - "tests/checkers/execution/test_material_migration.py", "tests/checkers/post_submit/test_catalogue.py", "tests/checkers/post_submit/test_compiled_policy.py", "tests/checkers/post_submit/test_configuration.py", @@ -272,11 +296,6 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/test_checkers.py", "tests/test_default_pre_submit_execution.py", "tests/test_approved_guide_intake.py", - "tests/test_submission_evaluation_capacity.py", - "tests/test_post_submit_materialization.py", - "tests/test_post_submit_selection.py", - "tests/test_checker_output_custody.py", - "tests/test_checker_output_storage.py", "tests/test_pre_submit_attempt_recovery.py", "tests/test_pre_submit_attempt_contracts.py", "tests/test_pre_submit_attempt_authority_integration.py", @@ -346,9 +365,9 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/reviews/acceptance/test_storage.py", "tests/reviews/acceptance/test_migration.py", runner.ADMIN_RUNNER_MODULE, - } | static_contracts | post_submit_storage_contracts | set(catalogue.OBSERVABILITY_MODULES) | set(catalogue.TASK_ROUTING_REQUEST_MODULES) == modules_by_lane[ - "schema_contracts" - ] + } | static_contracts | post_submit_storage_contracts | set( + catalogue.OBSERVABILITY_MODULES + ) | set(catalogue.TASK_ROUTING_REQUEST_MODULES) == modules_by_lane["schema_contracts"] assert { "tests/authorization/admin_access/test_bootstrap_cli.py", "tests/authorization/admin_access/test_api_journey.py", @@ -773,7 +792,9 @@ def test_routing_request_proofs_use_schema_lane_with_measured_headroom(): } assert set(catalogue.TASK_ROUTING_REQUEST_MODULES) == expected for lane in LANES: - assert set(lane.modules) & expected == (expected if lane.name == "schema_contracts" else set()) + assert set(lane.modules) & expected == ( + expected if lane.name == "schema_contracts" else set() + ) assert not expected & set(catalogue.PARTITION_LANES_BY_MODULE) @@ -785,18 +806,22 @@ def test_observability_proofs_use_schema_lane_with_measured_headroom(): } assert set(catalogue.OBSERVABILITY_MODULES) == expected for lane in LANES: - assert set(lane.modules) & expected == (expected if lane.name == "schema_contracts" else set()) + assert set(lane.modules) & expected == ( + expected if lane.name == "schema_contracts" else set() + ) assert not expected & set(catalogue.PARTITION_LANES_BY_MODULE) -def test_routing_authorization_proofs_run_once_on_task_c(): +def test_routing_authorization_proofs_run_once_on_project_c(): expected = { "tests/authorization/post_submit_routing/test_contracts.py", "tests/authorization/post_submit_routing/test_prepared.py", } assert set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) == expected for lane in LANES: - assert set(lane.modules) & expected == (expected if lane.name == "task_lifecycle_c" else set()) + assert set(lane.modules) & expected == ( + expected if lane.name == "project_lifecycle_c" else set() + ) assert not expected & set(catalogue.PARTITION_LANES_BY_MODULE) @@ -809,7 +834,10 @@ def test_shared_acceptance_owner_proofs_are_in_partitioned_project_lanes(): "tests/tasks/accepted_effects/test_postgresql.py", } assert expected <= set(catalogue.PROJECT_MODULES) - assert all(catalogue.PARTITION_LANES_BY_MODULE[path] == catalogue.PARTITIONED_PROJECT_LANES for path in expected) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[path] == catalogue.PARTITIONED_PROJECT_LANES + for path in expected + ) def test_evaluation_custody_proofs_use_project_lane_headroom(): @@ -817,7 +845,10 @@ def test_evaluation_custody_proofs_use_project_lane_headroom(): "tests/tasks/post_submit_routing/test_evaluation_guard.py", "tests/tasks/post_submit_routing/test_evaluation_currentness.py", "tests/tasks/post_submit_routing/test_review_admission_currentness.py", - "tests/tasks/post_submit_routing/test_source_preparation.py", + "tests/tasks/post_submit_routing/test_source_preparation.py", } assert expected <= set(catalogue.PROJECT_MODULES) - assert all(catalogue.PARTITION_LANES_BY_MODULE[path] == catalogue.PARTITIONED_PROJECT_LANES for path in expected) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[path] == catalogue.PARTITIONED_PROJECT_LANES + for path in expected + ) diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index 9d9b51d0e..8ca079e3e 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -136,8 +136,12 @@ This is not a production configuration or proof of host-power-loss durability: An exhausted mount fails the job; it does not silently change storage or skip tests. The `project_lifecycle_a`, `project_lifecycle_b`, and `project_lifecycle_c` lanes -partition PROJECT nodes; `task_lifecycle_a`, `task_lifecycle_b`, and `task_lifecycle_c` use the same -deterministic partition mechanism for TASK and checker nodes. The single `schema_contracts` lane owns all baseline/PostgreSQL schema, reset and +partition PROJECT nodes. Project A also owns checker execution, materialization, +evaluation capacity and output custody beside initial dispatch/delivery; project C +owns routing AUTH preparation. These exclusive workloads use measured spare +capacity after TASK B/C exhausted their unchanged execution cap. +`task_lifecycle_a`, `task_lifecycle_b`, and `task_lifecycle_c` use the same +deterministic partition mechanism for the remaining TASK and checker nodes. The single `schema_contracts` lane owns all baseline/PostgreSQL schema, reset and isolated-runner contracts. The `shared_foundations_a` and `shared_foundations_b` lanes deterministically partition exact node IDs from the remaining authorization, artifact, API, and From b5401577ba510fe49553b791e0c36ab82dd88c2a Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 18:14:54 +0100 Subject: [PATCH 5/5] ci: partition checker delivery across project lanes --- .../changes/backend-ci-cache-and-timing.md | 36 +++++++++++++--- backend/scripts/test_lane_catalogue.py | 16 ++++---- backend/tests/test_ci_lane_catalogue.py | 41 ++++++++++++------- docs/operations_backend_testing.md | 8 ++-- 4 files changed, 69 insertions(+), 32 deletions(-) diff --git a/.commitrail/changes/backend-ci-cache-and-timing.md b/.commitrail/changes/backend-ci-cache-and-timing.md index 0da957738..4295f757d 100644 --- a/.commitrail/changes/backend-ci-cache-and-timing.md +++ b/.commitrail/changes/backend-ci-cache-and-timing.md @@ -43,7 +43,7 @@ run only; this record makes no hosted performance claim before a candidate run. timeout or treating those logs as fan-in evidence. - `backend/scripts/test_lane_catalogue.py` and `backend/tests/test_ci_lane_catalogue.py`: rebalance measured checker-delivery - and routing-preparation workloads into existing project-lane capacity; + and routing-preparation workloads into the existing three-way project partition; preserve every test node, deterministic assignment and all nine lanes. - `scripts/test_lightweight_agent_gates.py` and `backend/tests/test_ci_test_lanes.py`: focused positive and adversarial @@ -81,9 +81,27 @@ capacity and output-custody/storage proof together to PROJECT A, beside the existing initial-dispatch and evaluation-delivery owners. Move routing AUTH PREP proof from TASK C to PROJECT C. Keep the original per-node hash mechanism, UUID seed, node IDs, isolation, 1200-second limits and exact aggregate custody. -Use catalogue/inventory regressions and a dropped/duplicated-owner mutant -before independent CI-integrity and QA review. The recorded durations guide -this allocation; only a fresh hosted run can establish runtime success. +That initial allocation used catalogue/inventory regressions and a +dropped/duplicated-owner mutant. The next measurement below supersedes its +exclusive placement; recorded forecasts are not hosted success. + + +The next hosted measurement, Backend run `37809916459` at merge +`dbc84f10e1ca1926412856a5f34e4bc91ab49c5d`, disproved the exclusive +PROJECT A placement: it exhausted 1200 seconds after 665 of 745 nodes, with +80 unfinished, zero skips and confirmed PostgreSQL/MinIO cleanup. PROJECT B +finished in 769.294 seconds and PROJECT C in 483.442 seconds. Assigning the +entire delivery group to one runner was the incorrect assumption. + +Repair that allocation by partitioning all delivery and routing-preparation +nodes across the same three PROJECT lanes using the existing exact-node hash. +No new jobs, runtime controls, retries, skips, evidence formats or product changes +are allowed. Update the existing ownership regressions to reject an exclusive +placement and retain inventory, exact node identity and fan-in checks. Compare +all 8,803 nodes from that hosted manifest before and after this repair; only these +15 modules may change assignment. Focused tests, a regression mutant restoring +exclusive placement, and independent CI-integrity/security/QA/test-delta review +are required before pushing. Hosted speed remains unproven until fresh CI. ## Design and decisions @@ -130,7 +148,8 @@ promise an eight-minute Backend completion time. timeout, log and evidence contracts. - [x] The nine lanes, full inventory, fan-in failure propagation, CLI dependency, aggregate validation, permissions and timeouts are unchanged; only the named - checker-delivery and routing-preparation workloads change lane ownership. + checker-delivery and routing-preparation workloads change lane ownership; + all use the existing three-way PROJECT partition. - [x] Contributor guidance permits only failed-job reruns for a diagnosed same-head transient, explains aggregate revalidation of successful lane evidence, and requires diagnosis or fresh current-tree CI in the other cases. @@ -156,6 +175,11 @@ promise an eight-minute Backend completion time. | Timeout diagnosis and allocation | Retained summaries and complete TASK A duration phases from Backend run `37799550116` | TASK B/C interrupted at 1200 seconds; delivery-group transfer forecasts about 152/161 seconds of relief and about 1036 seconds for PROJECT A | Forecast extrapolates TASK A module costs; fresh hosted execution is required | | Rebalance integrity | Catalogue/runner/evidence/merge focused pytest batch; dropped and duplicated delivery-owner mutations | 124 tests passed; recursive inventory rejected both mutations; all node IDs and existing hash/UUID-seed mechanisms remain | This proves assignment and evidence contracts, not hosted speed | +| Exclusive-placement diagnosis | Actual run `37809916459` PROJECT A/B/C summaries and cleanup records | A interrupted at 1200.966s with 665/745 completed; B/C passed at 769.294/483.442s; A cleanup complete | A interruption prevents a final duration table; previous cost forecast was insufficient | +| Three-way allocation integrity | Replay the retained 8,803-node hosted manifest through the repaired catalogue; recursive inventory validator | Same IDs and execution kinds, exactly once; 127 assignments changed only within the 15 named modules; delivery A/B/C 41/32/39 and routing PREP 28/28/16 | This is a historical inventory replay, not a claim that current-head full CI completed | +| Repair behavior and gates | Focused catalogue/runner/evidence/merge pytest batch; workflow unittest suite; Ruff | 124 passed in 46.13s and 19 workflow tests passed; lint/format clean | Fresh hosted runtime measurement remains required | +| Exclusive-placement regression | Temporarily restore the predecessor catalogue and run the updated exact partition regression, then restore the repair | Predecessor fails because delivery has no three-way partition; restored candidate passes in 0.65s | Existing inventory and fan-in regressions retain loss/duplication/failure checks | + ## Review findings Initial workflow review found that `runner.temp` is not an allowed workflow-level @@ -177,7 +201,7 @@ environment export and fresh install in order; the relocation mutant fails it. ## Reconciliation - Current-source reconciliation: Reconciled with `main` at - `b169e83f816bba417fc0618a6e2d419acb2f94ed`; no product or migration owner is + `36e8a615f01801cecd6ccf83d7411235ab1cfa8f`; no product or migration owner is affected. - Next usable boundary: Validate the measured allocation in fresh hosted CI and measure cross-commit cache reuse before further allocation or DAG changes. diff --git a/backend/scripts/test_lane_catalogue.py b/backend/scripts/test_lane_catalogue.py index 838faab6b..5bf65776f 100644 --- a/backend/scripts/test_lane_catalogue.py +++ b/backend/scripts/test_lane_catalogue.py @@ -394,8 +394,8 @@ class TestLane: "tests/test_projects.py", ) -# Checker delivery shares the project dispatch owner and uses measured -# project-A capacity after task B/C exhausted their unchanged execution cap. +# Checker delivery shares project dispatch ownership and the existing +# three-way partition; exclusive placement exceeded the hosted execution cap. CHECKER_DELIVERY_MODULES = ( "tests/checkers/execution/test_results.py", "tests/checkers/execution/test_execution.py", @@ -512,8 +512,7 @@ class TestLane: "tests/test_celery_observability.py", ) -# Routing PREP proofs stay together beside project-policy preparation, using -# measured project-C capacity after task C exhausted its execution cap. +# Routing PREP proofs share the three-way project-policy partition. ROUTING_AUTH_PREPARATION_MODULES = ( "tests/authorization/post_submit_routing/test_contracts.py", "tests/authorization/post_submit_routing/test_prepared.py", @@ -521,7 +520,10 @@ class TestLane: PARTITION_GROUPS = ( (PARTITIONED_SHARED_LANES, SHARED_FOUNDATION_MODULES), - (PARTITIONED_PROJECT_LANES, PROJECT_MODULES), + ( + PARTITIONED_PROJECT_LANES, + PROJECT_MODULES + CHECKER_DELIVERY_MODULES + ROUTING_AUTH_PREPARATION_MODULES, + ), (PARTITIONED_TASK_LANES, TASK_MODULES), ) PARTITION_LANES_BY_MODULE = { @@ -563,9 +565,7 @@ class TestLane: *( TestLane( name, - PROJECT_MODULES - + (CHECKER_DELIVERY_MODULES if name == "project_lifecycle_a" else ()) - + (ROUTING_AUTH_PREPARATION_MODULES if name == "project_lifecycle_c" else ()), + PROJECT_MODULES + CHECKER_DELIVERY_MODULES + ROUTING_AUTH_PREPARATION_MODULES, ) for name in PARTITIONED_PROJECT_LANES ), diff --git a/backend/tests/test_ci_lane_catalogue.py b/backend/tests/test_ci_lane_catalogue.py index b94a72f7a..1701603b1 100644 --- a/backend/tests/test_ci_lane_catalogue.py +++ b/backend/tests/test_ci_lane_catalogue.py @@ -42,7 +42,7 @@ def test_committed_lanes_cover_recursive_inventory_exactly_once() -> None: ) -def test_checker_delivery_and_routing_preparation_keep_exact_exclusive_owners() -> None: +def test_checker_delivery_and_routing_preparation_partition_every_node_once() -> None: delivery = { "tests/checkers/execution/test_results.py", "tests/checkers/execution/test_execution.py", @@ -60,19 +60,27 @@ def test_checker_delivery_and_routing_preparation_keep_exact_exclusive_owners() } assert set(catalogue.CHECKER_DELIVERY_MODULES) == delivery groups = ( - (delivery, "project_lifecycle_a"), - (set(catalogue.ROUTING_AUTH_PREPARATION_MODULES), "project_lifecycle_c"), + delivery, + set(catalogue.ROUTING_AUTH_PREPARATION_MODULES), ) - for modules, owner in groups: + for modules in groups: assert not modules & set(catalogue.TASK_MODULES) - assert not modules & set(catalogue.PARTITION_LANES_BY_MODULE) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[module] == catalogue.PARTITIONED_PROJECT_LANES + for module in modules + ) for module in modules: - assert [lane.name for lane in LANES if module in lane.modules] == [owner] - nodes = [f"{module}::test_custody[{index}]" for module in modules for index in range(3)] + assert ( + tuple(lane.name for lane in LANES if module in lane.modules) + == catalogue.PARTITIONED_PROJECT_LANES + ) + nodes = [f"{module}::test_custody[{index}]" for module in modules for index in range(256)] manifest = runner.build_manifest("a" * 40, list(reversed(nodes))) assert len(manifest["nodes"]) == len(nodes) assert {row["nodeid"] for row in manifest["nodes"]} == set(nodes) - assert {row["lane"] for row in manifest["nodes"]} == {owner} + assert {row["lane"] for row in manifest["nodes"]} == set( + catalogue.PARTITIONED_PROJECT_LANES + ) def test_measured_hotspots_have_explicit_semantic_owners() -> None: @@ -80,10 +88,12 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: modules_by_lane = {lane.name: set(lane.modules) for lane in LANES} assert ( - modules_by_lane["project_lifecycle_a"] - set(catalogue.CHECKER_DELIVERY_MODULES) + modules_by_lane["project_lifecycle_a"] == modules_by_lane["project_lifecycle_b"] - == modules_by_lane["project_lifecycle_c"] - set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) - == { + == modules_by_lane["project_lifecycle_c"] + == set(catalogue.CHECKER_DELIVERY_MODULES) + | set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) + | { "tests/tasks/evaluation_delivery/test_custody.py", "tests/tasks/evaluation_delivery/test_delivery.py", "tests/tasks/evaluation_delivery/test_isolation.py", @@ -812,7 +822,7 @@ def test_observability_proofs_use_schema_lane_with_measured_headroom(): assert not expected & set(catalogue.PARTITION_LANES_BY_MODULE) -def test_routing_authorization_proofs_run_once_on_project_c(): +def test_routing_authorization_proofs_use_the_project_partition(): expected = { "tests/authorization/post_submit_routing/test_contracts.py", "tests/authorization/post_submit_routing/test_prepared.py", @@ -820,9 +830,12 @@ def test_routing_authorization_proofs_run_once_on_project_c(): assert set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) == expected for lane in LANES: assert set(lane.modules) & expected == ( - expected if lane.name == "project_lifecycle_c" else set() + expected if lane.name in catalogue.PARTITIONED_PROJECT_LANES else set() ) - assert not expected & set(catalogue.PARTITION_LANES_BY_MODULE) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[module] == catalogue.PARTITIONED_PROJECT_LANES + for module in expected + ) def test_shared_acceptance_owner_proofs_are_in_partitioned_project_lanes(): diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index 8ca079e3e..24c9993d7 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -136,10 +136,10 @@ This is not a production configuration or proof of host-power-loss durability: An exhausted mount fails the job; it does not silently change storage or skip tests. The `project_lifecycle_a`, `project_lifecycle_b`, and `project_lifecycle_c` lanes -partition PROJECT nodes. Project A also owns checker execution, materialization, -evaluation capacity and output custody beside initial dispatch/delivery; project C -owns routing AUTH preparation. These exclusive workloads use measured spare -capacity after TASK B/C exhausted their unchanged execution cap. +partition PROJECT nodes, checker execution, materialization, evaluation capacity, +output custody and routing AUTH preparation beside initial dispatch/delivery. +All use the existing exact-node hash across the three lanes: placing the entire +checker delivery group exclusively on project A exhausted its execution cap. `task_lifecycle_a`, `task_lifecycle_b`, and `task_lifecycle_c` use the same deterministic partition mechanism for the remaining TASK and checker nodes. The single `schema_contracts` lane owns all baseline/PostgreSQL schema, reset and isolated-runner contracts. The