diff --git a/.commitrail/changes/backend-ci-cache-and-timing.md b/.commitrail/changes/backend-ci-cache-and-timing.md new file mode 100644 index 000000000..4295f757d --- /dev/null +++ b/.commitrail/changes/backend-ci-cache-and-timing.md @@ -0,0 +1,209 @@ +# Backend CI Caches And Measured Lane Balance + +- Initiative: `None` +- Durable disposition: `Complete` +- Intended merge outcome: Backend CI reuses an exact content-addressed MinIO + source image and pip download cache, moves measured workloads into existing + lane capacity, and retains every test and the fail-closed aggregate. + +## Intent + +Reduce repeated setup work and repair measured task-lane overload in the required +Backend workflow. Exact cache inputs, retained duration diagnostics and explicit +workload placement preserve complete test and evidence coverage. + +## Current behavior + +`.github/workflows/backend.yml` builds the source-pinned MinIO image once per +commit because its cache key includes `github.sha`. Backend jobs create fresh +Python environments and repeatedly download the same dependency inputs. The +semantic runner retains each lane log but reports only the 25 slowest pytest +phases, limiting later balancing evidence. + +GitHub Backend run `37784941972` on PR #507 is the measurement source for this +slice. Its first MinIO job spent 145 seconds building after a cache miss. Lane +jobs also repeatedly spent about 25--28 seconds installing backend dependencies, +and queued for materially different periods. These measurements describe that +run only; this record makes no hosted performance claim before a candidate run. + +## Bounded change + +### Allowed + +- `.github/workflows/backend.yml`: make the MinIO tar cache key depend on the + runner OS/architecture, exact `docker/minio/**` content and workflow recipe, + retaining exact per-run artifact identity, checksums and live provider + verification. +- `.github/workflows/backend.yml`: restore a pip download cache for the exact + backend dependency/workflow inputs and save it only after trusted-main + validation; every job still performs a fresh install into its fresh hosted + Python environment. +- `backend/scripts/run_test_lanes.py`: retain all pytest duration diagnostics in + the existing redacted per-lane diagnostic logs without changing selection or + timeout or treating those logs as fan-in evidence. +- `backend/scripts/test_lane_catalogue.py` and + `backend/tests/test_ci_lane_catalogue.py`: rebalance measured checker-delivery + and routing-preparation workloads into the existing three-way project partition; + preserve every test node, deterministic assignment and all nine lanes. +- `scripts/test_lightweight_agent_gates.py` and + `backend/tests/test_ci_test_lanes.py`: focused positive and adversarial + regressions for cache identity, fresh installs, runtime verification, artifact + identity, and lane duration options. +- `docs/operations_backend_testing.md`: extend the canonical failure/rerun owner + with the exact failed-job command, repeated-timeout diagnosis, fresh-current- + tree rule and explicit limits on what the cache optimization can improve. +- `docker/minio/README.md`: describe trusted-main cache publication, exact + content/workflow/platform reuse and the separate per-run artifact binding. +- `CONTRIBUTING.md`: link contributors to that canonical Backend rerun guidance + without copying its operational contract. +- This standalone Commitrail record. + +### Not allowed + +- No product source, migrations, dependencies, lane + count, test selection, skips, coverage behavior, timeout, retry, runner + permission, concurrency, service image or branch-protection change. +- No installed virtual-environment cache, broad restore prefix, commit-SHA MinIO + cache key, unverified image reuse, affected-only tests or hidden failures. +- No lane-count/DAG redesign or performance claim before hosted measurement. + +### Timeout repair plan + +Backend run `37799550116` tested merge `9ea0fc9a2f9b0f9927ae98f06f9ea0a487daa050`. +TASK B and C exhausted 1200 seconds with 22 and 30 nodes unfinished. Their +databases and MinIO resources were cleaned up; the aggregate correctly failed. +PROJECT A completed in 529 seconds, PROJECT B in 959 and PROJECT C in 758; +TASK A completed in 969. This demonstrates available capacity in the existing +jobs, not that a same-head retry would solve the imbalance. + +Move checker execution, post-submit materialization/selection, evaluation +capacity and output-custody/storage proof together to PROJECT A, beside the +existing initial-dispatch and evaluation-delivery owners. Move routing AUTH +PREP proof from TASK C to PROJECT C. Keep the original per-node hash mechanism, +UUID seed, node IDs, isolation, 1200-second limits and exact aggregate custody. +That initial allocation used catalogue/inventory regressions and a +dropped/duplicated-owner mutant. The next measurement below supersedes its +exclusive placement; recorded forecasts are not hosted success. + + +The next hosted measurement, Backend run `37809916459` at merge +`dbc84f10e1ca1926412856a5f34e4bc91ab49c5d`, disproved the exclusive +PROJECT A placement: it exhausted 1200 seconds after 665 of 745 nodes, with +80 unfinished, zero skips and confirmed PostgreSQL/MinIO cleanup. PROJECT B +finished in 769.294 seconds and PROJECT C in 483.442 seconds. Assigning the +entire delivery group to one runner was the incorrect assumption. + +Repair that allocation by partitioning all delivery and routing-preparation +nodes across the same three PROJECT lanes using the existing exact-node hash. +No new jobs, runtime controls, retries, skips, evidence formats or product changes +are allowed. Update the existing ownership regressions to reject an exclusive +placement and retain inventory, exact node identity and fan-in checks. Compare +all 8,803 nodes from that hosted manifest before and after this repair; only these +15 modules may change assignment. Focused tests, a regression mutant restoring +exclusive placement, and independent CI-integrity/security/QA/test-delta review +are required before pushing. Hosted speed remains unproven until fresh CI. + +## Design and decisions + +The reusable MinIO tar has an exact recipe/platform cache identity: a version, +runner OS and architecture, and a hash of `docker/minio/**` plus the workflow +that owns the build command. Rare workflow-only edits intentionally invalidate +the cache. A cold rebuild is not guaranteed byte-identical because the Docker +recipe consumes live apt repositories. Pull requests can restore only; a cache +miss is saved only after a trusted `main` push verifies the built image. The +cache has no restore prefix. Every workflow attempt still publishes a +SHA-and-attempt-named artifact after loading the tar, checking the image version, +starting it, probing health, and producing a checksum that every consumer verifies. + +Pinned cache restore/save actions manage a pip download directory keyed by +Python/platform and the hashes of `backend/pyproject.toml` plus this workflow, +which includes the separate exact Ruff pin. Pull requests restore only; the +validated preflight on a trusted `main` push may save a miss. No installed +environment is cached. Existing `pip install` commands remain mandatory in +preflight, every lane and aggregation. + +The lane runner changes `--durations=25` to `--durations=0`; pytest retains all +nontrivial setup/call/teardown timings in the existing diagnostic log. A lane +terminated by the unchanged deadline may not emit the final duration summary, +so completion evidence and interruption metadata remain the authority. + +The operator guidance treats rerun as evidence recovery, never a green-result +loop. Removing the observed 145-second rebuild when its exact context is +unchanged does not solve 17-minute queue waits or 20-minute lanes and does not +promise an eight-minute Backend completion time. + +## Acceptance criteria + +- [x] Unchanged MinIO Docker context and Backend build workflow on the same + runner OS/architecture use one exact cache key across commits; any context, + workflow or platform change misses it. +- [x] MinIO cache restore has no broad prefix and cannot bypass image load, + version execution, live health probe, per-run artifact name or checksum. +- [x] Pull requests never save MinIO or pip caches; only a verified trusted-main + miss can populate a key that later pull requests restore. +- [x] Backend jobs cache pip downloads only and still run the existing fresh + package installation commands from exact dependency inputs. +- [x] Ordinary and schema-admin lane commands request complete duration + diagnostics while retaining the same exact node list, coverage, isolation, + timeout, log and evidence contracts. +- [x] The nine lanes, full inventory, fan-in failure propagation, CLI dependency, + aggregate validation, permissions and timeouts are unchanged; only the named + checker-delivery and routing-preparation workloads change lane ownership; + all use the existing three-way PROJECT partition. +- [x] Contributor guidance permits only failed-job reruns for a diagnosed + same-head transient, explains aggregate revalidation of successful lane + evidence, and requires diagnosis or fresh current-tree CI in the other cases. +- [x] Focused workflow/runner tests and mutation probes fail when cache identity, + verification, fresh installation, duration output or required gates weaken. + +## Risk and review routing + +- Risk class: `L1` +- Required reviewers: `ci_integrity`, `security`, `qa`, `test_delta` +- Human review focus: Cross-commit cache trust and invalidation, fresh-install + preservation, exact artifact/runtime verification, unchanged gate propagation, + and whether duration output remains diagnostic rather than authoritative. + +## Evidence + +| Claim | Command or proof | Result | Remaining uncertainty | +|---|---|---|---| +| Measured bottlenecks | Inspect PR #507 Backend run `37784941972` job/step timestamps from `/tmp/ws-ci-507-jobs.json` | MinIO build took 145 seconds; repeated lane installs took about 25--28 seconds | Hosted queue and runner variability are uncontrolled | +| Cache custody | `python -m unittest -v scripts.test_lightweight_agent_gates`; checksum-verified actionlint 1.7.7; cache-key, invalid-context and per-job-install relocation mutants | 19 workflow gates passed; candidate linted cleanly; all three cache/install mutants were rejected | A real cross-commit hit requires a trusted-main seed and later hosted run | +| Lane diagnostics | Focused lane/evidence pytest batch plus duration-option mutant | 123 tests passed; reverting both commands to `--durations=25` failed the exact lane-command regression | A timed-out pytest process cannot print its final duration table | +| Full gate preservation | Workflow inventory/fan-in tests, exact diff inspection, Commitrail and Markdown gates | Existing lane count, selection, timeouts, permissions, fan-in and fresh installs remain; repository gates passed | Root owns hosted aggregate validation | +| Timeout diagnosis and allocation | Retained summaries and complete TASK A duration phases from Backend run `37799550116` | TASK B/C interrupted at 1200 seconds; delivery-group transfer forecasts about 152/161 seconds of relief and about 1036 seconds for PROJECT A | Forecast extrapolates TASK A module costs; fresh hosted execution is required | +| Rebalance integrity | Catalogue/runner/evidence/merge focused pytest batch; dropped and duplicated delivery-owner mutations | 124 tests passed; recursive inventory rejected both mutations; all node IDs and existing hash/UUID-seed mechanisms remain | This proves assignment and evidence contracts, not hosted speed | + +| Exclusive-placement diagnosis | Actual run `37809916459` PROJECT A/B/C summaries and cleanup records | A interrupted at 1200.966s with 665/745 completed; B/C passed at 769.294/483.442s; A cleanup complete | A interruption prevents a final duration table; previous cost forecast was insufficient | +| Three-way allocation integrity | Replay the retained 8,803-node hosted manifest through the repaired catalogue; recursive inventory validator | Same IDs and execution kinds, exactly once; 127 assignments changed only within the 15 named modules; delivery A/B/C 41/32/39 and routing PREP 28/28/16 | This is a historical inventory replay, not a claim that current-head full CI completed | +| Repair behavior and gates | Focused catalogue/runner/evidence/merge pytest batch; workflow unittest suite; Ruff | 124 passed in 46.13s and 19 workflow tests passed; lint/format clean | Fresh hosted runtime measurement remains required | +| Exclusive-placement regression | Temporarily restore the predecessor catalogue and run the updated exact partition regression, then restore the repair | Predecessor fails because delivery has no three-way partition; restored candidate passes in 0.65s | Existing inventory and fan-in regressions retain loss/duplication/failure checks | + +## Review findings + +Initial workflow review found that `runner.temp` is not an allowed workflow-level +`env` context. Each existing identity step now exports `PIP_CACHE_DIR` through +`GITHUB_ENV` before installation; actionlint and an invalid-context mutant prove +that regression is rejected. Cache trust review also changed both caches to +restore-only on pull requests and save-on-miss only after trusted-main +verification. + +Documentation review also corrected the `gh` command description: `--failed` +is the selective form, while the bare command is the whole-workflow rerun; there +is no `--all` option. The MinIO owner documentation now distinguishes the +trusted cross-commit cache from each SHA-and-attempt-bound artifact. +QA review found that a repository-wide install count could miss moving the +aggregate install into preflight. The regression now inspects preflight, lane +and aggregate blocks independently and requires exactly one cache restore, +environment export and fresh install in order; the relocation mutant fails it. + +## Reconciliation + +- Current-source reconciliation: Reconciled with `main` at + `36e8a615f01801cecd6ccf83d7411235ab1cfa8f`; no product or migration owner is + affected. +- Next usable boundary: Validate the measured allocation in fresh hosted CI and + measure cross-commit cache reuse before further allocation or DAG changes. +- Remaining risks: GitHub-hosted cache availability and queue time vary outside + repository control; no improvement is claimed until measured. diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index fe9c656e2..1acacc19c 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -36,12 +36,12 @@ jobs: test "$(git rev-parse HEAD)" = "${GITHUB_SHA}" echo "artifact=minio-source-${GITHUB_SHA}-${GITHUB_RUN_ATTEMPT}" >> "${GITHUB_OUTPUT}" - - name: Cache exact source image + - name: Restore exact source image from trusted cache id: cache - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 with: path: ${{ runner.temp }}/minio-image/minio.tar - key: minio-source-v1-${{ github.sha }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('docker/minio/**') }} + key: minio-source-v2-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('docker/minio/**', '.github/workflows/backend.yml') }} - name: Build source-pinned MinIO once if: steps.cache.outputs.cache-hit != 'true' @@ -74,6 +74,13 @@ jobs: done exit 1 + - name: Save verified source image to trusted cache + if: github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.cache.outputs.cache-hit != 'true' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/minio-image/minio.tar + key: minio-source-v2-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('docker/minio/**', '.github/workflows/backend.yml') }} + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 with: name: ${{ steps.identity.outputs.artifact }} @@ -96,6 +103,13 @@ jobs: with: python-version: "3.12" + - name: Restore exact backend pip downloads from trusted cache + id: pip-cache + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/backend-pip-cache + key: backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }} + - name: Bind exact checked-out tree shell: bash run: | @@ -103,6 +117,7 @@ jobs: test "$(git rev-parse HEAD)" = "${GITHUB_SHA}" test -z "$(git status --porcelain)" test "$(git rev-parse "${GITHUB_SHA}^{tree}")" = "$(git rev-parse HEAD^{tree})" + echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}" - name: Install backend architecture dependencies working-directory: backend @@ -139,6 +154,13 @@ jobs: --ledger ../.ci/auth-boundaries/TEST_STRUCTURE_DEBT.json python -m scripts.behavior_ownership validate + - name: Save verified backend pip downloads to trusted cache + if: github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.pip-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/backend-pip-cache + key: backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }} + lanes: needs: minio-image runs-on: ubuntu-latest @@ -192,6 +214,12 @@ jobs: with: python-version: "3.12" + - name: Restore exact backend pip downloads from trusted cache + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/backend-pip-cache + key: backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }} + - id: identity name: Bind exact checked-out tree shell: bash @@ -204,6 +232,7 @@ jobs: test "$(git rev-parse "${tree_sha}^{tree}")" = "$(git rev-parse HEAD^{tree})" echo "job_start_epoch=${job_start_epoch}" >> "${GITHUB_OUTPUT}" echo "tree_sha=${tree_sha}" >> "${GITHUB_OUTPUT}" + echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}" - name: Install backend and exact Ruff working-directory: backend @@ -358,6 +387,12 @@ jobs: with: python-version: "3.12" + - name: Restore exact backend pip downloads from trusted cache + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 + with: + path: ${{ runner.temp }}/backend-pip-cache + key: backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }} + - id: identity name: Bind exact checked-out tree shell: bash @@ -369,6 +404,7 @@ jobs: test -z "$(git status --porcelain)" echo "job_start_epoch=${job_start_epoch}" >> "${GITHUB_OUTPUT}" echo "tree_sha=${tree_sha}" >> "${GITHUB_OUTPUT}" + echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}" - name: Install backend working-directory: backend diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f237e24cc..85341f0a7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -149,6 +149,9 @@ Different initiatives may proceed concurrently in separate branches or worktrees. If another pull request changes the base, inspect the new delta and rerun affected checks; unchanged evidence does not need ceremonial repetition. +For Backend failure diagnosis and safe partial reruns, follow the canonical +[Backend testing operations guide](docs/operations_backend_testing.md#failure-diagnosis-and-reruns). + ## Behavior Ownership Catalogue The hosted behavior-mutation check is temporarily retired because its diff --git a/backend/scripts/run_test_lanes.py b/backend/scripts/run_test_lanes.py index 644416e2a..79b394e1b 100644 --- a/backend/scripts/run_test_lanes.py +++ b/backend/scripts/run_test_lanes.py @@ -496,7 +496,7 @@ def lane_command( *_plugin_args(), "--cov=app", "--cov-report=", - "--durations=25", + "--durations=0", *nodes, ] return [ @@ -525,7 +525,7 @@ def admin_runner_command(nodes: list[str]) -> list[str]: *_plugin_args(), "--cov=app", "--cov-report=", - "--durations=25", + "--durations=0", *nodes, ] return [sys.executable, "-c", ADMIN_REDACTING_WRAPPER, *pytest_command] diff --git a/backend/scripts/test_lane_catalogue.py b/backend/scripts/test_lane_catalogue.py index 61df134d7..5bf65776f 100644 --- a/backend/scripts/test_lane_catalogue.py +++ b/backend/scripts/test_lane_catalogue.py @@ -29,7 +29,6 @@ class TestLane: "tests/authorization/post_submit/test_live_authority.py", "tests/authorization/post_submit/test_principals.py", "tests/authorization/post_submit/test_timeout.py", - "tests/authorization/project_roles/test_cancellation_postgresql.py", "tests/authorization/project_roles/test_constraint_postgresql.py", "tests/authorization/project_roles/test_lifecycle_postgresql.py", @@ -395,13 +394,31 @@ class TestLane: "tests/test_projects.py", ) +# Checker delivery shares project dispatch ownership and the existing +# three-way partition; exclusive placement exceeded the hosted execution cap. +CHECKER_DELIVERY_MODULES = ( + "tests/checkers/execution/test_results.py", + "tests/checkers/execution/test_execution.py", + "tests/checkers/execution/test_coordination.py", + "tests/checkers/execution/test_concurrency.py", + "tests/checkers/execution/test_storage.py", + "tests/checkers/execution/test_migration.py", + "tests/checkers/execution/test_material_lineage.py", + "tests/checkers/execution/test_material_migration.py", + "tests/test_post_submit_materialization.py", + "tests/test_post_submit_selection.py", + "tests/test_submission_evaluation_capacity.py", + "tests/test_checker_output_custody.py", + "tests/test_checker_output_storage.py", +) + + TASK_MODULES = ( "tests/reviews/decision/test_contracts.py", "tests/reviews/decision/test_storage.py", "tests/reviews/decision/test_migration.py", "tests/reviews/packet/test_repository.py", "tests/reviews/packet/test_migration.py", - "tests/tasks/post_submit_routing/test_contracts.py", "tests/tasks/post_submit_routing/test_storage.py", "tests/tasks/post_submit_routing/test_migration.py", @@ -420,7 +437,6 @@ class TestLane: "tests/authorization/task_audit_evidence/test_authority.py", "tests/authorization/task_audit_evidence/test_history.py", "tests/authorization/task_audit_evidence/test_transactions_concurrency.py", - "tests/authorization/task_queues/test_authority.py", "tests/authorization/task_queues/test_contracts.py", "tests/authorization/task_queues/test_transactions.py", @@ -445,14 +461,6 @@ class TestLane: "tests/tasks/test_assignment_invalidation_races.py", "tests/tasks/test_contribution_claim_races.py", "tests/tasks/test_submission_lineage.py", - "tests/checkers/execution/test_results.py", - "tests/checkers/execution/test_execution.py", - "tests/checkers/execution/test_coordination.py", - "tests/checkers/execution/test_concurrency.py", - "tests/checkers/execution/test_storage.py", - "tests/checkers/execution/test_migration.py", - "tests/checkers/execution/test_material_lineage.py", - "tests/checkers/execution/test_material_migration.py", "tests/checkers/post_submit/test_catalogue.py", "tests/checkers/post_submit/test_compiled_policy.py", "tests/checkers/post_submit/test_configuration.py", @@ -467,11 +475,6 @@ class TestLane: "tests/checkers/test_effective_intake_rules.py", "tests/test_default_pre_submit_execution.py", "tests/test_approved_guide_intake.py", - "tests/test_submission_evaluation_capacity.py", - "tests/test_post_submit_materialization.py", - "tests/test_post_submit_selection.py", - "tests/test_checker_output_custody.py", - "tests/test_checker_output_storage.py", "tests/test_pre_submit_attempt_recovery.py", "tests/test_pre_submit_attempt_contracts.py", "tests/test_pre_submit_attempt_authority_integration.py", @@ -509,8 +512,7 @@ class TestLane: "tests/test_celery_observability.py", ) -# Routing PREP proofs stay together on task C, which has measured headroom -# after task A reached the unchanged execution cap. +# Routing PREP proofs share the three-way project-policy partition. ROUTING_AUTH_PREPARATION_MODULES = ( "tests/authorization/post_submit_routing/test_contracts.py", "tests/authorization/post_submit_routing/test_prepared.py", @@ -518,7 +520,10 @@ class TestLane: PARTITION_GROUPS = ( (PARTITIONED_SHARED_LANES, SHARED_FOUNDATION_MODULES), - (PARTITIONED_PROJECT_LANES, PROJECT_MODULES), + ( + PARTITIONED_PROJECT_LANES, + PROJECT_MODULES + CHECKER_DELIVERY_MODULES + ROUTING_AUTH_PREPARATION_MODULES, + ), (PARTITIONED_TASK_LANES, TASK_MODULES), ) PARTITION_LANES_BY_MODULE = { @@ -557,10 +562,12 @@ class TestLane: ADMIN_RUNNER_MODULE, ), ), - *(TestLane(name, PROJECT_MODULES) for name in PARTITIONED_PROJECT_LANES), - *(TestLane( - name, - TASK_MODULES - + (ROUTING_AUTH_PREPARATION_MODULES if name == "task_lifecycle_c" else ()), - ) for name in PARTITIONED_TASK_LANES), + *( + TestLane( + name, + PROJECT_MODULES + CHECKER_DELIVERY_MODULES + ROUTING_AUTH_PREPARATION_MODULES, + ) + for name in PARTITIONED_PROJECT_LANES + ), + *(TestLane(name, TASK_MODULES) for name in PARTITIONED_TASK_LANES), ) diff --git a/backend/tests/test_ci_lane_catalogue.py b/backend/tests/test_ci_lane_catalogue.py index 97703dae8..1701603b1 100644 --- a/backend/tests/test_ci_lane_catalogue.py +++ b/backend/tests/test_ci_lane_catalogue.py @@ -42,6 +42,47 @@ def test_committed_lanes_cover_recursive_inventory_exactly_once() -> None: ) +def test_checker_delivery_and_routing_preparation_partition_every_node_once() -> None: + delivery = { + "tests/checkers/execution/test_results.py", + "tests/checkers/execution/test_execution.py", + "tests/checkers/execution/test_coordination.py", + "tests/checkers/execution/test_concurrency.py", + "tests/checkers/execution/test_storage.py", + "tests/checkers/execution/test_migration.py", + "tests/checkers/execution/test_material_lineage.py", + "tests/checkers/execution/test_material_migration.py", + "tests/test_post_submit_materialization.py", + "tests/test_post_submit_selection.py", + "tests/test_submission_evaluation_capacity.py", + "tests/test_checker_output_custody.py", + "tests/test_checker_output_storage.py", + } + assert set(catalogue.CHECKER_DELIVERY_MODULES) == delivery + groups = ( + delivery, + set(catalogue.ROUTING_AUTH_PREPARATION_MODULES), + ) + for modules in groups: + assert not modules & set(catalogue.TASK_MODULES) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[module] == catalogue.PARTITIONED_PROJECT_LANES + for module in modules + ) + for module in modules: + assert ( + tuple(lane.name for lane in LANES if module in lane.modules) + == catalogue.PARTITIONED_PROJECT_LANES + ) + nodes = [f"{module}::test_custody[{index}]" for module in modules for index in range(256)] + manifest = runner.build_manifest("a" * 40, list(reversed(nodes))) + assert len(manifest["nodes"]) == len(nodes) + assert {row["nodeid"] for row in manifest["nodes"]} == set(nodes) + assert {row["lane"] for row in manifest["nodes"]} == set( + catalogue.PARTITIONED_PROJECT_LANES + ) + + def test_measured_hotspots_have_explicit_semantic_owners() -> None: """Keep lane balance tied to subsystem ownership and measured schema cost.""" modules_by_lane = {lane.name: set(lane.modules) for lane in LANES} @@ -50,7 +91,9 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: modules_by_lane["project_lifecycle_a"] == modules_by_lane["project_lifecycle_b"] == modules_by_lane["project_lifecycle_c"] - == { + == set(catalogue.CHECKER_DELIVERY_MODULES) + | set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) + | { "tests/tasks/evaluation_delivery/test_custody.py", "tests/tasks/evaluation_delivery/test_delivery.py", "tests/tasks/evaluation_delivery/test_isolation.py", @@ -193,7 +236,7 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: assert ( modules_by_lane["task_lifecycle_a"] == modules_by_lane["task_lifecycle_b"] - == modules_by_lane["task_lifecycle_c"] - set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) + == modules_by_lane["task_lifecycle_c"] == { "tests/authorization/submission_history/test_reads.py", "tests/authorization/submission_history/test_privacy.py", @@ -232,7 +275,6 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/authorization/task_audit_evidence/test_authority.py", "tests/authorization/task_audit_evidence/test_history.py", "tests/authorization/task_audit_evidence/test_transactions_concurrency.py", - "tests/tasks/test_management_queue.py", "tests/tasks/test_task_detail.py", "tests/tasks/test_work_context.py", @@ -250,14 +292,6 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/tasks/test_payment_policy_migration.py", "tests/tasks/test_contribution_claim_races.py", "tests/tasks/test_submission_lineage.py", - "tests/checkers/execution/test_results.py", - "tests/checkers/execution/test_execution.py", - "tests/checkers/execution/test_coordination.py", - "tests/checkers/execution/test_concurrency.py", - "tests/checkers/execution/test_storage.py", - "tests/checkers/execution/test_migration.py", - "tests/checkers/execution/test_material_lineage.py", - "tests/checkers/execution/test_material_migration.py", "tests/checkers/post_submit/test_catalogue.py", "tests/checkers/post_submit/test_compiled_policy.py", "tests/checkers/post_submit/test_configuration.py", @@ -272,11 +306,6 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/test_checkers.py", "tests/test_default_pre_submit_execution.py", "tests/test_approved_guide_intake.py", - "tests/test_submission_evaluation_capacity.py", - "tests/test_post_submit_materialization.py", - "tests/test_post_submit_selection.py", - "tests/test_checker_output_custody.py", - "tests/test_checker_output_storage.py", "tests/test_pre_submit_attempt_recovery.py", "tests/test_pre_submit_attempt_contracts.py", "tests/test_pre_submit_attempt_authority_integration.py", @@ -346,9 +375,9 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/reviews/acceptance/test_storage.py", "tests/reviews/acceptance/test_migration.py", runner.ADMIN_RUNNER_MODULE, - } | static_contracts | post_submit_storage_contracts | set(catalogue.OBSERVABILITY_MODULES) | set(catalogue.TASK_ROUTING_REQUEST_MODULES) == modules_by_lane[ - "schema_contracts" - ] + } | static_contracts | post_submit_storage_contracts | set( + catalogue.OBSERVABILITY_MODULES + ) | set(catalogue.TASK_ROUTING_REQUEST_MODULES) == modules_by_lane["schema_contracts"] assert { "tests/authorization/admin_access/test_bootstrap_cli.py", "tests/authorization/admin_access/test_api_journey.py", @@ -773,7 +802,9 @@ def test_routing_request_proofs_use_schema_lane_with_measured_headroom(): } assert set(catalogue.TASK_ROUTING_REQUEST_MODULES) == expected for lane in LANES: - assert set(lane.modules) & expected == (expected if lane.name == "schema_contracts" else set()) + assert set(lane.modules) & expected == ( + expected if lane.name == "schema_contracts" else set() + ) assert not expected & set(catalogue.PARTITION_LANES_BY_MODULE) @@ -785,19 +816,26 @@ def test_observability_proofs_use_schema_lane_with_measured_headroom(): } assert set(catalogue.OBSERVABILITY_MODULES) == expected for lane in LANES: - assert set(lane.modules) & expected == (expected if lane.name == "schema_contracts" else set()) + assert set(lane.modules) & expected == ( + expected if lane.name == "schema_contracts" else set() + ) assert not expected & set(catalogue.PARTITION_LANES_BY_MODULE) -def test_routing_authorization_proofs_run_once_on_task_c(): +def test_routing_authorization_proofs_use_the_project_partition(): expected = { "tests/authorization/post_submit_routing/test_contracts.py", "tests/authorization/post_submit_routing/test_prepared.py", } assert set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) == expected for lane in LANES: - assert set(lane.modules) & expected == (expected if lane.name == "task_lifecycle_c" else set()) - assert not expected & set(catalogue.PARTITION_LANES_BY_MODULE) + assert set(lane.modules) & expected == ( + expected if lane.name in catalogue.PARTITIONED_PROJECT_LANES else set() + ) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[module] == catalogue.PARTITIONED_PROJECT_LANES + for module in expected + ) def test_shared_acceptance_owner_proofs_are_in_partitioned_project_lanes(): @@ -809,7 +847,10 @@ def test_shared_acceptance_owner_proofs_are_in_partitioned_project_lanes(): "tests/tasks/accepted_effects/test_postgresql.py", } assert expected <= set(catalogue.PROJECT_MODULES) - assert all(catalogue.PARTITION_LANES_BY_MODULE[path] == catalogue.PARTITIONED_PROJECT_LANES for path in expected) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[path] == catalogue.PARTITIONED_PROJECT_LANES + for path in expected + ) def test_evaluation_custody_proofs_use_project_lane_headroom(): @@ -817,7 +858,10 @@ def test_evaluation_custody_proofs_use_project_lane_headroom(): "tests/tasks/post_submit_routing/test_evaluation_guard.py", "tests/tasks/post_submit_routing/test_evaluation_currentness.py", "tests/tasks/post_submit_routing/test_review_admission_currentness.py", - "tests/tasks/post_submit_routing/test_source_preparation.py", + "tests/tasks/post_submit_routing/test_source_preparation.py", } assert expected <= set(catalogue.PROJECT_MODULES) - assert all(catalogue.PARTITION_LANES_BY_MODULE[path] == catalogue.PARTITIONED_PROJECT_LANES for path in expected) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[path] == catalogue.PARTITIONED_PROJECT_LANES + for path in expected + ) diff --git a/backend/tests/test_ci_test_lanes.py b/backend/tests/test_ci_test_lanes.py index 18726aa0f..65cfb1e3b 100644 --- a/backend/tests/test_ci_test_lanes.py +++ b/backend/tests/test_ci_test_lanes.py @@ -98,6 +98,12 @@ def test_lane_command_uses_exact_nodes_and_isolation_contract(tmp_path: Path) -> assert command[-1] == nodes[0] assert "--cov=app" in command assert "--cov-report=" in command + assert "--durations=0" in command + assert "--durations=25" not in command + + admin = runner.admin_runner_command([f"{runner.ADMIN_RUNNER_MODULE}::test_one"]) + assert "--durations=0" in admin + assert "--durations=25" not in admin def test_lane_environment_uses_private_evidence_and_coverage(tmp_path: Path) -> None: diff --git a/docker/minio/README.md b/docker/minio/README.md index fc2dad15e..6ad9fabf7 100644 --- a/docker/minio/README.md +++ b/docker/minio/README.md @@ -28,14 +28,19 @@ requires network access and Go compilation resources; do not launch it on an already memory-constrained workstation. Subsequent builds reuse Docker layers. -Backend CI builds or restores one image cache keyed by the exact Git commit, -this directory's contents and runner platform. Older PR commits cannot supply a -cached executable to a new commit; retries of the same commit can reuse its -image. CI verifies server startup, then supplies a checksummed image -artifact to the existing lanes and aggregate job. Jobs never substitute a mock -storage provider. A missing build, artifact or health check fails verification. -The source-image artifact is independent of test/coverage evidence and cannot -make a failed test lane pass. +Backend CI builds or restores one image tar cache keyed by a cache version, the +runner OS/architecture, this directory's contents and the Backend workflow that +owns the build command. Pull requests restore only. A trusted `main` push saves +a miss after the image passes its version command and live health probe. The +same exact cache identity can therefore be reused across commits; a Docker +context, workflow or platform change misses it. + +Every run still loads and probes the image, then supplies a newly checksummed, +Git-SHA-and-attempt-named artifact to the existing lanes and aggregate job. +Those consumers verify the checksum before loading it. Jobs never substitute a +mock storage provider. A missing build, artifact or health check fails +verification. The source-image artifact is independent of test/coverage +evidence and cannot make a failed test lane pass. When updating upstream source, update the commit, archive checksum, provenance and relevant build pins together. Require a fresh image build, health check and diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index ab85af6f4..24c9993d7 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -136,8 +136,12 @@ This is not a production configuration or proof of host-power-loss durability: An exhausted mount fails the job; it does not silently change storage or skip tests. The `project_lifecycle_a`, `project_lifecycle_b`, and `project_lifecycle_c` lanes -partition PROJECT nodes; `task_lifecycle_a`, `task_lifecycle_b`, and `task_lifecycle_c` use the same -deterministic partition mechanism for TASK and checker nodes. The single `schema_contracts` lane owns all baseline/PostgreSQL schema, reset and +partition PROJECT nodes, checker execution, materialization, evaluation capacity, +output custody and routing AUTH preparation beside initial dispatch/delivery. +All use the existing exact-node hash across the three lanes: placing the entire +checker delivery group exclusively on project A exhausted its execution cap. +`task_lifecycle_a`, `task_lifecycle_b`, and `task_lifecycle_c` use the same +deterministic partition mechanism for the remaining TASK and checker nodes. The single `schema_contracts` lane owns all baseline/PostgreSQL schema, reset and isolated-runner contracts. The `shared_foundations_a` and `shared_foundations_b` lanes deterministically partition exact node IDs from the remaining authorization, artifact, API, and @@ -220,11 +224,14 @@ coverage tampering before coverage combination. - API contract or evidence-integrity failure: the required job remains failed; lane completion cannot compensate. A lower coverage percentage is not a failure. -On the same exact head, rerun failed lanes (and their dependent final job), or -rerun only the final job when the lane evidence already passed. Successful lanes -not rerun retain their previous attempt's evidence; a rerun lane's newest bundle -must independently pass all existing checks. A failed, cancelled or skipped -required job still blocks fan-in. Never edit or upload evidence manually. +After diagnosing a transient failure on the same exact head, rerun only failed +jobs and their dependents with `gh run rerun RUN_ID --failed`; avoid the bare +whole-workflow rerun command for that case. Successful lanes not rerun retain +their previous attempt's evidence, while a rerun lane's newest bundle must +independently pass all existing checks. The required aggregate job selects those +bundles and revalidates the complete exact-head union. A failed, cancelled or +skipped required job still blocks fan-in. Never edit or upload evidence +manually. A repeated timeout requires diagnosis rather than retries until green. Review submission or dismissal does not rerun Backend because it does not change the tested tree. A new PR commit starts a new run and cancels the superseded same-PR run. Every new commit requires complete evidence because @@ -240,6 +247,13 @@ does not override otherwise passing correctness, custody, service-contract, API, and complete-execution gates. Coverage is diagnostic only. Never skip nodes or add a silent fallback to meet the target. +A source push or base-branch change creates a different current tree and +requires fresh CI; prior same-head retry evidence cannot be carried forward. +The content-addressed MinIO cache can remove the measured 145-second image +rebuild when its exact Docker context, Backend workflow and runner platform are +unchanged. It does not solve observed 17-minute queue waits or 20-minute lanes +and does not promise an eight-minute Backend completion time. + ## Retired changed-scope behavior mutation The hosted `Behavior Mutation Gate` is temporarily removed. Its callable-wide diff --git a/scripts/test_lightweight_agent_gates.py b/scripts/test_lightweight_agent_gates.py index 493abbfe5..6c0b672e1 100644 --- a/scripts/test_lightweight_agent_gates.py +++ b/scripts/test_lightweight_agent_gates.py @@ -228,12 +228,39 @@ def test_minio_source_image_is_built_once_and_shared_without_bypassing_lanes(sel image_job = workflow.split("\n minio-image:\n", 1)[1].split("\n auth-boundary-preflight:\n", 1)[0] self.assertEqual(workflow.count('docker build --tag "${MINIO_IMAGE}" docker/minio'), 1) self.assertNotIn("quay.io/minio", workflow) - self.assertIn("hashFiles('docker/minio/**')", image_job) + self.assertEqual( + image_job.count("hashFiles('docker/minio/**', '.github/workflows/backend.yml')"), + 2, + ) self.assertIn( - "key: minio-source-v1-${{ github.sha }}-${{ runner.os }}-${{ runner.arch }}-", + "key: minio-source-v2-${{ runner.os }}-${{ runner.arch }}-", image_job, ) + cache_step = image_job.split( + " - name: Restore exact source image from trusted cache\n", 1 + )[1].split("\n - name:", 1)[0] + self.assertIn( + "uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830", + cache_step, + ) + self.assertNotIn("github.sha", cache_step) self.assertNotIn("restore-keys:", image_job) + save_step = image_job.split( + " - name: Save verified source image to trusted cache\n", 1 + )[1].split("\n - uses:", 1)[0] + self.assertIn( + "if: github.event_name == 'push' && github.ref == 'refs/heads/main' " + "&& steps.cache.outputs.cache-hit != 'true'", + save_step, + ) + self.assertIn( + "uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830", + save_step, + ) + self.assertLess( + image_job.index(" - name: Verify the cached or freshly built provider"), + image_job.index(" - name: Save verified source image to trusted cache"), + ) self.assertIn("minio-source-${GITHUB_SHA}-${GITHUB_RUN_ATTEMPT}", image_job) self.assertIn("artifact: ${{ steps.identity.outputs.artifact }}", image_job) self.assertIn("sha256sum minio.tar > minio.tar.sha256", image_job) @@ -249,6 +276,60 @@ def test_minio_source_image_is_built_once_and_shared_without_bypassing_lanes(sel self.assertIn('docker load --input "${RUNNER_TEMP}/minio-image/minio.tar"', job) self.assertIn('"${MINIO_IMAGE}" server /data --address :9000', job) + def test_backend_python_cache_keeps_fresh_installs_and_exact_inputs(self) -> None: + workflow = Path(".github/workflows/backend.yml").read_text(encoding="utf-8") + + key = ( + "backend-pip-v1-${{ runner.os }}-${{ runner.arch }}-py312-" + "${{ hashFiles('backend/pyproject.toml', '.github/workflows/backend.yml') }}" + ) + self.assertEqual(workflow.count("Restore exact backend pip downloads from trusted cache"), 3) + self.assertEqual(workflow.count(f"key: {key}"), 4) + self.assertEqual( + workflow.count( + "uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830" + ), + 4, + ) + self.assertEqual( + workflow.count("uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830"), + 2, + ) + install = 'python -m pip install -e ".[dev,agents]"' + cache_env = 'echo "PIP_CACHE_DIR=${RUNNER_TEMP}/backend-pip-cache" >> "${GITHUB_ENV}"' + job_blocks = ( + workflow.split("\n auth-boundary-preflight:\n", 1)[1].split("\n lanes:\n", 1)[0], + workflow.split("\n lanes:\n", 1)[1].split("\n cli-public-contract:\n", 1)[0], + workflow.split("\n test:\n", 1)[1], + ) + for job in job_blocks: + self.assertEqual(job.count("Restore exact backend pip downloads from trusted cache"), 1) + self.assertEqual(job.count(install), 1) + self.assertEqual(job.count(cache_env), 1) + self.assertLess(job.index("Restore exact backend pip downloads"), job.index(cache_env)) + self.assertLess(job.index(cache_env), job.index(install)) + self.assertEqual(workflow.count(install), 3) + self.assertEqual(workflow.count(cache_env), 3) + root_env = workflow.split("\nenv:\n", 1)[1].split("\njobs:\n", 1)[0] + self.assertNotIn("runner.", root_env) + pip_save = workflow.split( + " - name: Save verified backend pip downloads to trusted cache\n", 1 + )[1].split("\n lanes:\n", 1)[0] + self.assertIn( + "if: github.event_name == 'push' && github.ref == 'refs/heads/main' " + "&& steps.pip-cache.outputs.cache-hit != 'true'", + pip_save, + ) + preflight = workflow.split("\n auth-boundary-preflight:\n", 1)[1].split( + "\n lanes:\n", 1 + )[0] + self.assertLess( + preflight.index(" - name: Validate module, AUTH, and test boundaries"), + preflight.index(" - name: Save verified backend pip downloads to trusted cache"), + ) + self.assertNotIn("restore-keys:", workflow) + self.assertNotIn(".venv", workflow) + def test_parallel_preflight_and_lanes_fail_closed_at_fan_in(self) -> None: workflow = Path(".github/workflows/backend.yml").read_text(encoding="utf-8") lanes = workflow.split("\n lanes:\n", 1)[1].split("\n test:\n", 1)[0]