bench def: tpc6 image retargeted to 8d13104e (same binary content; ad… #457
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Enforced gate for the sweep engine (lib/sweep.sh) — the component that produces EVERY published | |
| # throughput number. Runs lib/sweep_peak_test.sh on push, so a change that re-quantises fast gateways | |
| # onto doubling rungs (the exact off-by-one class the test guards, incl. the H5 relative-TOL fix) or | |
| # regresses the adaptive/prior-seeded path (M5) or the sweep_c1 honesty gate (M6) fails CI instead of | |
| # shipping silently (audit H6). Fast (stubbed loadgen/rig, no network, seconds). | |
| name: bench-tests | |
| on: | |
| push: | |
| paths: | |
| - "engine/**" | |
| - "Cargo.toml" | |
| - "Cargo.lock" | |
| - ".cargo/config.toml" | |
| - "mock/**" | |
| - "gateways/**" | |
| - "lib/**" | |
| - "run-on-ec2.sh" | |
| # The audit gate's own subjects. Without these, an edit that broke bench-audit.py (the exact | |
| # wrong-key-path class its header describes) or a results push that violated every invariant | |
| # never triggered the job that exists to catch it - the same paths-as-decoration defect the | |
| # rig-delivery job below documents for itself. | |
| - "bench-audit.py" | |
| - "verify-latency.py" | |
| - "verify-frontier.py" | |
| - "verify-turnover.py" | |
| - "audit-every-metric.py" | |
| - "verify_tools_test.py" | |
| - "bench-audit_test.py" | |
| - "bench-dashboard.py" | |
| - "bench-dashboard_test.py" | |
| - "bench-dashboard_coverage_test.py" | |
| - "bench-cost.py" | |
| - "bench-cost_test.py" | |
| - "results/**" | |
| # The audit parses site/check-consistency.mjs to prove the two copies of C6_GROSS_PCT still | |
| # agree, and the `site` job's suites IMPORT it (its C1-C5 lints are pure exported functions, | |
| # driven from site/test.mjs against both the real bundle and synthetic violating source). | |
| # Without this path, the ONE pull request that cross-check exists for - a tune of the JS side | |
| # alone - would not run it. | |
| - "site/check-consistency.mjs" | |
| # The site suites' own subjects. site/test.mjs and site/test-coverage.mjs run in the `site` | |
| # job below, which builds its own bundle from this checkout - so the surfaces they read | |
| # (app.js), the envelope they read it through (seal.mjs), the producer that seals it | |
| # (gen-data.mjs) and the suites themselves are all subjects of this gate. Without these lines | |
| # a change to app.js alone - the file the whole board renders through - triggered NO job that | |
| # runs a site test, which is the same paths-as-decoration defect the rig-delivery job below | |
| # documents for itself. | |
| - "site/app.js" | |
| - "site/seal.mjs" | |
| - "site/gen-data.mjs" | |
| - "site/test.mjs" | |
| - "site/test-coverage.mjs" | |
| - ".github/workflows/bench-tests.yml" | |
| pull_request: | |
| paths: | |
| - "engine/**" | |
| - "Cargo.toml" | |
| - "Cargo.lock" | |
| - ".cargo/config.toml" | |
| - "mock/**" | |
| - "gateways/**" | |
| - "lib/**" | |
| - "run-on-ec2.sh" | |
| # The audit gate's own subjects. Without these, an edit that broke bench-audit.py (the exact | |
| # wrong-key-path class its header describes) or a results push that violated every invariant | |
| # never triggered the job that exists to catch it - the same paths-as-decoration defect the | |
| # rig-delivery job below documents for itself. | |
| - "bench-audit.py" | |
| - "verify-latency.py" | |
| - "verify-frontier.py" | |
| - "verify-turnover.py" | |
| - "audit-every-metric.py" | |
| - "verify_tools_test.py" | |
| - "bench-audit_test.py" | |
| - "bench-dashboard.py" | |
| - "bench-dashboard_test.py" | |
| - "bench-dashboard_coverage_test.py" | |
| - "bench-cost.py" | |
| - "bench-cost_test.py" | |
| - "results/**" | |
| # The audit parses site/check-consistency.mjs to prove the two copies of C6_GROSS_PCT still | |
| # agree, and the `site` job's suites IMPORT it (its C1-C5 lints are pure exported functions, | |
| # driven from site/test.mjs against both the real bundle and synthetic violating source). | |
| # Without this path, the ONE pull request that cross-check exists for - a tune of the JS side | |
| # alone - would not run it. | |
| - "site/check-consistency.mjs" | |
| # The site suites' own subjects - see the push block above for why each one is here. | |
| - "site/app.js" | |
| - "site/seal.mjs" | |
| - "site/gen-data.mjs" | |
| - "site/test.mjs" | |
| - "site/test-coverage.mjs" | |
| - ".github/workflows/bench-tests.yml" | |
| workflow_dispatch: | |
| permissions: | |
| contents: read | |
| jobs: | |
| engine: | |
| # The Rust engine replaced the shell measurement core entirely (no shell measurement path is left | |
| # to run a differential test against); its own tests are the reason for the port as much as its | |
| # types are, so CI runs them here - an untested test suite is decoration. | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: dtolnay/rust-toolchain@stable | |
| with: | |
| components: clippy, rustfmt | |
| # rustfmt has been INSTALLED here since this job was written and never run, so the workspace | |
| # drifted out of shape and the drift only surfaced when someone ran `cargo fmt` and watched it | |
| # rewrite 23 files they had not touched. That turns every future formatting run into a diff | |
| # that buries the change it travelled with. Gating it is what keeps a `cargo fmt` a no-op. | |
| - name: Formatting | |
| run: cargo fmt --all --check | |
| - name: Build | |
| run: cargo build --workspace --all-targets | |
| - name: Clippy (denies unwrap/expect/panic outside tests) | |
| run: cargo clippy --workspace --all-targets -- -D warnings | |
| - name: Unit and property tests | |
| run: cargo test --workspace | |
| # Cross-compiling to the arm64 boxes is part of the contract: the engine ships as a prebuilt | |
| # static binary rather than being built on a box, so a break here is a shipping break. | |
| - name: Cross-compile the shipped arm64 binary | |
| run: | | |
| rustup target add aarch64-unknown-linux-musl | |
| cargo build --release --target aarch64-unknown-linux-musl --bin otb | |
| file target/aarch64-unknown-linux-musl/release/otb | |
| audit: | |
| # THE CROSS-METRIC INVARIANTS, AND THE PROOF THEY CAN FAIL. | |
| # | |
| # Auditing a run used to mean throwaway python in a scratch directory and an opinion at the end. | |
| # An opinion cannot be handed to anyone, does not survive the next run, and answers "is this data | |
| # good?" with "the checks I happened to think of, passed". `bench-audit.py` is those checks as a | |
| # program; this job is what makes "the audit is done" mean green rather than mean me. | |
| # | |
| # The self-test runs FIRST and is the more important of the two. The audit's own first draft | |
| # shipped a check that read the wrong JSON path, found nothing, and reported PASS on a board that | |
| # violates it on every cell - the same species as `transient_budget()` called by nothing and 27 | |
| # site tests asserting against an empty board. Every check must reject a cell built to violate it | |
| # and accept a clean one, or it is guarding nothing. | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Every audit check can fail (and does not fail a clean cell) | |
| run: python3 bench-audit_test.py | |
| # The dashboard's test ran NOWHERE until this line - a test nothing executes reads as covered | |
| # (this file's own words, one job down). | |
| - name: Dashboard self-test | |
| run: python3 bench-dashboard_test.py | |
| # The coverage siblings: the fail-CLOSED served counter (the "10/8 served, ETA zero" | |
| # regression), the phantom-gateway filter, the midnight-crossing clock, the phase-weighted | |
| # early ETA. Written the same day as the dashboard fixes and, like the self-test above it, | |
| # executed by nothing until this line. | |
| - name: Dashboard coverage siblings (served counter, ETA clocks, fanout filter) | |
| run: python3 bench-dashboard_coverage_test.py | |
| # bench-cost.py's snapshot selection had no test at all, and every wrong answer it can give is | |
| # silent - an older snapshot, another gateway's, or one from a different engine all render a | |
| # plausible cost table. bench-cost.py was already a path trigger here while nothing ran its | |
| # test, so this job triggered on the subject and then checked nothing about it. | |
| - name: Cost-report snapshot selection (newest-per-gateway, gateway + engine filters) | |
| run: python3 bench-cost_test.py | |
| # Runs against the newest engine's committed snapshots, so a board that violates an invariant | |
| # is red until it is re-measured. Expected to be red on any board published before the engine | |
| # stopped discarding the p99 of its own throughput windows. | |
| - name: Published board holds every cross-metric invariant | |
| run: python3 bench-audit.py | |
| # THE TWO AUDITORS THAT HAD NO GATE. verify-latency.py and audit-every-metric.py were written | |
| # during a live run and shipped with no test file and no CI invocation - which is the same shape | |
| # as the defects they exist to catch. A typo'd field name that always resolved to None would | |
| # leave their counters at 0 and print PASS on every board forever, and nothing would notice. | |
| # This runs their red fixtures, so each check is shown to fail on the violation it names. | |
| - name: Auditor red fixtures (all four external auditors) | |
| run: python3 verify_tools_test.py | |
| rig-delivery: | |
| # The launcher's file fetch had no gate, and its one defect cost a full 14-box run: a raw download | |
| # writes whatever the umask says, so build.sh arrived non-executable and all three source-built | |
| # entrants died at their build step while the eleven container entrants ran fine. The test that | |
| # reproduces it was written the same day and then never ran anywhere - this workflow's paths | |
| # covered engine/ and mock/ but not lib/ or run-on-ec2.sh, so the guard sat in the tree as | |
| # decoration. A test nothing executes is worse than no test: it reads as covered. | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: The fetch lands files with the mode the commit recorded | |
| run: bash lib/fetch_modes_test.sh | |
| mock-shape: | |
| # The mock must never bottleneck and must serve every dialect identically; its per-dialect | |
| # request_shape_ok is what the matrix's leg-3 body_ok trusts. Gate it so the bedrock/cohere | |
| # tightening (R3-M6 — reject an unconverted OpenAI body) can never silently regress. | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: dtolnay/rust-toolchain@stable | |
| - name: mock unit tests (request_shape_ok per dialect + routing) | |
| run: cargo test --manifest-path mock/Cargo.toml --tests | |
| site: | |
| # THE SITE SUITES, IN A JOB THAT ACTUALLY REACHES THEM. | |
| # | |
| # Both JS suites lived only in cf-pages.yml, behind `node site/gen-data.mjs`. On a cleared board | |
| # (results/snapshots/ empty - a legitimate state, see render-charts.yml's own note) gen-data | |
| # hard-fails with FRESHNESS FAILURE, the deploy job dies there, and `node site/test.mjs` is | |
| # never reached. So the entire JS side of the board has been gating nothing: green cf-pages runs | |
| # meant "the bundle built", and red ones meant "the bundle did not build" - in neither case did | |
| # anyone learn whether the table, drawer, compare and charts still agree. | |
| # | |
| # Here they run with no bundle build in front of them. site/test.mjs runs gen-data ITSELF into a | |
| # temp dir from this checkout, so it never needed the deploy's build to have happened; it was | |
| # only ever ORDERED behind it. test-coverage.mjs needs no bundle at all - it drives the URL | |
| # codec and the per-surface envelope readers directly - so it runs FIRST and reports even when | |
| # the board cannot be built, which is exactly the case that used to report nothing. | |
| # | |
| # cf-pages keeps its own copy of test.mjs as the pre-deploy guard: that one must gate the thing | |
| # being shipped. This job is what makes the suite gate the SOURCE. | |
| runs-on: ubuntu-latest | |
| # SAME PUBLISHING POLICY THE DEPLOY USES. site/test.mjs builds its OWN bundle from this checkout, | |
| # so without this it would gate a differently-built board than the one cf-pages ships - the suite | |
| # would fail C8 on a mix that the deployed board never contains. See the long note in | |
| # cf-pages.yml for why n/a beats both re-measuring the field and overriding the guard. | |
| env: | |
| OTB_SINGLE_ENGINE: "1" | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: actions/setup-node@v4 | |
| with: | |
| node-version: 22 | |
| # No bundle required: enumerated URL-codec fixed points + one-envelope-one-story across | |
| # table/drawer/popup/rank. Passes on an empty board, so it is the first thing to speak. | |
| - name: Site coverage siblings (URL codec fixed points, one envelope one story) | |
| run: node site/test-coverage.mjs | |
| # Builds its own bundle from this checkout, then holds table == drawer == compare == charts | |
| # and the C1-C5 structural invariants with their RED-before proofs. | |
| - name: Consistency guard (table == drawer == compare == charts) | |
| run: node site/test.mjs |