ci(bench): measure what the macOS mimalloc pin costs Linux and Windows (#448)

.github/workflows/allocator-bench.yml

@@ -0,0 +1,166 @@
+# TEMPORARY — delete this file once #448 has its measurement.
+#
+# This is an INSTRUMENT, not a gate. It is `workflow_dispatch`-only: it never
+# fires on a push, a PR or a schedule, so it cannot fail anyone's build and
+# cannot silently become part of the required set. It is inventoried in
+# `ags-wiki/design/reliquary.json` for exactly that reason — "temporary" survives
+# by nobody remembering, and the reliquary is this repo's answer to that.
+#
+# ## What it measures, and why there is no macOS leg
+#
+# The mimalloc v2 pin (#294/#301) exists for a **macOS-specific** fault: v3
+# defaulted to Apple's fixed TLS slots 108/109, so two co-resident v3 instances
+# shared them. The pin is not platform-gated, so Linux and Windows carry a macOS
+# fix and nobody has measured what it costs them (#448).
+#
+# That question is WITHIN-platform — same box, three binaries — so a macOS
+# baseline would answer a different question. Comparing Linux-v3 against
+# macOS-v2 measures the machines.
+#
+# ## The two legs, and why they are not the same kind of runner
+#
+# Linux runs on the self-hosted pool: a dedicated box is the right instrument
+# for a timing comparison, where GitHub's shared vCPUs are not. Windows has no
+# self-hosted runner, so it takes `windows-2022` — which means the Windows
+# numbers are noisier, and the harness's own drift probe is what says whether
+# they are readable at all rather than a reader having to guess.
+#
+# The harness (`tools/bench/allocator_ab.py`) carries the method: A/B/A ordering,
+# a system-allocator control arm that must separate or the run proves nothing,
+# and a distinct-binary check so a failed feature swap cannot report three
+# readings of the same build.
+
+name: "allocator-bench (temporary, #448)"
+
+on:
+  workflow_dispatch:
+    inputs:
+      reps:
+        description: "Timed runs per arm (more = steadier, slower)"
+        required: false
+        default: "15"
+      size:
+        description: "Fixture size for `forge scale` (e.g. 10MB, 25MB)"
+        required: false
+        default: "25MB"
+      windows_runner:
+        description: "Windows leg: github-hosted, or the self-hosted box once it is registered on THIS repo"
+        required: false
+        default: github-hosted
+        type: choice
+        options:
+          - github-hosted
+          - self-hosted
+
+permissions:
+  contents: read
+
+jobs:
+  bench:
+    name: ${{ matrix.name }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # The self-hosted pool, by the same label set ci.yml uses.
+          - name: linux (self-hosted)
+            runner: '["self-hosted","linux","x64","laterite-linux"]'
+          # Windows is switchable, because the good runner is not reachable from
+          # here YET. A self-hosted Windows box exists — `ags5-win11-runner` —
+          # but it is registered on the PRIVATE satellite, and `niko86` is a User
+          # account rather than an Organization, so there are no runner groups to
+          # share it through: a runner belongs to the repo it was configured
+          # against. Asking for those labels here does not fail, it QUEUES, until
+          # the job times out an hour later.
+          #
+          # Registering the same machine a second time against this repo is the
+          # fix (one box can host several runner services). Label it
+          # `laterite-windows`, to match `laterite-linux` and because
+          # `ags5-win11` carries a name that outlived what this repo holds. Then
+          # dispatch with `windows_runner: self-hosted` — no edit needed here.
+          #
+          # Until then the default is `windows-2022`, which release.yml already
+          # builds the Windows wheel on. Its numbers are noisier; the harness's
+          # own noise floor is what says whether they are readable.
+          #
+          # One line on purpose: a folded scalar here keeps its newlines inside
+          # the `${{ }}` (the continuations are more-indented, so YAML does not
+          # fold them), and this is the wrong place to find out whether the
+          # expression parser minds — a bad runner label does not error, it
+          # queues until the timeout.
+          - name: windows (${{ inputs.windows_runner }})
+            runner: "${{ inputs.windows_runner == 'self-hosted' && '[\"self-hosted\",\"windows\",\"x64\",\"laterite-windows\"]' || '\"windows-2022\"' }}"
+    runs-on: ${{ fromJSON(matrix.runner) }}
+    timeout-minutes: 60
+
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      # Installed rather than assumed. Both images probably carry a toolchain,
+      # but "probably" is the wrong footing for a measurement: an arm built by a
+      # different compiler than its sibling is not a comparison, and the pin is
+      # what makes the three arms the same build in every respect but the
+      # allocator.
+      - name: Rust toolchain
+        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
+
+      # Python comes from uv rather than from the box. The Windows runner has no
+      # interpreter on PATH, and "install Python on the runner" would make the
+      # job depend on hand-configured machine state — the thing a self-healing
+      # pool exists to avoid, and something the github-hosted fallback leg would
+      # not share. This is also the house pattern: nothing here uses
+      # `setup-python`; every workflow gets its interpreter from uv, and
+      # release.yml already proves that path on a self-hosted Windows VM.
+      #
+      # The harness is stdlib-only on purpose, so `--no-project` is enough — no
+      # sync, no lockfile, no wheel build.
+      - name: Install uv
+        uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+        with:
+          enable-cache: true
+
+      # SPLIT BY SHELL, copying release.yml, which learned both of these on the
+      # very Windows VM this job now targets:
+      #
+      #   * bash is not on PATH for the Windows runner service, so `shell: bash`
+      #     is not portable across these legs;
+      #   * the DEFAULT Windows shell is `pwsh` (PowerShell 7+), which is not
+      #     pre-installed — so a step with no `shell:` fails there. `powershell`
+      #     is the built-in 5.1, and `-ExecutionPolicy Bypass` gets past the
+      #     Win11 default of Restricted, per-step rather than machine-wide.
+      #
+      # Everything else the job needs to say — the toolchain versions, the
+      # results table, the job summary — the harness does for itself, precisely
+      # so there is no third shell-shaped step to keep portable.
+      - name: Install Python (Linux / macOS)
+        if: runner.os != 'Windows'
+        shell: bash
+        run: uv python install 3.14
+
+      - name: Install Python (Windows)
+        if: runner.os == 'Windows'
+        shell: powershell -NoProfile -ExecutionPolicy Bypass -File {0}
+        run: uv python install 3.14
+
+      # Deliberately NOT sccache/mold: the three arms must be built the same way
+      # as each other, and a shared compile cache that hits for one arm and
+      # misses for another changes nothing about the RUN but plenty about how
+      # much the build log can be trusted.
+      - name: Measure (Linux / macOS)
+        if: runner.os != 'Windows'
+        shell: bash
+        run: uv run --no-project python tools/bench/allocator_ab.py --reps "${{ inputs.reps }}" --size "${{ inputs.size }}"
+
+      - name: Measure (Windows)
+        if: runner.os == 'Windows'
+        shell: powershell -NoProfile -ExecutionPolicy Bypass -File {0}
+        run: uv run --no-project python tools/bench/allocator_ab.py --reps "${{ inputs.reps }}" --size "${{ inputs.size }}"
+
+      - name: Keep the readings
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        with:
+          name: allocator-bench-${{ strategy.job-index }}
+          path: output/allocator-ab/bench.log
+          retention-days: 14
+          if-no-files-found: warn

ags-wiki/design/reliquary.json

@@ -324,6 +324,14 @@
       "status": "spotted",
       "removed_in": "the release that bumps the engine tier off 0.9.0",
       "evidence": "Renamed to `is_clean` in #321, when a warning stopped failing a file and \"zero findings\" stopped being the verdict — two public `is_valid` in one crate that disagree on a warning-carrying file. Kept as a `#[deprecated]` delegating alias so the rename reaches a crates.io consumer as a warning naming both replacements rather than a build failure; 0 callers in this tree."
+    },
+    {
+      "symbol": "`.github/workflows/allocator-bench.yml` + `tools/bench/allocator_ab.py`",
+      "verify": "`.github/workflows/allocator-bench.yml::allocator-bench (temporary, #448)`",
+      "axis": "code",
+      "status": "spotted",
+      "removed_in": "the PR that records the #448 measurement",
+      "evidence": "Added deliberately temporary, and inventoried in the same change rather than after it. It is an INSTRUMENT, not a gate: `workflow_dispatch`-only, so it never fires on a push, a PR or a schedule and cannot enter the required set. It answers one question — on Linux and Windows, does mimalloc v3 beat the v2 pin by enough to narrow that pin to `cfg(target_os = \"macos\")`? — and once #448 carries the answer there is nothing left for it to measure. A measurement harness kept past its measurement becomes a build cost nobody can justify and nobody dares delete, which is what this row exists to prevent."
     }
   ]
 }

ags-wiki/design/reliquary.md

@@ -94,6 +94,7 @@ it, and name that PR in *Removed-in*). Verify `—` = display-only.
 | `build_ags4_ipc()` (wasm) | `rust-packages/laterite-ags4-wasm/src/lib.rs::pub fn build_ags4_ipc` | code | keep | — (published, feature-gated) | Inventoried by #349 as never imported by either worker — true (the app's Export pane uses the JSON door `build_ags4`). Not a relic: it is the **Arrow** door of the same verb, gated behind the `arrow` feature, documented in `wasm-api.md` as "build from source" and in `surfaces/browser.md`, and carried in the cross-surface modality register (`modality.json`, held by `packages/laterite/tests/test_modality_parity.py`) — so removing it would drop a modality row and fail that gate. `check-wasm-tier1.mjs` names it among the exports whose ABSENCE defines tier 1, which is a dependency on the name too. |
 | `dictionary()` (wasm) | `rust-packages/laterite-ags4-wasm/src/lib.rs::pub fn dictionary` | code | keep | — (published surface) | Made caller-less on the web by **this** change (#349): the app's Dictionary browser and Template generator read the static union `ags_dictionary.json` via `lib/dict.ts`, which #349 settled as the web's single dict source, so the worker op and client fn were removed. Recorded here per the relic discipline — add the row in the PR that strands the export — but `keep`, on the same grounds as its four siblings above: it is named in `EXPECTED_FUNCTIONS` in BOTH `tools/release/check-wasm-slim.mjs` and `check-wasm-tier1.mjs` and documented available in `web/docs-site/docs/reference/wasm-api.md`, so it is published npm surface. Its TYPE is still load-bearing web-side even with no caller: `web/src/lib/validator.ts` re-exports `StandardDict` from the engine so the local projection must keep conforming to what this export returns. |
 | `laterite_ags4_validator::is_valid()` | `rust-packages/laterite-ags4-validator/src/lib.rs::pub fn is_valid` | code | spotted | the release that bumps the engine tier off 0.9.0 | Renamed to `is_clean` in #321, when a warning stopped failing a file and "zero findings" stopped being the verdict — two public `is_valid` in one crate that disagree on a warning-carrying file. Kept as a `#[deprecated]` delegating alias so the rename reaches a crates.io consumer as a warning naming both replacements rather than a build failure; 0 callers in this tree. |
+| `.github/workflows/allocator-bench.yml` + `tools/bench/allocator_ab.py` | `.github/workflows/allocator-bench.yml::allocator-bench (temporary, #448)` | code | spotted | the PR that records the #448 measurement | Added deliberately temporary, and inventoried in the same change rather than after it. It is an INSTRUMENT, not a gate: `workflow_dispatch`-only, so it never fires on a push, a PR or a schedule and cannot enter the required set. It answers one question — on Linux and Windows, does mimalloc v3 beat the v2 pin by enough to narrow that pin to `cfg(target_os = "macos")`? — and once #448 carries the answer there is nothing left for it to measure. A measurement harness kept past its measurement becomes a build cost nobody can justify and nobody dares delete, which is what this row exists to prevent. |
 <!-- END GENERATED: reliquary-register -->
 
 ## Finding relics — tooling