Repository navigation
ci(bench): measure what the macOS mimalloc pin costs Linux and Windows (#448) - #476
Merged
Merged
Conversation
#448) The v2 pin (#294/#301) exists for a fault that is **macOS-specific**: mimalloc v3 defaulted to Apple's fixed TLS slots 108/109, so two co-resident v3 instances shared them regardless of symbol visibility. The pin is not platform-gated, so Linux and Windows carry a macOS fix — and what it costs them has never been measured. #448's own re-run list says to re-measure rather than quote the figure from when the allocator landed. `tools/bench/allocator_ab.py` builds `lat` three ways — v2, v3, and no `#[global_allocator]` at all — and times them on a `forge scale` fixture, which is synthesised from a seed rather than taken from a delivery, so it reproduces anywhere and carries nobody's data. No macOS leg, deliberately. The question is WITHIN-platform: same box, three binaries. Comparing Linux-v3 against macOS-v2 would measure the machines. Four things stop it reporting a number it has not earned. - **The system arm is a positive control.** The perf ledger records mimalloc-vs-system as a real measured win, so if this harness cannot separate system from v2 it cannot resolve allocator effects at all — and "v2 and v3 look the same" would mean the instrument is broken, not that the allocators are equivalent. That verdict is computed, not left to the reader. - **A distinct-binary check.** If the feature swap fails to reach the compiler the three arms are one build measured three times; the run aborts on identical digests rather than reporting it. - **An exact-match patcher.** A manifest that did not change still builds and still runs, reporting a number under the wrong label. A near-miss fails loudly. - **A noise floor from two estimates, cleared by 2x.** A/B/A drift catches wander between arms; the widest within-arm spread catches a jittery box that lands both v2 passes together. An earlier revision called 1.5% "measurably faster" against a 1.0% floor — exactly the over-claim the harness exists to stop. The spawn-per-rep overhead is a constant shared by every arm, so it shrinks the measured gap rather than inflating it. The instrument under-claims, which is the right bias for deciding whether to unpin. Linux runs on the self-hosted pool because a dedicated box is the right instrument for a timing comparison; Windows has no self-hosted runner and takes `windows-2022`, which is noisier — and the harness's own floor is what says whether those numbers are readable rather than a reader having to guess. `workflow_dispatch`-only: it never fires on a push, a PR or a schedule, so it cannot fail a build or enter the required set. Inventoried in the reliquary in this same change, with removal keyed to the PR that records the measurement — "temporary" otherwise survives by nobody remembering.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…ming a shell
The Windows runner is registered on this repo now (`laterite-windows`), so the
leg becomes switchable rather than pinned to `windows-2022`: a `windows_runner`
dispatch input picks between them, defaulting to the github-hosted one so the
job still works if the VM is offline. The runner expression is one line on
purpose — a folded scalar keeps its newlines inside the `${{ }}`, and a bad
runner label does not error, it queues until the timeout.
Two corrections that `release.yml` had already paid for on this same VM, and
that this job would otherwise have rediscovered:
- **bash is not on PATH for the Windows runner service.** Four steps here used
`shell: bash`. Split per-OS, as release.yml does for exactly this reason.
- **The default Windows shell is `pwsh`, which is not pre-installed.** A step
with no `shell:` fails there. `powershell` is the built-in 5.1, and
`-ExecutionPolicy Bypass` gets past the Win11 default of Restricted per-step
rather than machine-wide.
Python comes from uv rather than from the box — it is not on PATH there, and
installing it would make the job depend on hand-configured machine state, which
the github-hosted fallback leg would not share. It is also the house pattern:
nothing here uses `setup-python`, and release.yml already proves the uv path on
this VM.
The rest is the harness taking back what the workflow could not say portably.
It now prints its own toolchain banner, writes its own `bench.log`, and appends
its own table to `GITHUB_STEP_SUMMARY` — so the `tee`/`sed` pipeline is gone
along with the third shell-shaped step, and a local run produces the same
artifacts as a CI one.
Verified by running it with `GITHUB_STEP_SUMMARY` set: log written, summary
written, and the verdict correct. At 4 reps on the 1 MB rung it reported *"the
instrument proved nothing"* — the control did not clear 2x the noise floor —
which is the guard working on a deliberately under-powered run rather than
reporting the 2.2% v3 gap sitting above it.
This was referenced Aug 20, 2026
niko86
added a commit
that referenced
this pull request
Aug 20, 2026
…easure (#448) (#479) Added deliberately temporary in #476 and inventoried in the same change, with its removal keyed to "the PR that records the #448 measurement". This is that PR. The measurement is recorded on #448. Two independent runs on both self-hosted runners at the 25 MB rung, 15 reps per arm: - the system-allocator control separated by 35–36% on Linux and 53–63% on Windows, so the harness demonstrably resolved allocator effects at this scale; - against that, every v2/v3 gap sat inside its own run's noise floor, and the Windows delta CHANGED SIGN between runs (-1.7%, then +0.4%). So narrowing the pin to `cfg(target_os = "macos")` buys Linux and Windows nothing measurable, which inverts the case for doing it. The cost half of #448 is no longer an open premise; the issue stays open on its actual trigger, which has not fired — `libmimalloc-sys` 0.1.49 still vendors v3 3.3.2. The harness goes rather than staying "in case we want it again". Kept past its question it would be a build cost nobody could justify and nobody would dare delete — which is the failure mode the reliquary row was written to prevent, in the same commit that created the thing it was watching. The row flips to `removed` rather than disappearing, so the record of why it existed outlives the code. Re-deriving it is cheap if a future question needs it: the method is on #448 and the file is in this history.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✅ Based on
main— mergeable now. A temporary instrument for #448, inventoried for deletion in the same change.Why, and why there's no macOS leg
The v2 pin (#294/#301) exists for a macOS-specific fault: mimalloc v3
defaulted to Apple's fixed TLS slots 108/109, which libpthread reserves for the
Swift runtime, so two co-resident v3 instances shared them regardless of symbol
visibility. The pin is not platform-gated —
mimalloc = { version = "0.1", features = ["v2"] }, flat, on all three surfaces. So Linux and Windows carry amacOS fix and nobody has measured what it costs them.
You asked whether a macOS baseline is needed. It isn't, and that's structural:
the question is within-platform — same box, three binaries. Comparing
Linux-v3 against macOS-v2 measures the machines, not the allocator. What a null
result needs isn't another platform, it's a control arm — which is the third
build.
What stops it reporting a number it hasn't earned
perf-campaign.mdledger row 10records mimalloc-vs-system as a real measured win. So if the harness can't
separate
systemfromv2, it can't resolve allocator effects at all — and"v2 and v3 look the same" means the instrument is broken, not that the
allocators are equivalent. The verdict says so in those words rather than
leaving it to the reader.
the three arms are one build measured three times. Identical digests abort the
run.
runs, and reports a number under the wrong label. A near-miss fails loudly.
between arms; the widest within-arm spread catches a jittery box that lands
both v2 passes together. Whichever is worse is the floor.
That last one earned its place: an earlier revision of this script called 1.5%
"measurably faster" against a 1.0% noise floor — clearing the floor by a
rounding error — which is precisely the over-claim the harness exists to
prevent. Caught by running it, not by reading it.
The spawn-per-rep overhead is a constant shared by all arms, so it shrinks
the measured gap rather than inflating it. The instrument under-claims, which is
the right bias for a decision about removing a safety pin.
Verified by running it
Locally on this Mac, 8 reps at the 10 MB rung — three distinct binaries built,
tree restored afterwards:
v2v3v2 (again)systemNoise floor 2.8 ms (the within-arm spread, not the 0.3 ms drift — which is
exactly why the second estimate is there). Control separated at 8.6 ms; v3/v2 at
1.1 ms did not clear 2×. Verdict: indistinguishable. That's a macOS reading
and decides nothing about the pin — it's here as evidence the harness works.
The two legs
Linux takes the self-hosted pool (
laterite-linux) because a dedicated box isthe right instrument for a timing comparison and GitHub's shared vCPUs aren't.
Windows has no self-hosted runner — it takes
windows-2022, the same imagerelease.ymlbuilds the Windows wheel on. Those numbers will be noisier, and theharness's own floor is what says whether they're readable.
The Rust toolchain is installed rather than assumed on both: an arm built by a
different compiler than its sibling isn't a comparison.
python3is resolved viacommand -vbecause the Windows image has the interpreter on PATH aspython.It is not permanent
workflow_dispatch-only — it never fires on a push, a PR or a schedule, so itcan't fail a build or enter the required set. Inventoried in
reliquary.jsonin this change,status: spotted, removal keyed to the PRthat records the #448 measurement. "Temporary" otherwise survives by nobody
remembering, and the reliquary is this repo's answer to that.
Gates
ruff check·ruff format --check·ty check·gen_wiki_tables --check·check_doc_refs --check· wikilint.py(the reliquary row'sVerifytoken isheld to
path::tokenand had to be fixed once) ·reindex.py --check·gen_changelog --check·check_changelog --base origin/main·tests/test_self_named_gates.py.No changelog entry:
check_changelogcorrectly classifies this as tooling, andan entry for something scheduled for deletion is churn.
One bug caught by parsing the YAML rather than trusting it:
name: allocator-bench (temporary, #448)silently truncated at#, which starts acomment in an unquoted YAML scalar. Quoted now.
To run it: Actions → allocator-bench (temporary, #448) → Run workflow. Defaults
are 15 reps at the 25 MB rung.