Skip to content

probe(vm): comment-only control for the repeat A/B noise floor - #709

Closed
mparrett wants to merge 1 commit into
mainfrom
wt/perf-noise-control
Closed

probe(vm): comment-only control for the repeat A/B noise floor#709
mparrett wants to merge 1 commit into
mainfrom
wt/perf-noise-control

Conversation

@mparrett

@mparrett mparrett commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Measurement control. Do not merge. Closing it once the numbers land, same as #691.

The entire diff is a comment. The point is to push a build with identical machine code through the repeat A/B lane, so whatever deltas come back are the lane's own noise at N=7 rather than a property of any change.

The code really is identical

Worth checking rather than assuming, since the whole value of the control rests on it. Built base and head with -trimpath and compared:

  • go tool nm: 12907 symbols in both, at identical addresses — same layout, nothing moved.
  • __text section: 0x338184 bytes in both.
  • The binaries do differ, but only near the start of the file: the Go build ID hashes the source bytes, and a comment is a source byte.

So this holds code and placement constant, which is what makes any reported delta attributable to the harness and the runner alone.

Why now

#706 is a semantically inert change — it deletes a *Frame parameter that was never read. On this lane it measured SeqIteration/List/01000 +8.30% and FrameDispatch +4.97%, each consistent across all seven cycles, alongside −0.75% on FuncInvoke/Direct. I could not tell from that run whether those are real or whether the lane can produce numbers that size on its own, and that is not a question #706 should have to answer.

It is also the measurement #705 §1 is about. The 5% ratchet budget has never been checked against a measured floor, and #564 already found byte-identical binaries landing 3.7% and 1.1% apart on two packages.

Reading the result

Either way the number is worth having, and it belongs on #705 rather than buried in a PR thread. I will post it there too.

Measurement control. Do not merge.

The entire diff is a comment. Verified that it changes no code: built base and
head with -trimpath and compared `go tool nm` output -- 12907 symbols at
identical addresses in both, and __text is 0x338184 bytes in both. The
binaries differ only in the Go build ID, which hashes the source bytes.

So any per-family delta the repeat A/B lane reports on this PR is the lane's
own noise at N=7, measured with code and layout held constant. That number
does not exist today, which is what makes the 5% ratchet budget uncalibrated
(#705).

Prompted by #706, where a semantically inert change measured +8.30% on
SeqIteration/List/01000 and +4.97% on FrameDispatch, both consistent across
all seven cycles.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mparrett mparrett added the perf-repeat Run the repeat A/B (variance-reduced) perf check label Aug 9, 2026
@mparrett

Copy link
Copy Markdown
Collaborator Author

Closing this — it did its job. Numbers below, and the combined analysis of all three A/A runs is on #705.

This run, comment-only diff so the machine code was identical (verified: same symbol addresses, same __text size, only the build ID differs):

  • 53 families, clean at 6, 8 and 10%.
  • Largest median move: −12.91% on VectorCreation/ArrayVector/10000. Negative, so it did not trip the gate, which is itself worth noting — a budget that only looks at regressions misses half of what the noise does.
  • Largest single-cycle swing: 49.66%.

Taken with the two accidental A/A runs on #708, the conclusion is that the floor is per-family and spans roughly 0.3% to 13.4% on this tier. The allocation-heavy families at sizes 1000 and 10000 are where nearly all of it lives; 29 of 53 families stay under 2%. Details and the per-family table are on #705.

The one thing I would carry forward from building this: a deliberate control was not needed. Both of the other samples came from PRs that happened to touch no Go source, and this repo produces those routinely. Recording A/A results when they occur would accumulate the per-family floor for free.

@mparrett mparrett closed this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

perf-repeat Run the repeat A/B (variance-reduced) perf check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant