Stop Compare from favouring the candidate that ran alone last

[TomTonic]

Fixes #111.

Problem

Compare ran ValidateHarness(a), then ValidateHarness(b), then Collect with a single warm-up pair. When the working set was near the size of the last-level cache, B started the measurement with the cache full of its own data. Short, calibrated batches did not turn the cache over during the run, so identical code came out tens of percent apart and was reported as resolved.

Changes

• ValidatePair(a, b, ValidationOptions) (new, exported) validates both candidates together, interleaved batch by batch in the order A, B, B′, A′ and back. In that order each half of an A/A experiment has the same history. The first version used A, B, A′, B′, where A followed itself and A′ never did; a test now checks the history balance. Compare uses ValidatePair, and ValidateHarness is unchanged for direct callers.
• Warm-up by duration: CollectOptions.WarmupDuration (new; DefaultWarmupDuration = 300 ms). The warm-up runs until both the count and the duration are reached, and alternates the order of the candidates (ABBA) instead of always running a then b. A validation pays for the duration only before its first run.
• Diagnosis: Report.DriftRatio runs DetectDrift on B/A for each pair of neighbouring batches. A warning appears only when the trend is significant and larger than the resolution, i.e. max(noise floor, half the interval width).
• Docs: a new HOWTO section "Whatever ran last starts warm", plus updates to Compare, Collect, CollectOptions, ValidateHarness and the README.
• Reproduction: cmd/rtcompare-aa (A/A pointer chase, -mode prefix|compare, -build ab|ba|mixed), as the issue asks for. Intervals and noise floor cover only within-process noise; results scatter between processes far beyond them #109 will extend it.

Internally, Collect now runs through a schedule (the resolved options) and measureInTurn (a balanced loop for any number of candidates). With two candidates that loop produces exactly the old ABBA, Random and Sequential orders.

Measurements

Ryzen 9 7900 (32 MB L3), WSL2, Go 1.27.1. Two identical 16 MB fixtures, 5 processes, each in both role assignments. Delta = 1 − A/B, pooled across both role assignments:

Setting	before	after
Repeats 41, ValidationRuns 2, build ab	−29.4 % (7 of 10 resolved)	+0.35 %
Defaults, build ab	−0.5 %	+0.34 %
Repeats 41, ValidationRuns 2, build ba	–	+0.19 %
In-cache control (4K nodes)	±0.2 %	−0.06 %

What remains: after the fix, the fixture built second is 3–5 % faster in both roles, and the sign flips with -build ba. A 2 s warm-up leaves that unchanged, so it comes from the memory layout of the two fixtures, not from a warm cache. That is #109, and the HOWTO now explains how to tell the two apart (swap the roles, swap the build order).

The noise floors in the 41/2 case also went down, from 3–9 % to 1–4 %. For the example's candidates with GCBetween, ValidatePair and two ValidateHarness calls gave the same floors within noise (6 repetitions × 40 runs each).

Behaviour change

• Compare takes 0.6 s longer (0.3 s with SkipValidation), and each Collect 0.3 s longer.
• Callers who set Warmup: N now also get the minimum duration. WarmupDuration: time.Nanosecond warms up by count alone.
• Test suite: 61 s → 69 s. fastCompare in the tests cuts the warm-up to 10 ms, because those candidates stay in the cache.

Checks

• go test ./... and go test ./... -race pass, and golangci-lint run reports 0 issues.
• New tests (documented outside-in): warm-up duration and order, measureInTurn with 3 candidates, ValidatePair (interleaving, calibration to the larger batch, errors), history balance of pairSlots, warm-up once per validation, no long single-candidate stretches in Compare, pairRatios, and warning cases for DriftRatio.

🤖 Generated with Claude Code