Summary
Extend racket/prologos/tools/bench-ab.rkt with a --refs flag (accepting multiple commit refs) so a single invocation can produce a multi-way A/B/C+ comparison report (vs the current --ref which supports only A/B).
Origin
Surfaced during PPN 4C Tropical Quantale Addendum 1C-vi A/B/C report work (commit TBD, 2026-05-16). Per the addendum's §13.7 "Bench infrastructure" cross-track note, this was flagged as a "small tool enhancement" deferred to a future track:
A/B/C comparison tool: tools/bench-ab.rkt currently supports A/B (--ref HEAD~1). For A/B/C: run 3 separate bench-ab.rkt invocations against tagged baseline commits (e.g., --ref pre-1c-baseline for A; current HEAD for B/C). Or extend bench-ab.rkt with --refs for multi-way comparison (Phase 1C-vi requirement; small tool enhancement).
Per 1C-vi β3 resolution: a one-off markdown A/B/C report sufficed for THIS addendum because A baseline data file already existed (live A re-measurement was structurally impossible — OLD struct-field counter retired at 1C-iv-b). Future tracks with all 3+ variants LIVE will benefit from tool-level support.
Consumed by (forward-references)
- OE Series Track 1 (weighted parsing) — multi-variant cost comparison (Goodman semiring framework on PPN). See
docs/tracking/MASTER_ROADMAP.org § OE Series.
- PReduce Track 4 (cost-guided extraction with tropical-quantale residuation) — multiple cost-extraction strategies to compare. See
docs/tracking/MASTER_ROADMAP.org § PReduce Series + docs/tracking/2026-05-02_PREDUCE_MASTER.md.
- Future PAR tracks (parallel scheduler variants comparison) — Variant A / Variant B / future scheduler patterns.
Design sketch
racket tools/bench-ab.rkt --refs HEAD~5,HEAD~2,HEAD benchmarks/comparative/
- Multiple
--ref flags OR comma-separated --refs argument
- Each ref produces a sample set (stash + checkout + run + restore; existing pattern)
- Output: per-ref Mann-Whitney U vs the others (currently bench-ab.rkt does U for 2 samples; extend to pairwise or vs-baseline matrix)
- Markdown report option (currently bench-ab.rkt outputs JSON; add markdown table generation)
Cross-references
- Design doc:
docs/tracking/2026-04-26_PPN_4C_TROPICAL_QUANTALE_ADDENDUM_DESIGN.md §13.7 (per-phase measurement plan; "small tool enhancement" cross-track note)
- 1C-vi A/B/C report (consumer; one-off workaround):
docs/tracking/2026-05-16_TROPICAL_1C_VI_ABC_REPORT.md
- Tracking:
docs/tracking/DEFERRED.md (this Issue cross-referenced from there)
- Tracking:
docs/tracking/MASTER_ROADMAP.org § OE Series + § PReduce Series (forward-references)
Multi-surface tracking discipline
This issue is part of a dual-surface tracking pattern (Issue + MASTER_ROADMAP forward-refs + DEFERRED.md entry, all cross-referenced) — established as a stronger discipline than DEFERRED.md alone, per the user direction at the 1C-vi β resolution conversation (2026-05-16): "capturing in deferred is not meaningful follow up and likely will be work lost." Each surface explicitly cites the others so any one getting updated/closed propagates via the cross-references.
Summary
Extend
racket/prologos/tools/bench-ab.rktwith a--refsflag (accepting multiple commit refs) so a single invocation can produce a multi-way A/B/C+ comparison report (vs the current--refwhich supports only A/B).Origin
Surfaced during PPN 4C Tropical Quantale Addendum 1C-vi A/B/C report work (commit TBD, 2026-05-16). Per the addendum's §13.7 "Bench infrastructure" cross-track note, this was flagged as a "small tool enhancement" deferred to a future track:
Per 1C-vi β3 resolution: a one-off markdown A/B/C report sufficed for THIS addendum because A baseline data file already existed (live A re-measurement was structurally impossible — OLD struct-field counter retired at 1C-iv-b). Future tracks with all 3+ variants LIVE will benefit from tool-level support.
Consumed by (forward-references)
docs/tracking/MASTER_ROADMAP.org§ OE Series.docs/tracking/MASTER_ROADMAP.org§ PReduce Series +docs/tracking/2026-05-02_PREDUCE_MASTER.md.Design sketch
--refflags OR comma-separated--refsargumentCross-references
docs/tracking/2026-04-26_PPN_4C_TROPICAL_QUANTALE_ADDENDUM_DESIGN.md§13.7 (per-phase measurement plan; "small tool enhancement" cross-track note)docs/tracking/2026-05-16_TROPICAL_1C_VI_ABC_REPORT.mddocs/tracking/DEFERRED.md(this Issue cross-referenced from there)docs/tracking/MASTER_ROADMAP.org§ OE Series + § PReduce Series (forward-references)Multi-surface tracking discipline
This issue is part of a dual-surface tracking pattern (Issue + MASTER_ROADMAP forward-refs + DEFERRED.md entry, all cross-referenced) — established as a stronger discipline than DEFERRED.md alone, per the user direction at the 1C-vi β resolution conversation (2026-05-16): "capturing in deferred is not meaningful follow up and likely will be work lost." Each surface explicitly cites the others so any one getting updated/closed propagates via the cross-references.