Skip to content

Extend bench-ab.rkt with --refs for multi-way A/B/C+ comparison #63

Description

@hierophantos

Summary

Extend racket/prologos/tools/bench-ab.rkt with a --refs flag (accepting multiple commit refs) so a single invocation can produce a multi-way A/B/C+ comparison report (vs the current --ref which supports only A/B).

Origin

Surfaced during PPN 4C Tropical Quantale Addendum 1C-vi A/B/C report work (commit TBD, 2026-05-16). Per the addendum's §13.7 "Bench infrastructure" cross-track note, this was flagged as a "small tool enhancement" deferred to a future track:

A/B/C comparison tool: tools/bench-ab.rkt currently supports A/B (--ref HEAD~1). For A/B/C: run 3 separate bench-ab.rkt invocations against tagged baseline commits (e.g., --ref pre-1c-baseline for A; current HEAD for B/C). Or extend bench-ab.rkt with --refs for multi-way comparison (Phase 1C-vi requirement; small tool enhancement).

Per 1C-vi β3 resolution: a one-off markdown A/B/C report sufficed for THIS addendum because A baseline data file already existed (live A re-measurement was structurally impossible — OLD struct-field counter retired at 1C-iv-b). Future tracks with all 3+ variants LIVE will benefit from tool-level support.

Consumed by (forward-references)

  • OE Series Track 1 (weighted parsing) — multi-variant cost comparison (Goodman semiring framework on PPN). See docs/tracking/MASTER_ROADMAP.org § OE Series.
  • PReduce Track 4 (cost-guided extraction with tropical-quantale residuation) — multiple cost-extraction strategies to compare. See docs/tracking/MASTER_ROADMAP.org § PReduce Series + docs/tracking/2026-05-02_PREDUCE_MASTER.md.
  • Future PAR tracks (parallel scheduler variants comparison) — Variant A / Variant B / future scheduler patterns.

Design sketch

racket tools/bench-ab.rkt --refs HEAD~5,HEAD~2,HEAD benchmarks/comparative/
  • Multiple --ref flags OR comma-separated --refs argument
  • Each ref produces a sample set (stash + checkout + run + restore; existing pattern)
  • Output: per-ref Mann-Whitney U vs the others (currently bench-ab.rkt does U for 2 samples; extend to pairwise or vs-baseline matrix)
  • Markdown report option (currently bench-ab.rkt outputs JSON; add markdown table generation)

Cross-references

  • Design doc: docs/tracking/2026-04-26_PPN_4C_TROPICAL_QUANTALE_ADDENDUM_DESIGN.md §13.7 (per-phase measurement plan; "small tool enhancement" cross-track note)
  • 1C-vi A/B/C report (consumer; one-off workaround): docs/tracking/2026-05-16_TROPICAL_1C_VI_ABC_REPORT.md
  • Tracking: docs/tracking/DEFERRED.md (this Issue cross-referenced from there)
  • Tracking: docs/tracking/MASTER_ROADMAP.org § OE Series + § PReduce Series (forward-references)

Multi-surface tracking discipline

This issue is part of a dual-surface tracking pattern (Issue + MASTER_ROADMAP forward-refs + DEFERRED.md entry, all cross-referenced) — established as a stronger discipline than DEFERRED.md alone, per the user direction at the 1C-vi β resolution conversation (2026-05-16): "capturing in deferred is not meaningful follow up and likely will be work lost." Each surface explicitly cites the others so any one getting updated/closed propagates via the cross-references.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions