Add support for benchmark comparisons

[jamesx-improving]

Note

Copied from ikolomi/resp-bench#16.
Originally opened by @jeremyprime on 2026-04-09T17:51:13Z.

Summary

Enable A/B comparison of two versions, branches, or commits by running the same workload against each and reporting the delta. This is the primary use case for detecting performance regressions in PRs and between releases.

Current Behavior

Benchmarks run against a single version. Comparing two versions requires manually running benchmarks separately and visually comparing the results. The interactive graph generator supports delta charts, but there's no first-class workflow for running both versions and producing a comparison report.

Proposed Behavior

Accept two versions (Maven version or or commit ID) as input, run the same workload against each, and produce a comparison report showing the delta for key metrics (latency and throughput).

This could be implemented as:

• A wrapper script or mode in the matrix runner that builds and benchmarks version A, then version B, then generates delta charts
• Or a standalone comparison script that takes two existing result directories and produces the delta report

The output should clearly show regressions to support quick pass/fail decisions.

Implementation Notes

• The interactive graph generator already supports delta traces — this work is primarily about orchestrating the two runs and feeding results into the existing comparison tooling
• Should support both local CLI usage and integration into GitHub Actions
• Multi-iteration support (already in the matrix runner) is important here to reduce noise in comparisons