Skip to content

[Measurement] Sort findingMetricsByType keys in benchmark runner for stable baseline diffs #118

Description

@AndresL230

Summary

benchmark/baseline.json's findingMetricsByType object preserves insertion order from the runner, which follows actual-emission order. This causes cosmetic diff noise across PRs whenever the set of emitted finding types changes (PR-2 removed cache, PR-4 removed rate_limit — each refresh reshuffles the remaining rows).

Evidence

What to do

  • In benchmark/runner.ts main() (around the --update-baseline write), sort the keys of findingMetricsByType alphabetically before serialization. Same for any other dynamic-key map in the JSON.
  • Pure runner change; does not touch the metrics computation.

Acceptance criteria

  • findingMetricsByType keys appear in alphabetical order in baseline.json after --update-baseline.
  • Refresh baseline once to apply the sort; subsequent PR diffs touch only value changes, not order.
  • No metric values change as a result.

Reference

Final code review on PR #111.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/measurementdocs/accuracy/measurement.md — benchmark corpus and CI gatesenhancementNew feature or requestwave/5-housekeepingSmall benchmark-infra cleanups: baseline key sort, CI gate verify

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions