Skip to content

Keep heterogeneous evaluation result columns aligned - #4281

Closed
FlorianPfaff wants to merge 17 commits into
mainfrom
agent/align-heterogeneous-evaluation-results
Closed

Keep heterogeneous evaluation result columns aligned#4281
FlorianPfaff wants to merge 17 commits into
mainfrom
agent/align-heterogeneous-evaluation-results

Conversation

@FlorianPfaff

Copy link
Copy Markdown
Owner

Summary

  • allow grouped evaluation rows to contain optional metric fields
  • backfill newly introduced columns with None for earlier rows
  • append None when a later row omits a previously seen field
  • add focused regression coverage for both late-added and missing metrics

Bug

group_results_by_filter() assumed that every row for a filter contained exactly the same keys. When a later configuration introduced an optional metric, the function indexed a column that had never been initialized and raised KeyError. When a later configuration omitted a metric, the corresponding list was not extended, so grouped columns represented different numbers of rows and no longer aligned by parameter.

Impact

Evaluation and plotting code can now safely consume summaries whose configurations expose different optional metrics. Every grouped column contains exactly one value per sorted row, using None for unavailable values.

Validation

  • exercised the regression in isolation with late-added and omitted metric keys
  • verified the existing mixed numeric/string/None parameter ordering remains unchanged
  • compared the branch against current main; only the grouping helper and its focused test are modified

A full repository test run was not available in this environment because the checkout could not be downloaded; repository access was provided through the GitHub connector.

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

MegaLinter analysis: Success

Descriptor Linter Files Fixed Errors Warnings Elapsed time
✅ COPYPASTE jscpd yes no no 12.29s
✅ JSON prettier 7 0 0 0 0.41s
✅ JSON v8r 7 0 0 2.58s
✅ MARKDOWN markdownlint 68 0 0 0 0.94s
✅ MARKDOWN markdown-table-formatter 68 0 0 0 0.2s
✅ PYTHON black 1489 182 0 0 53.52s
✅ PYTHON isort 1489 327 0 0 1.52s
✅ REPOSITORY betterleaks yes no no 1.35s
✅ REPOSITORY checkov yes no no 36.76s
✅ REPOSITORY gitleaks yes no no 9.5s
✅ REPOSITORY git_diff yes no no 0.09s
✅ REPOSITORY secretlint yes no no 35.08s
✅ REPOSITORY syft yes no no 4.28s
✅ REPOSITORY trivy-sbom yes no no 4.91s
✅ REPOSITORY trufflehog yes no no 20.05s
✅ YAML prettier 11 0 0 0 0.5s
✅ YAML v8r 11 0 0 5.95s
✅ YAML yamllint 11 0 0 0.44s

Notices

📣 MegaLinter 9.5.0 is out! Discover the new features and security recommendations in the release announcement. (Skip this info by defining SECURITY_SUGGESTIONS: false)

See detailed reports in MegaLinter artifacts

Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining FLAVOR_SUGGESTIONS: false)

  • Documentation: Custom Flavors
  • Command: npx mega-linter-runner@9.6.0 --custom-flavor-setup --custom-flavor-linters PYTHON_BLACK,PYTHON_ISORT,COPYPASTE_JSCPD,JSON_V8R,JSON_PRETTIER,MARKDOWN_MARKDOWNLINT,MARKDOWN_MARKDOWN_TABLE_FORMATTER,REPOSITORY_CHECKOV,REPOSITORY_GIT_DIFF,REPOSITORY_GITLEAKS,REPOSITORY_BETTERLEAKS,REPOSITORY_SECRETLINT,REPOSITORY_SYFT,REPOSITORY_TRIVY_SBOM,REPOSITORY_TRUFFLEHOG,YAML_PRETTIER,YAML_YAMLLINT,YAML_V8R

MegaLinter is graciously provided by OX Security
Show us your support by starring ⭐ the repository

@FlorianPfaff
FlorianPfaff force-pushed the agent/align-heterogeneous-evaluation-results branch from f0ac388 to 5f8b7c9 Compare July 13, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant