Skip to content

feat: add retrieval eval advisory report#19

Merged
cafitac merged 1 commit intomainfrom
feat/retrieval-eval-advisory-report
Apr 30, 2026
Merged

feat: add retrieval eval advisory report#19
cafitac merged 1 commit intomainfrom
feat/retrieval-eval-advisory-report

Conversation

@cafitac
Copy link
Copy Markdown
Owner

@cafitac cafitac commented Apr 30, 2026

Summary

  • add a structured advisory_report to retrieval eval JSON results
  • render the advisory report and recommended actions in text reports
  • document that reports now include severity, affected task IDs, and next-action guidance

Contract

advisory_report includes:

  • severity: ok, medium, or high
  • summary
  • current_failure_task_ids
  • baseline_weak_spot_task_ids
  • current_regression_task_ids
  • recommended_actions
  • baseline_mode

Test plan

  • uv run pytest tests/test_retrieval_evaluation.py -q
  • uv run pytest tests/ -q
  • uv run python scripts/check_release_metadata.py
  • uv run python scripts/smoke_release_readiness.py
  • npm pack --dry-run
  • git diff --check
  • static diff secret scan: 0 findings

@cafitac cafitac merged commit b3ce9fe into main Apr 30, 2026
2 checks passed
@cafitac cafitac deleted the feat/retrieval-eval-advisory-report branch April 30, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant