title: "bug: exec_accuracy under-scores company_facts when filter returns multiple rows"
labels: [bug, eval, w2]
assignees: [SebAustin]
Problem
Live edgar-msft-revenues-fy2025 can return several FY2025 rows for
RevenueFromContractWithCustomerExcludingAssessedTax. Deterministic
exec_accuracy requires an exact list length match against expected, so
the case scores 0.0 even when the judge confirms the correct fact is
present.
Acceptance criteria
References
src/mcp_financial_data/evals/metrics.py
src/mcp_financial_data/evals/dispatch.py::_filter_xbrl_facts_for_eval
title: "bug: exec_accuracy under-scores company_facts when filter returns multiple rows"
labels: [bug, eval, w2]
assignees: [SebAustin]
Problem
Live
edgar-msft-revenues-fy2025can return several FY2025 rows forRevenueFromContractWithCustomerExcludingAssessedTax. Deterministicexec_accuracyrequires an exact list length match againstexpected, sothe case scores
0.0even when the judge confirms the correct fact ispresent.
Acceptance criteria
exec_accuracy(or a company-facts-specific helper) matches whenexpected.xbrl_factsis a subset ofactual.xbrl_factsby concept,fiscal year, and value tolerance.
tests/unit/evals/test_metrics.pycover multi-rowcompany-facts payloads.
--fullrun reportsmean_exec_accuracy >= 0.95withoutlowering judge scores.
References
src/mcp_financial_data/evals/metrics.pysrc/mcp_financial_data/evals/dispatch.py::_filter_xbrl_facts_for_eval