Estimate missing benchmark scores from arena category distributions

[DiTo97]

Take-home assignment. Self-contained and vertical: methodology → algorithm → SDK surface → validation → docs → tests. Unlike the sibling scraping issues, the core of this one is pure, offline-testable logic — that is deliberate. Read the whole issue before starting; the acceptance criteria are the contract.

Problem

Even with arena.ai scores and a public-benchmark catalog ingested, coverage will be sparse and uneven. Most (model, reasoning_effort) pairs will have a published score on a handful of benchmarks and nothing on the rest — newer models, smaller vendors, and non-default reasoning efforts especially. A query layer that can only rank models with complete data will rank almost nothing.

So: where a direct benchmark score is missing, estimate it by mapping arena.ai category scores onto the distribution of that benchmark's category, and aggregate. The estimate is a genuine product feature — but only if it is honest about being an estimate and about how uncertain it is.

The failure mode we are most worried about is a plausible number with no error bars silently outranking a measured one. Design against that.

Scope

• Estimation algorithm. Given: arena category scores per (model, reasoning_effort); the benchmark catalog with its tag taxonomy; and the set of observed (model, reasoning_effort) → benchmark → value pairs. Produce: an estimated value for an unobserved pair, with an uncertainty measure.
• Category → benchmark mapping. The link from arena categories to benchmarks is the crux. Treat it as a distribution rather than a lookup: a benchmark relates to several arena categories with differing strength, and that relationship should be derived from the observed data where enough of it exists, not hand-asserted. Hand-asserted priors are acceptable as a cold-start fallback if they are labelled as such.
• Uncertainty is mandatory. Every estimate carries an interval or variance, plus how much evidence it rests on. An estimate with one supporting observation and one with fifty must be distinguishable by a caller who is not reading the source.
• Provenance and labelling. Every value the system can return is either measured or estimated, always distinguishable, at every layer — storage, SDK return types, CLI output, docs. No code path may present an estimate as a measurement. This is the single hardest requirement in the issue; treat it as a type-system problem, not a formatting convention.
• Documented methodology. A written methodology doc: the assumptions, the maths, what makes an estimate unreliable, and the conditions under which the system should refuse to estimate at all rather than emit noise. Refusal is a valid and expected output.
• Validation. Backtest by holding out known scores and estimating them: report error (MAE or equivalent), calibration of the intervals, and how error degrades as evidence thins. Numbers in the PR, reproducible by a checked-in command.
• Testability without live scraping. The estimator takes data structures as input, not URLs. It must be fully unit-testable against synthetic and fixture data with no network. This is an explicit design constraint, not an implementation detail.
• SDK surface. Expose estimates through the SDK alongside measurements, with the distinction visible in the return type. Async and sync surfaces stay paired per libraries/python/AGENTS.md.

Out of scope

• The remote-database / SQLite migration. Issues Migration: JSON-driven database → derived SQLite read layer for faster UI & SDK queries #60 and SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1 #61 are open and cover a derived SQLite read layer. Do not touch them or build on them. Storage stays git-committed JSON.
• Workload-specific leaderboard / arena-per-agent generation. Deferred to a later phase.
• CI cron jobs, API keys, or paid subscriptions. Explicitly excluded.
• Scraping. You consume the outputs of the arena and benchmark-catalog issues. If those have not landed, define the minimal input interfaces you need, note the coupling in your PR, and test against fixtures — do not build a competing ingestion path.
• Training a learned model on external data. Interpretable statistical aggregation is what we want; if you believe something heavier is justified, argue it in the PR rather than assuming it.

Suggested approach

1. Write the methodology doc first, in prose, with the maths. If you cannot explain why the estimate is defensible before implementing it, the implementation will not fix that.
2. Decide early how a "category distribution" is represented and how it is fitted from observed data. Be explicit about what happens with thin evidence — that is the common case, not the edge case.
3. Make the measured/estimated distinction structural. If a caller can accidentally drop the flag by destructuring a dict, the design is wrong.
4. Build the backtest harness before tuning anything, so you are not choosing parameters by vibes.
5. Test the refusal path as carefully as the happy path: assert that below your evidence threshold the system returns "no estimate" rather than a number.
6. Keep the estimator a pure function of its inputs. No I/O, no clock reads, no hidden global state — that is what makes it reviewable.

Acceptance criteria

• Estimation algorithm implemented as pure, offline-testable logic taking data structures as input; no network, no I/O, deterministic.
• Arena-category → benchmark relationship is modelled as a distribution and derived from observed data where evidence permits; any hand-asserted prior is labelled as such.
• Every estimate carries an uncertainty measure and an evidence count.
• Measured vs estimated is structurally distinguishable at storage, SDK, and CLI layers — enforced by types or tests, not by convention. A test asserts an estimate cannot be returned unlabelled.
• The system refuses to estimate below a documented evidence threshold, and this is tested.
• Methodology document committed: assumptions, maths, failure modes, refusal conditions, known biases.
• Backtest implemented and runnable via a checked-in command; results (error metric + interval calibration + degradation with thinning evidence) reported in the PR.
• Estimates exposed through the SDK with async/sync parity per libraries/python/AGENTS.md, and the estimated flag visible in the public return type.
• uv run pytest -q and uv run pre-commit run -a green; full test suite passes with no network access.
• Docs explain to an end user what an estimate means and when not to trust it.

Working agreement

Follow the self-contained-task workflow in libraries/python/AGENTS.md:29-37: linked branch via gh issue develop <issue> --checkout, draft PR early with plan and acceptance criteria, then plan → review → TDD (failing first) → implement minimally → verify → report.

uv for Python. CI is path-filtered; pushes to main touching libraries/** auto-release.

We will judge this on statistical honesty above all — a modest estimator with well-calibrated intervals and a clear refusal rule is a pass; a clever one that emits confident numbers from thin evidence is not. Reporting that your estimator performs poorly in some regime, with evidence, is a good outcome.