Toolkit query layer: rank best model for a task across quality, cost, SLO and budget

[DiTo97]

Take-home assignment. Self-contained and vertical: query semantics → SDK → CLI → skill → docs → tests. This is the capstone that makes the other five useful. Read the whole issue before starting; the acceptance criteria are the contract.

Problem

Today the only way to ask this repo a question is tokenpricing pricing MODEL and tokenpricing cost MODEL --in N --out M — you must already know which model you want. PricingData.search_models (libraries/python/src/tokenpricing/modeling.py:141) takes four filters (provider, category, vision, function-calling) over a linear scan, and returns unranked results.

The point of tokenvalue is the inverse question: "given what I need, which model should I use?" — where "what I need" combines quality evidence (benchmark scores, arena categories), cost, inference SLO (latency, throughput, rate limits), deprecation risk, and a budget, and where those criteria must be combinable rather than applied one at a time.

This issue builds that query layer on top of the unified index from the other five, and exposes it so an AI agent can call it directly.

Scope

• Query semantics. Design a query model that supports both hard filters (must be text-out, must be on an approved platform, must not be deprecated within N months, must support ≥X TPM, must cost under Y per million) and soft weighted preferences (favour quality, favour cheapness, favour latency, with relative weights). Getting the semantics right — and writing them down — matters more than the implementation.
• Ranking. A composite score over incommensurable dimensions (a benchmark percentage, dollars per million tokens, milliseconds). Normalization across those units is the central design problem. Whatever you choose, it must be documented and defensible, and the caller must be able to understand the outcome.
• Explainability, non-negotiable. Every result explains why it ranked where it did: which criteria it satisfied, which it failed, what its component scores were, and — critically — which inputs were estimated rather than measured and which were simply missing. A ranked list of names with no justification is a failing submission.
• Graceful degradation. Real data will be full of holes. Decide what happens when a candidate lacks the benchmark score, or the latency figure, or the rate limit that the query cares about — excluded, penalised, or ranked with a coverage caveat — and make the choice explicit and configurable rather than implicit. Silently dropping under-documented models systematically biases results toward whichever vendors document best; call that out.
• Cost realism. Cost must be computable for a described workload (input/output/cache token mix), not just as a headline per-million rate — the existing compute_cost logic is the base. Budget constraints should be expressible against a workload, not only against a unit price.
• Surfaces. Three, sharing one implementation:
  • SDK — Python with async/sync parity per libraries/python/AGENTS.md, and the TypeScript mirror.
  • CLI — a new command alongside pricing and cost, with --json output, following the Click patterns in libraries/python/src/tokenpricing/commandline.py.
  • Skill — update or extend the skill layer so an agent can drive the query and interpret the results, following skills/tokenpricing/SKILL.md conventions and registered in .claude-plugin/marketplace.json.
• Reasoning effort as a dimension. The same model at different reasoning efforts is a different candidate with different cost, latency, and quality. The query layer must rank across variants, not just across models — that is a large part of why the identity work in the data-model issue exists.

Out of scope

• The remote-database / SQLite migration. Issues Migration: JSON-driven database → derived SQLite read layer for faster UI & SDK queries #60 and SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1 #61 are open and cover a derived SQLite read layer. Do not touch them or build on them. Storage stays git-committed JSON, and the current row count makes linear scans acceptable — do not build an index engine.
• Workload-specific leaderboard / arena-per-agent generation. Deferred to a later phase. That means: no ingesting an existing agentic workload's specs or traces, no data-residency constraint solving, no per-workload "open book" leaderboard generation. This issue takes an explicitly stated query, not an inferred one.
• CI cron jobs, API keys, or paid subscriptions. Explicitly excluded.
• Scraping of any kind. You consume the unified layer. Where the upstream issues have not landed, define the minimal input interfaces you need, note the coupling in your PR, and test against fixtures.
• Dashboard UI for the query builder — nice if you get there, not required, and not worth trading against explainability.

Suggested approach

1. Write the query semantics down first — the grammar of what can be asked, and precisely what each operator means. Review that before implementing. Most of the risk in this issue is semantic, not algorithmic.
2. Pick three or four realistic questions and use them as your acceptance scenarios throughout. For example: cheapest model scoring above X on a coding benchmark, available on our approved platforms, not deprecated within 12 months; lowest-latency multilingual model under a monthly budget for a stated token mix; best quality-per-dollar at high reasoning effort. Keep them as integration tests.
3. Solve normalization explicitly and visibly. Hiding it inside a scoring function nobody can audit is the easy wrong answer.
4. Build explainability into the result type from the first commit. Bolting it on afterwards produces exactly the vague output we are trying to avoid.
5. TDD against fixtures with deliberately missing data — that is the realistic case, and it is where the degradation policy earns its keep.
6. Keep the ranking core a pure function of its inputs so it can be tested without any I/O.

Acceptance criteria

• Query model supports hard filters and weighted soft preferences over quality, cost, latency/throughput, rate limits, and deprecation horizon, combinable in a single query.
• Ranking normalizes across incommensurable units with a documented, defensible method.
• Every result is explainable: component scores, criteria met and failed, and explicit marking of estimated vs measured vs missing inputs. Enforced by tests.
• Missing-data policy is explicit, configurable, and documented; a test covers each behaviour. The bias introduced by excluding under-documented models is acknowledged in the docs.
• Cost is computable against a described workload token mix (input/output/cache), and budget constraints can be expressed against a workload, reusing the existing cost logic.
• Candidates are ranked across (model, reasoning_effort, serving_platform) variants, not just models.
• SDK exposes the query with async/sync parity (Python) and a TypeScript mirror.
• CLI command added alongside pricing and cost, with --json, following existing Click patterns.
• Skill layer updated so an agent can drive the query and interpret the explanation; registered in .claude-plugin/marketplace.json.
• The three or four realistic scenarios exist as integration tests against fixtures, passing offline.
• uv run pytest -q, the TypeScript vitest suite, and uv run pre-commit run -a all green.
• Docs: query semantics, ranking methodology, missing-data behaviour, and worked examples.

Working agreement

Follow the self-contained-task workflow in libraries/python/AGENTS.md:29-37: linked branch via gh issue develop <issue> --checkout, draft PR early with plan and acceptance criteria, then plan → review → TDD (failing first) → implement minimally → verify → report.

uv for Python, pnpm for libraries/typescript, npm for services/dashboard. CI is path-filtered; pushes to main touching libraries/** auto-release, so keep the blast radius intentional.

We will read the query semantics and the explainability output first. A narrow query language that is precisely specified and fully explainable beats a broad one that returns unjustified rankings.

[DiTo97]

Two additions from the AA spike (#85)

1. Openness / licensing is a new query dimension this issue should accommodate.

#85 will land the AA Openness Index — a 0–18 score per model across six subcomponents, covering open and proprietary models. It answers questions the current dimension list (quality, cost, latency/throughput, rate limits, deprecation) cannot express, and they are exactly the kind of hard filter an enterprise caller wants:

• "open weights only" — availability/access subcomponent ≥ 3
• "commercially usable without attribution" — availability/license subcomponent ≥ 2
• "self-hostable and reproducible" — transparency subcomponents

These are hard filters over a licensing fact, not soft quality preferences, so they should sit alongside the existing must-clauses rather than in the weighted-score half. Worth naming in the query semantics even if the data lands later.

2. A concrete acceptance scenario, from real captured data.

The scenario list in Suggested Approach point 2 could use this one, because it is a genuine trap for naive ranking. From the spike's sample — same model, same benchmark scores, different serving providers:

	Kimi K3 @ Parasail	Kimi K3 @ Together AI
Cost per task	$0.78	$1.43
Median tokens/s	151	38
Median first chunk	1.96s	3.42s
P95 first chunk	2.32s	112.70s
Total response	18.57s	69.94s
Intelligence Index	60	60 (identical)

A ranker using median latency sees a ~1.7× difference and might reasonably pick either. A ranker aware of P95 sees a ~49× difference, and only one of these is viable behind a user-facing SLO. Since #85 supplies P5/P25/P75/P95 for both throughput and latency, the query layer should be able to express which percentile an SLO constraint binds against — "median latency under 2s" and "P95 latency under 5s" are different queries and the second is usually the one people mean.

Worse case in the same sample: Qwen3.5 397B A17B @ DigitalOcean reports 10 tokens/s and a 379s total response, against ~80 tok/s and ~47s for the same model on Alibaba Cloud and Novita. Any "cheapest provider above quality threshold X" logic walks straight into it — it is among the cheapest rows in the sample.

Both are good integration-test fixtures, and both argue for percentile-aware SLO semantics rather than single-value latency filters.