Skill: arena.ai category score scraping for model + reasoning-effort variants

[DiTo97]

Take-home assignment. Self-contained and vertical: skill definition → process doc → deterministic ingestion code → storage → tests. Read the whole issue before starting; the acceptance criteria are the contract.

Problem

We have no signal about how good a model is — only what it costs. database/current/prices.json carries pricing, context window, and three capability booleans, and nothing else. Grepping the tree for arena|lmarena|leaderboard|elo returns zero substantive hits.

For the tokenvalue evolution we need arena.ai category scores attached to model variants, refreshed over time, so that "cheapest model above X on coding" becomes answerable.

The delivery mechanism matters as much as the data. We are deliberately not building this as a CI cron job: arena.ai has no stable public API we hold a key for, and we do not want the canonical dataset to depend on paid subscriptions or on scrapers that rot silently inside a scheduled workflow. Instead the unit of work is a skill — an agentskills.io-style SKILL.md plus an explicit process document — that any AI harness can execute, today by hand and later on a schedule, with deterministic code doing the parsing and validation so the harness is never trusted with correctness.

The repo has exactly one skill today (skills/tokenpricing/SKILL.md, registered via .claude-plugin/marketplace.json). It is a usage skill, not a data-acquisition skill. You are establishing the pattern for the latter.

Scope

A skill that fetches and refreshes arena.ai scores per (model, reasoning_effort) and per arena category, keyed to the identity introduced in the data-model foundation issue.

• Skill + process doc. skills/<name>/SKILL.md with frontmatter matching the existing skill's conventions, plus a process document a harness follows step by step: where to go, what to extract, how to handle pagination or partial loads, what to do when the page shape changes, when to abort rather than guess.
• Harness-agnostic. No assumption of a specific agent runtime, no vendor-specific tool names in the process doc beyond what is genuinely needed. It must be executable by a human following the same steps.
• Deterministic ingestion. The skill's output is a raw captured payload. A checked-in parser/normalizer — real code, tested — turns that into validated records. Do not let the model free-form JSON directly into the canonical dataset; capture, then parse, then validate.
• Identity resolution. Arena's display names will not match our model_id values, and arena frequently encodes reasoning effort into the display name. You need an alias/mapping layer with an explicit confidence notion, and an unresolved report: every arena entry we could not confidently key must be surfaced, not silently dropped. Coverage is a number we want to see.
• Storage. A place in database/ for arena data, with the same discipline as the existing pipeline: current snapshot, and enough history to see movement over time. Follow the conventions in services/sync/src/tokenpricing_sync/paths.py and cli.py.
• Idempotent refresh. Re-running against unchanged upstream data produces no spurious diff. This matters because the sync pipeline commits to main and a noisy diff pollutes the changelog.
• Schedulability. Structure the work so a future scheduled runner can invoke it unattended — a clear entrypoint, machine-readable success/failure, no interactive prompts. Do not actually add a scheduled workflow.

Out of scope

• The remote-database / SQLite migration. Issues Migration: JSON-driven database → derived SQLite read layer for faster UI & SDK queries #60 and SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1 #61 are open and cover a derived SQLite read layer. Do not touch them or build on them. Storage stays git-committed JSON.
• Workload-specific leaderboard / arena-per-agent generation. Deferred to a later phase.
• Adding a CI cron job or anything requiring an API key or paid subscription we do not already hold. Explicitly excluded. The existing database-sync.yml cron is not a template to copy here.
• Estimating scores for variants arena does not cover — that is the category→benchmark estimation issue.
• Public benchmark cataloguing — separate issue.

Suggested approach

1. Start by characterising what arena.ai actually publishes: which categories exist, what the score scale and its direction are, whether confidence intervals or vote counts are exposed, how reasoning effort appears in names, how often it changes. Write that down before designing the schema.
2. Design the record shape around the composite identity from the data-model issue. If that issue has not landed yet, define the minimal identity you need and note the coupling in your PR — do not invent a competing identity scheme.
3. Write the parser against captured fixtures first, TDD-style. Tests must not hit the network. Trimmed real payloads checked into the test tree, as the sync service already does.
4. Then write SKILL.md and the process doc, and validate them the honest way: have someone (or a fresh harness session) follow the doc cold and see whether they produce a payload your parser accepts. If they can't, the doc is wrong, not the person.
5. Treat score staleness as a first-class field. A score without a capture timestamp is not usable for the ranking work later.

Acceptance criteria

• skills/<name>/SKILL.md exists, follows the frontmatter conventions of skills/tokenpricing/SKILL.md, and is registered in .claude-plugin/marketplace.json.
• A process document describes the acquisition procedure precisely enough for a human or any harness to follow without prior context, including failure and page-shape-change handling.
• Checked-in, tested parser/normalizer converts a captured payload into validated records — schema-validated, not free-form.
• Records are keyed by (model, reasoning_effort) consistent with the data-model foundation issue, plus arena category and a capture timestamp.
• Identity resolution produces a coverage figure and an explicit unresolved list; nothing is silently dropped.
• Arena data lands in database/ following existing path conventions, with current state and history.
• Re-running with unchanged upstream data yields a no-op diff (test this).
• Tests pass offline with no network access. uv run pytest -q and uv run pre-commit run -a green.
• Docs updated: how to run the skill, what the data means, what its known limitations are. Do not overstate the reliability of scraped scores.

Working agreement

Follow the self-contained-task workflow in libraries/python/AGENTS.md:29-37: linked branch via gh issue develop <issue> --checkout, draft PR early with plan and acceptance criteria, then plan → review → TDD (failing first) → implement minimally → verify → report.

uv for Python. CI is path-filtered per package; pushes to main touching libraries/** auto-release, so keep the blast radius intentional.

Be candid in the PR about what your scraper will not survive. A skill that admits its brittleness and fails loudly beats one that quietly returns stale numbers.

[DiTo97]

Pattern coordination with #85

#85 (Artificial Analysis acquisition) is a second data-acquisition skill with the same shape as this one: leaderboard source → raw capture → tested parser → identity resolution with an unresolved report → database/ with history → idempotent refresh.

This issue should land the pattern; #85 should follow it, not invent a parallel one. Whoever picks up #85 has been told to defer to the conventions established here if this has landed first.

One finding from the #85 spike that is directly relevant, since this issue notes "arena frequently encodes reasoning effort into the display name": AA does the same thing, and does it inconsistently — the same effort level is in the slug for some models and only in the display name for others, and two genuinely different variants can share one display name. Full evidence in the comment on #74. If arena turns out to behave similarly, the alias/mapping layer with an explicit confidence notion that this issue already requires is the right shape — just budget for it being messier than a name-normalisation function.