Skill: serving platform / CSP capability scraping (deprecation, TPM, RPM)

[DiTo97]

Take-home assignment. Self-contained and vertical: schema extension → skill definition → process doc → ingestion code → notifier integration → tests. Read the whole issue before starting; the acceptance criteria are the contract.

Scope narrowed (2026-08-13). This issue originally also covered latency and throughput. A feasibility spike established that Artificial Analysis publishes third-party measured latency and throughput for ~512 (model × provider) pairs, acquirable in a single page load — far better data than scraping vendor-claimed figures out of documentation, and available for many more platforms. Latency and throughput now belong to #85. This issue keeps deprecation and rate limits, which AA does not publish and which genuinely do require per-platform documentation scraping. See experiments/aa-scrape-spike/ on branch spike/aa-data-acquisition.

Problem

We know what models cost. We know nothing about whether we can actually run them: no rate limits, and no real deprecation information.

The only lifecycle signal in the repo is a regex heuristic in the notifier:

# services/notifier/src/notifier/service.py:33
DEPRECATED_STATUS_PATTERN = re.compile(
    r"(?<![\w-])(deprecated|legacy|retired|sunset|eol)(?![\w-])"
)

services/notifier/README.md:57-60 says plainly that upstream publishes no deprecation field, so this matches on free text and both false positives and false negatives are expected. There is no deprecation date, no policy window, and no successor model anywhere. Grepping for tpm|rpm|rate.?limit|slo returns nothing substantive.

The serving platforms and CSPs do publish this — deprecation schedules and rate limit tables per tier. It is spread across documentation pages in inconsistent formats, which is exactly why it fits a skill rather than a brittle scraper wired into CI.

Without this, tokenvalue cannot answer two of the questions that block production adoption: will this model still exist in six months, and can I get the throughput allowance I need. (The achieved throughput and latency question is answered by #85.)

Scope

Per (model, reasoning_effort, serving_platform) — the composite identity from #74 — capture and maintain:

• Deprecation. Announcement date, effective/shutdown date, the platform's stated policy window (e.g. "N months notice"), successor/replacement model where named, and the source URL. Distinguish announced from inferred. This should become the authoritative source that supersedes the notifier's regex guess.
• Rate limits. TPM and RPM, plus any other published limits that matter (daily caps, concurrency). These are almost always tier-dependent — the same model on the same platform has different limits per account tier — so tier must be part of the record, not averaged away.
• Skill + process doc. skills/<name>/SKILL.md following the conventions of skills/tokenpricing/SKILL.md, registered in .claude-plugin/marketplace.json, plus a per-platform process document: which documentation pages hold what, how the tables are laid out, what to do when a platform restructures its docs, when to record "not published" rather than guess.
• "Not published" is a real value. Most platforms will not publish most of these fields. Absent, unknown, and zero are three different things and must not collapse into one another.
• Deterministic ingestion. Capture raw, parse with tested code, validate against a schema. The harness does not write canonical records directly.
• Feeds the unified layer. These attributes attach to the offering entity from Data model: unified model x reasoning effort x serving platform pricing layer #74 — not a parallel side-table with its own identity scheme. They must sit alongside Skill: Artificial Analysis acquisition - provider leaderboard (model x platform) + Openness Index #85's performance attributes on the same offering, not duplicate them.
• Notifier integration. services/notifier already emits model_deprecated events off the regex. Once real deprecation data exists, wire it through so events fire on announced facts, with the heuristic demoted to a fallback. Do not break existing subscriptions or the webhook contract (docs/notifications.md).
• Coverage reporting. How many offerings have each attribute, per platform. We want to know how thin this data is, precisely.

A useful corroborating signal

The AA provider leaderboard (#85) filters to Status: Current by default, which excludes deprecated and legacy endpoints. Flipping that filter yields a set-difference that is a genuine third-party signal about which (model, provider) offerings have been retired — not authoritative, and no dates attached, but a useful cross-check against what the platforms themselves announce, and a way to catch deprecations we missed. Worth considering as a corroborating input; it does not replace the vendor documentation work.

Out of scope

• Latency and throughput. Moved to Skill: Artificial Analysis acquisition - provider leaderboard (model x platform) + Openness Index #85, which sources them from Artificial Analysis as third-party measured values with percentile spreads. Do not ingest vendor-claimed latency figures here — they would compete with better data on the same fields. If a platform's docs happen to publish a rate limit expressed as a throughput ceiling, that is a rate limit and belongs here; an advertised "typical tokens/sec" figure does not.
• The remote-database / SQLite migration. Issues Migration: JSON-driven database → derived SQLite read layer for faster UI & SDK queries #60 and SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1 #61 are open and cover a derived SQLite read layer. Do not touch them or build on them. Storage stays git-committed JSON.
• Workload-specific leaderboard / arena-per-agent generation. Deferred to a later phase, including trace ingestion and data-residency constraint solving.
• CI cron jobs, API keys, or paid subscriptions. Explicitly excluded — no live benchmarking, no authenticated API calls to measure limits empirically.
• Measuring latency or throughput ourselves. Out of scope everywhere in this programme.
• Region availability and data residency: not required. If your schema leaves room for them, good; do not build them out here.

Suggested approach

1. Survey three or four platforms with genuinely different documentation shapes before designing the schema — a hyperscaler, a first-party model vendor, and an inference specialist will pull in different directions. Design for the union, not for the easiest one.
2. Model tier explicitly and early. Rate limits without tier context are misleading, and retrofitting the tier axis later is painful.
3. Decide how you represent "not published" versus "no limit" versus "zero", and make it impossible to confuse them in the type system.
4. TDD parsers against trimmed real fixtures checked into the test tree, as services/sync/tests/ does. No network in tests.
5. Do the notifier integration last, behind the existing event contract, and add a regression test proving the heuristic still works when real data is absent.
6. When a platform's policy is prose rather than a date ("we provide 12 months notice"), capture the policy and any concrete date separately. They answer different questions.
7. Coordinate the offering-attribute shape with Skill: Artificial Analysis acquisition - provider leaderboard (model x platform) + Openness Index #85 so deprecation, rate limits and measured performance hang off one entity rather than three.

Acceptance criteria

• Schema extends the offering entity from Data model: unified model x reasoning effort x serving platform pricing layer #74 with deprecation and rate-limit attributes — no parallel identity scheme, and no collision with the performance attributes owned by Skill: Artificial Analysis acquisition - provider leaderboard (model x platform) + Openness Index #85.
• Deprecation records carry announcement date, effective date, policy window, successor where named, source URL, and an announced-vs-inferred flag.
• Rate limits are tier-aware; tier is part of the record.
• Absent / unknown / zero are structurally distinct and cannot be confused. A test covers each.
• skills/<name>/SKILL.md exists, follows existing skill conventions, is registered in .claude-plugin/marketplace.json, and ships with a per-platform process document.
• At least three serving platforms with materially different doc formats ingested end to end, each with a tested parser and no network in tests.
• Notifier emits deprecation events from real announced data where available, falls back to the existing heuristic otherwise, and the webhook contract in docs/notifications.md is unchanged. Existing notifier tests still pass.
• Coverage report per platform per attribute, produced by a checked-in command.
• Data lands in database/ per existing path conventions; unchanged upstream yields a no-op diff (tested).
• uv run pytest -q (python, sync, notifier) and uv run pre-commit run -a green.
• Docs updated, with explicit limitations — especially which platforms publish nothing and how thin the resulting coverage is.

Working agreement

Follow the self-contained-task workflow in libraries/python/AGENTS.md:29-37: linked branch via gh issue develop <issue> --checkout, draft PR early with plan and acceptance criteria, then plan → review → TDD (failing first) → implement minimally → verify → report.

uv for Python. CI is path-filtered per package; the notifier has its own lint and test workflows that also trigger on libraries/python/**.

Three platforms modelled carefully, with honest coverage numbers and clear absence semantics, beats ten platforms of half-trusted scraped figures.