feat(aa-sync): daily Artificial Analysis acquisition workload

[DiTo97]

Turns the experiments/aa-scrape-spike proof-of-concept into a production service: a provider × model × reasoning-effort offering dataset joined to the Artificial Analysis Openness Index, refreshed daily.

Correction: the original design assumption was wrong

This PR originally shipped on two premises that turned out to be false, and the follow-up commit rewrites the acquisition around what the pages actually serve.

What it assumed: AA's public pages carry no per-token pricing (only Cost per Task USD), and the per-provider /providers/<slug> pages are the only place to get the superseded offerings the aggregate leaderboard hides. So the service fetched ~58 provider pages and parsed their 12-column HTML tables.

What is actually true: AA is a Next.js app, and every page streams its data into the initial HTML as self.__next_f.push([1,"…") chunks. The rendered table is a lossy view of that payload in both directions:

• Columns. "Expand Columns" is a client-side toggle over data that already shipped — expanding issues no network request. The 50-column view, per-token pricing and cache pricing included, was in the initial HTML all along. There was never a leaderboard-column-backfill to remove: the leaderboard was only ever used for slug discovery. The 12-column limit was a property of the default render, not of the response.

• Rows. The leaderboard renders only non-deprecated offerings (Status: Current) while its payload carries every offering:

  	rendered	payload	deprecated
  Nebius	25	35	10
  Azure	20	84	64
  all providers	516	1082	566

  rendered + deprecated == payload exactly for both, so Status: Current is deprecated == false — applied client-side to a payload that already contains everything.

So the provider pages were the redundant path, not the leaderboard. Every offering id on /providers/nebius is present in the leaderboard payload, and a per-key comparison found zero fields with a real value on the provider page but missing from the leaderboard, zero the reverse, and zero conflicts — the only differences were React's "$undefined" sentinel versus an omitted key, which normalise to the same None. 58 requests bought a strict subset of one.

Acquisition is now two GETs: the provider leaderboard and the Openness Index.

What the service does

1082 offerings across 58 providers, 62 published fields each, up from 1045 across 51. Superseded offerings are kept and flagged deprecated rather than filtered at acquisition time, so a model being replaced never punches a hole in price history; filter at query time instead.

Keyed on offering_id — AA's own uuid — because no composite of the human-readable fields is unique. Tested over all 1082 rows:

Candidate key	Duplicate groups
offering_id	0
(provider, model_slug, host_api_id, display_name)	4
(provider, model_slug, display_name)	6
(provider, model_slug)	23
(provider, host_api_id)	248

The irreducible collisions are real distinct endpoints AA does not name apart: openai/o3 appears twice under one host_api_id with the same label, differing only in price ($10/$40 vs $2/$8 per 1M) and whether performance was measured. There is no tier, region or variant field to separate them. normalize_sources asserts id uniqueness every run.

The openness join is an exact model_slug match. Openness records live in their own id space and reference a model by modelId, a uuid that appears nowhere in the leaderboard payload (0 of 298) — so offering_id cannot anchor this join. The openness page does supply a slug for all 298 scored models, and the leaderboard carries model_slug on every offering. That removes the name normalisation, the creator filter, the three confidence tiers and the serving-token vocabulary the display-name join required, and coverage improves: 698 matched against 563, with 0 ambiguous instead of 1.

Reasoning-effort grain is native. 299 labels carry an explicit effort token and 78 (provider, base model) groups span more than one effort. Each variant is its own offering with its own metrics; max owns the unsuffixed slug. Per-token price is constant across efforts while cost_per_task_usd varies, so effort matters economically even though pricing does not change with it.

No value cleaning. The payload is typed JSON, so the em-dash / U+2212 minus / currency-symbol / thousands-separator / trailing-asterisk handling is gone along with the table parsing that needed it. intelligence_index is 45.1382483763163 rather than 45, and intelligence_index_estimated is a real boolean rather than a trailing *.

Per-token pricing is no longer the blocker on a PricingInfo crosswalk that this PR originally described — input_price_usd_per_1m, output_price_usd_per_1m and both cache prices are now published. The remaining reason the dataset stays separate from prices.json is grain, not missing fields.

Failure and drift alerting

Three failure modes alert under their own names, each opening — or commenting on, never duplicating — one issue labelled aa-sync-failure:

Alert	Means	Detected by
payload-not-found	AA no longer ships data as a flight payload in the initial HTML	flight.reconstruct_payload
request-blocked	persistent non-200 or transport failure after retries	fetch.get_page
schema-drift	payload parsed but a field vanished, changed type, or changed units	schema.check_drift

Retryable statuses (408/425/429/5xx) get 4 attempts with 2s/8s/30s backoff; a refusal such as 403 is not retried, because repeating it will not change the answer.

Drift is checked against a tracked schema/offering-manifest.json, because "the pipeline did not crash" is not evidence the source is unchanged. A vanished or retyped field is breaking; so is a numeric field whose values move >10× outside their recorded span, which is what a unit change looks like from outside (fractions rescaled to percentages, per-1M prices restated per-1k) and which parses perfectly while meaning something else. A new field is informational only — additive change is how AA ships new benchmarks and never fails a run. On breaking drift the run alerts and publishes nothing: a stale snapshot is recoverable, a silently wrong one is not.

Delivery is one function, deliver_via_github_issue. Any (Alert) -> str callable is a channel, so routing to Teams or email later is a single new function and no other module changes.

Verification

89 tests. Each of the three failure modes is driven end-to-end against a deliberately broken input — a page with its 36 flight chunks stripped, a mocked persistent 403, and the real 1082-row payload with price1mInputTokens rescaled 1000× — asserting the alert kind, title and body that would be delivered. Delivery itself is injected in tests, so no test touches the network.

Sync moves to daily now that a run is two requests instead of ~60.

Refs #85, #78, #76