proxy: an unrecognised usage dialect is indistinguishable from no usage, and degraded silently for 4,015 requests

[amiddavid]

The defect

parseUsage returning ok=false means one of three different things, and nothing distinguishes them:

1. the provider genuinely reported no usage (an OpenAI response without stream_options, for instance);
2. the response carried usage in a spelling the parser does not recognise;
3. usage was present but every tier was zero.

Only the first is benign. The second is a measurement outage, and it degrades silently: a healthy 200, correct savings counters, correct latency, correct everything else — and fresh_input_tokens, cache_read_tokens, cache_write_tokens quietly at 0.

It has already happened, at full scale, for two iterations

Iteration 024 recorded usage_reported=false on 4,015 of 4,015 requests (1,808 arm A, 2,207 arm B, zero exceptions), with all four token fields at 0, every request provider=anthropic route=/v1/messages sse=False buffered=False.

Nothing surfaced it. It was found in a post-mortem two iterations later, while trying to answer a different question — and only because someone went looking for why a cost figure could not be bounded.

The usage was there. LOCA reported Avg Cost: $2.390637 per arm-seed, and litellm computes cost from the response's usage, so the body that flowed gateway → proxy → harness did carry it. The proxy passed it through and parseUsage found nothing in it.

Why a counter, and why this specific split

The parser handles three spellings and silently fails on others. Measured directly over candidate shapes:

Shape	parseUsage
Anthropic usage.input_tokens	parses
OpenAI usage.prompt_tokens	parses
Output-only usage.output_tokens	parses
Bedrock Converse camelCase usage.inputTokens	finds nothing
usage nested (e.g. response.usage)	finds nothing
usage absent	correctly nothing
usage present, all tiers zero	correctly nothing

So an unrecognised dialect is indistinguishable from a provider that said nothing — which is the same failure the codebase already refuses to accept in two other places:

• expand/unresolved.go splits malformed from missing, because a bare miss cannot tell them apart and they call for opposite responses;
• fix(store): give rewind payloads their own reserve, and refuse a removal whose original cannot be stored #188 split stash_missing from stash_refused for exactly the same reason — one counter was reporting both "nothing broke" and "something broke".

This is the third instance of that pattern, and the argument is identical.

Suggested fix

1. A counter — usage_unparsed at /stats and cg_usage_unparsed_total at /metrics: a response arrived, it was not empty, and no recognised usage spelling was found in it. Non-zero means the proxy is not accounting tokens for some route or provider, which is currently invisible.
2. A DEBUG record of the shape on the first such response per process — the top-level keys and the keys under usage, not the body (it carries content). That is the datum that would turn "usage_reported is false" into "the provider is sending camelCase" without needing a body dump at all.
3. Optionally distinguish case 3 (present but all-zero) in the log line, since it is legitimate on some responses and would otherwise inflate the counter.

Deliberately not part of this: adding the camelCase or nested dialects. That needs a real response body to get the field names right — guessing between cacheWriteInputTokens and cacheCreationInputTokens produces a parser that looks correct and reads zero, which is the failure this issue is about, reproduced in the code meant to fix it. The counter and the shape record are dialect-agnostic and are what make the next gap visible in one run rather than two iterations.

How it was found

Tracing why iteration 024 had no cache-token measurement, with the coref validation session — which also made the point that a counter here would have surfaced this in iteration 022 rather than in iteration 024's post-mortem. Related: #199 (needs these fields live to bound its cost question), #190 (needs them to check whether refusals suppress the recovery).

[amiddavid]

The shape record removes a dependency, which raises this issue's priority

Worth stating explicitly, because it changes what #200 is for. Bullet 2 — a DEBUG record of the response's shape (top-level keys plus the keys under usage, never the body) — answers "which dialect is this provider sending" on its own. So:

• the dialect fix stops waiting on a captured body. offload: the replay depth-gate bypass is justified per-message but keyed per-content, so a replay can splice at depth into uncompacted content #199 and the camelCase question currently wait on one instrumented benchmark run producing a raw response. If proxy: an unrecognised usage dialect is indistinguishable from no usage, and degraded silently for 4,015 requests #200 lands first, the shape record supplies the same answer from any deployment that hits the gap.
• it works for anyone who hits this, not only for someone with a capture hop in the request path. The alternative instrument — a hop-side body dump — exists in exactly one rig.
• it never touches content. A body dump on this workload writes several kilobytes of transcript to disk; the key-shape record cannot, by construction, because it records key names only.

The coref validation session had instrumented its capture hop to dump the first response body per run and, on seeing this, retargeted: #200's record is the primary route and the hop dump is the fallback (it keeps the dump only because it also shows Content-Encoding, which the key-shape record would not).

So this is not only an observability gap to close eventually — it is the cheapest instrument for diagnosing the dialect gap that #199 and the parseUsage fix are both blocked on, and it needs no rig, no extra requests and no content on disk.

[amiddavid]

The generalised rule, stated here so it can be cited

Recorded in this issue because three separate places in the tree now make versions of the same argument locally, and one statement they can point at beats each of them making it badly. This is the citable form until it lands in docs/design.md.

A counter must not report both a benign outcome and a broken promise.

The tell: when adding or reviewing a counter, ask what else could increment it, and whether any of those things would want the opposite response from the operator. If so, split it, and say in the help text which one is alertable.

Why it keeps happening: the benign case is what the help text promises, so the dangerous case hides behind a reassuring description — and the failure is silent by construction. Nothing errors, the request succeeds, and the only thing that moves is a number nobody is alerting on.

The three instances

Where	Benign	Broken promise	Opposite responses
expand/unresolved.go	malformed — the model invented an id	missing — we lost a payload we promised	ignore vs. page
#188 (components/offload)	stash_refused — a removal was declined, content left verbatim	stash_missing — a dangling marker went upstream	raise max_entries / stash_max_bytes vs. raise ttl_seconds
this issue (proxy/usage.go)	provider genuinely reported no usage	an unrecognised dialect silently zeroed three token fields	nothing vs. token accounting is offline

Instance 3 ran for 4,015 of 4,015 requests in one benchmark iteration and was found two iterations later, in a post-mortem chasing a different question. Instance 1's counter sat at 209 looking like ordinary expand traffic for the same reason.

Corollary, specific to parsers

When a parser can silently fail to recognise an input shape, do not only count the failure — record the shape: key names only, never the body, since bodies carry transcript content. That turns "we could not parse this" into "the provider is sending camelCase", which is the difference between an open question and a one-line fix. It is bullet 2 of this issue, and it is why this issue unblocks the parseUsage dialect gap rather than merely reporting it.