Track LLM cost in Datadog LLM Obs for the BiggiePockets review

[bockets]

What this does

Attributes the cost of a BiggiePockets review per model, on the same Datadog LLM Obs
trace as the review-quality metrics. Both LLM spans in the trace — codex.review and
claude.synthesize — now carry what the pass used and, where it is known, what it
cost.

Where the money comes from

No rate table lives in this repository. A list price committed to a file goes stale
silently and would be reported with the same confidence as a real one. Each span
instead carries whichever of the two things Datadog needs:

• Token counts, for a model in Datadog's pricing catalog. It prices the span
  itself, from rates it maintains.
• A total_cost metric, for a model it does not carry — the amount OpenRouter
  reports charging, from the cost field it returns on every response.

Today that splits cleanly: gpt-5.6-sol is in the catalog, and
deepseek-v4.1-flash is not.

Making the catalog match

Datadog keys its catalog on the bare model and the provider that originated it, not
on a gateway slug. The workflow names models the way OpenRouter routes them, so
openai/gpt-5.6-sol is split into model_name: gpt-5.6-sol /
model_provider: openai, and the routing is kept as a gateway:openrouter tag.
Without that split nothing matches the catalog and no model gets priced at all.

Reading usage from each pass

The two tools record it differently:

• Claude Code writes a usage object to its execution output.
• Codex writes running token counters to a session rollout on its own runner, and
  openai/codex-action@v1 declares exactly one output (final-message) — no usage.
  So the Codex job reads that rollout itself and hands the totals to the reporting job
  through the handoff artifact that already exists.

Claude Code's own total_cost_usd is deliberately ignored. It is computed against
Anthropic's list prices, while these passes are billed by OpenRouter for a
non-Anthropic model, so it describes a bill nobody was sent.

Configuration

Two organization-level variables, both optional. No Datadog endpoint or org
identifier is committed to this repository.

Variable	Purpose	Default
DD_SITE	Datadog site to report to. A private or internal hostname belongs here and nowhere else.	datadoghq.com
DD_LLMOBS_ML_APP	LLM Obs ml_app the trace is grouped under.	biggiepockets-review

Cost reporting rides on the existing DATADOG_API_KEY secret; with no key the whole
step is skipped.

Safety

Only usage objects are read — never message content, transcripts, prompts, or diffs.
The helper never raises and always exits 0, the reporting step is
continue-on-error, and unusable usage data falls back to a valid empty object so a
cost problem cannot cost us the quality metrics in the same payload.

Testing

• 29 unit tests in tests/test_llm_usage.py; 38 total across the suite, all passing.
• Extracted the workflow's actual jq span program and ran it against real helper
  output over a Codex rollout fixture and a Claude execution fixture. Result:
  codex.review carries input_tokens: 31751 / output_tokens: 2367 / cache_read_input_tokens: 14720 / total_tokens: 34118 as gpt-5.6-sol + openai;
  claude.synthesize carries its counts plus total_cost: 0.0031 as
  deepseek-v4.1-flash + deepseek, with the bogus total_cost_usd: 9.99 in the
  fixture correctly ignored.
• Confirmed against Datadog's pricing catalog that gpt-5.6-sol is priced and
  deepseek-v4.1-flash returns skipped_unsupported_model_pricing, which is what
  makes the two paths necessary.

Not verified: an end-to-end run against Datadog itself, and whether OpenRouter's
cost field survives the Anthropic-compatible round trip into what Claude Code
writes to disk. Both need a real review run with the org variables configured. If
that field does not come through, the Claude span reports tokens and no cost rather
than a guess.