Track LLM cost in Datadog LLM Obs for the BiggiePockets review

.github/workflows/biggiepockets-review.yml

@@ -9,7 +9,13 @@ name: BiggiePockets Code Review (reusable)
 # Secrets are supplied by the caller via `secrets: inherit`:
 #   OPENROUTER_API_KEY, JIRA_EMAIL, JIRA_API_TOKEN,
 #   BIGGIEPOCKETS_PAT, DATADOG_API_KEY (optional; enables review-quality
-#   metrics in Datadog LLM Obs).
+#   metrics and cost tracking in Datadog LLM Obs).
+#
+# Cost tracking: each LLM span in the review trace carries the token counts and,
+# where OpenRouter reported one, the cost the pass was charged, so Datadog attributes
+# spend per model on the same trace as the quality metrics. Two optional org-level
+# variables say where that lands: DD_SITE and DD_LLMOBS_ML_APP. No Datadog endpoint
+# or org identifier is committed here. See README.md.
 #
 # Prompt versioning: every review-stage prompt lives in this repo's prompts/
 # registry (BiggerPockets/.github) as a versioned template, plus one shared rule
@@ -246,12 +252,17 @@ jobs:
         run: |
           codex_end_ns=$(date +%s%N)
           echo "CODEX_END_NS=$codex_end_ns" >> "$GITHUB_ENV"
+          # Codex writes its token counters to a session rollout on this runner, and
+          # the action exposes no usage output, so read them here and hand them to the
+          # job that reports the trace. One line, because the handoff is read as env.
+          codex_usage=$(python3 registry/scripts/llm-usage.py codex "${CODEX_MODEL:-unspecified}" | tr -d '\n') || codex_usage=""
           {
             echo "BASE_REF=$BASE_REF"
             echo "REVIEW_START_NS=$REVIEW_START_NS"
             echo "CODEX_START_NS=$CODEX_START_NS"
             echo "CODEX_END_NS=$codex_end_ns"
             echo "CODEX_STATUS=${{ steps.codex.outcome }}"
+            echo "CODEX_USAGE_JSON=$codex_usage"
           } > review-context.env
           touch codex-findings.md
 
@@ -418,6 +429,11 @@ jobs:
         continue-on-error: true
         env:
           DD_API_KEY: ${{ secrets.DATADOG_API_KEY }}
+          # Site and ml_app come from org-level variables so no Datadog endpoint or
+          # org identifier is committed here; both fall back to public defaults.
+          DD_SITE: ${{ vars.DD_SITE }}
+          DD_LLMOBS_ML_APP: ${{ vars.DD_LLMOBS_ML_APP }}
+          EXECUTION_FILE: ${{ steps.claude.outputs.execution_file }}
           CODEX_PROMPT_TEMPLATE: ${{ steps.resolve.outputs.codex_prompt_template }}
           ARM_PROMPT_TEMPLATE: ${{ steps.resolve.outputs.arm_prompt_template }}
         run: |
@@ -508,6 +524,30 @@ jobs:
           codex_model_tag="${CODEX_MODEL:-unspecified}"
           claude_model_tag="${CLAUDE_MODEL:-unspecified}"
 
+          dd_site="${DD_SITE:-datadoghq.com}"
+          ml_app="${DD_LLMOBS_ML_APP:-biggiepockets-review}"
+
+          # Cost tracking. Each pass carries its own usage as span metrics plus a
+          # model identity split into the bare name and originating provider that
+          # Datadog's pricing catalog keys on ("openai/gpt-5.6-sol" -> gpt-5.6-sol +
+          # openai); the OpenRouter gateway is kept as a tag instead. Datadog then
+          # prices a catalogued model from its token counts, and takes the reported
+          # `total_cost` at face value for one it doesn't carry. Both numbers come
+          # from what the tools recorded — OpenRouter reports what it charged, and
+          # llm-usage.py holds no rate table of its own.
+          #
+          # Only usage objects are read, never message content. The Codex pass's
+          # counters travel from its own job in CODEX_USAGE_JSON, since the action
+          # exposes no usage output and its rollout stays on that runner.
+          # Fall back to an empty object if the helper can't run at all, so a cost
+          # problem can't cost us the quality metrics in the same payload.
+          no_fields='{"model_name":"unspecified","model_provider":"unspecified","metrics":{}}'
+          codex_fields="${CODEX_USAGE_JSON:-}"
+          claude_fields=$(python3 registry/scripts/llm-usage.py claude "$claude_model_tag" "${EXECUTION_FILE:-}") || claude_fields="$no_fields"
+          usable='has("model_name") and has("model_provider") and has("metrics")'
+          echo "$codex_fields" | jq -e "$usable" >/dev/null 2>&1 || codex_fields="$no_fields"
+          echo "$claude_fields" | jq -e "$usable" >/dev/null 2>&1 || claude_fields="$no_fields"
+
           run_id="$GITHUB_REPOSITORY-pr$PR-${{ github.run_id }}"
 
           trace_id=$(openssl rand -hex 16)
@@ -534,6 +574,8 @@ jobs:
             --arg codex_status "$codex_status" \
             --arg claude_status "$claude_status" \
             --arg root_status "$root_status" \
+            --argjson codex_fields "$codex_fields" \
+            --argjson claude_fields "$claude_fields" \
             --arg codex_findings_excerpt "$codex_findings_excerpt" \
             --arg summary_excerpt "$summary_excerpt" \
             --argjson codex_prompt "$codex_prompt_obj" \
@@ -561,10 +603,11 @@ jobs:
                 start_ns: $codex_start_ns,
                 duration: $codex_duration,
                 status: $codex_status,
+                metrics: $codex_fields.metrics,
                 meta: {
                   kind: "llm",
-                  model_name: $codex_model,
-                  model_provider: "openrouter",
+                  model_name: $codex_fields.model_name,
+                  model_provider: $codex_fields.model_provider,
                   input: {
                     messages: [{role: "user", content: "Review pr:\($pr) against ticket.json/pr.diff"}],
                     prompt: $codex_prompt
@@ -580,10 +623,11 @@ jobs:
                 start_ns: $claude_start_ns,
                 duration: $claude_duration,
                 status: $claude_status,
+                metrics: $claude_fields.metrics,
                 meta: {
                   kind: "llm",
-                  model_name: $claude_model,
-                  model_provider: "openrouter",
+                  model_name: $claude_fields.model_name,
+                  model_provider: $claude_fields.model_provider,
                   input: {
                     messages: [{role: "user", content: $codex_findings_excerpt}],
                     prompt: $arm_prompt
@@ -594,7 +638,7 @@ jobs:
             ]')
 
           payload=$(jq -n \
-            --arg ml_app "biggiepockets-review" \
+            --arg ml_app "$ml_app" \
             --arg repo "$GITHUB_REPOSITORY" \
             --arg pr "$PR" \
             --arg run_id "$run_id" \
@@ -629,6 +673,7 @@ jobs:
                     "control_arm:\($control_arm)",
                     "assignment_bucket:\($assignment_bucket)",
                     "experiment_split_percent:\($split_percent)",
+                    "gateway:openrouter",
                     "codex_model:\($codex_model)",
                     "claude_model:\($claude_model)",
                     "codex_findings_lines:\($codex_findings_lines)",
@@ -641,7 +686,7 @@ jobs:
             }')
 
           http_code=$(curl -sS -o /tmp/dd_response.json -w '%{http_code}' \
-            -X POST "https://api.datadoghq.com/api/intake/llm-obs/v1/trace/spans" \
+            -X POST "https://api.${dd_site}/api/intake/llm-obs/v1/trace/spans" \
             -H "Content-Type: application/json" \
             -H "DD-API-KEY: $DD_API_KEY" \
             -d "$payload") || http_code="000"

.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc

README.md

@@ -100,6 +100,50 @@ run and post normally without it, but no metrics are reported.
 The exact secret names each step expects are visible in the `env:` and `with:` blocks of
 [`.github/workflows/biggiepockets-review.yml`](.github/workflows/biggiepockets-review.yml).
 
+#### Cost tracking (Datadog LLM Observability)
+
+`secrets.DATADOG_API_KEY` also turns on cost tracking. Every LLM span in the review
+trace carries its token usage — input, output, total, and cached-read counts — under
+`metrics`, and Datadog prices the span from its own model pricing catalog. Cost is
+therefore attributed per pass and per model on the same trace as the quality metrics,
+and shows up in LLM Observability's spend views without a separate report.
+
+For the catalog to recognise a model, a span has to name it the way Datadog does: the
+bare model and the provider that originated it. The workflow runs everything through
+OpenRouter, whose slugs look like `openai/gpt-5.6-sol`, so `scripts/llm-usage.py`
+splits the slug into `model_name: gpt-5.6-sol` / `model_provider: openai` and records
+the routing as a `gateway:openrouter` tag.
+
+Cost itself is never computed here, and there is no rate table in the repository: a
+list price committed to a file goes stale silently and would be reported with the same
+confidence as a real one. Instead each span carries whichever of the two things
+Datadog needs. For a model in the catalog, token counts are enough. For one it does
+not carry, the span reports a `total_cost` metric — the amount OpenRouter says it
+charged, taken from the `cost` field it returns on every response, where that figure
+survives into what the tool wrote to disk.
+
+The two passes record their usage differently. Claude Code writes a `usage` object to
+its execution output. Codex writes running token counters to a session rollout on its
+own runner, and since its action exposes no usage output, the workflow reads that
+rollout in the Codex job and hands the totals to the reporting job. In both cases only
+usage objects are read — never message content, transcripts, prompts, or diffs. Claude
+Code's own `total_cost_usd` is ignored: it is computed against Anthropic's list prices,
+while these passes are billed by OpenRouter for a non-Anthropic model.
+
+Two **organization-level variables** (`vars`, not secrets — **Settings → Secrets and
+variables → Actions → Variables** at the org level) configure where the trace lands.
+Both are optional, and nothing here is committed to the repository:
+
+- `DD_SITE` — the Datadog site to report to (e.g. `datadoghq.eu`, `us5.datadoghq.com`).
+  Defaults to the public `datadoghq.com`. Set this to the organization's actual site;
+  a private or internal Datadog hostname belongs in this variable and nowhere else.
+- `DD_LLMOBS_ML_APP` — the LLM Obs `ml_app` the review trace is grouped under.
+  Defaults to `biggiepockets-review`.
+
+Cost tracking is best-effort and never fails a review. With no `DATADOG_API_KEY` the
+whole reporting step is skipped, and a missing execution file, an absent rollout, or
+malformed usage data degrades to fewer metrics on the span.
+
 #### 4. Set workflow permissions
 
 The reusable workflow's jobs need `pull-requests: read` and `id-token: write` (OIDC