Skip to content

fix: exclude zero-token <synthetic> records from Claude executions - #50

Open
samwagg wants to merge 1 commit into
splunk:mainfrom
samwagg:fix/claude-synthetic-pseudo-model-executions
Open

samwagg wants to merge 1 commit into
splunk:mainfrom
samwagg:fix/claude-synthetic-pseudo-model-executions

Conversation

@samwagg

@samwagg samwagg commented Sep 17, 2026

Copy link
Copy Markdown

Problem

Claude Code writes locally generated messages into the session transcript with the pseudo-model name <synthetic> and no token evidence. Both Claude adapter loops counted these as real model executions.

aggregates.py already drops a zero-usage <synthetic> row from the Models page (placeholder_model), and cost coverage is correct. The defect is upstream of that, in the Claude adapter itself, so the session's own evidence is wrong:

Value Before Expected
primary_model <synthetic> claude-sonnet-4-6
session models ['<synthetic>', 'sonnet-4-6'] ['sonnet-4-6']
context.latest 0 10010
executions 2 1

Two consequences worth calling out:

  • context.latest becomes a measured zero. A real 10,010-token context is reported as 0 because the trailing synthetic record overwrites it. This is the case AGENTS.md and CONTRIBUTING.md both name directly: "Unavailable evidence must not become a measured zero."
  • The phantom execution inflates per-execution metrics. executions feeds Output/execution on the Efficiency page, so a session's fragmentation signal is skewed by a record that never called a model.

Cost, cost_approx, and total_cost are not affected — that part is already correct on main.

What changed

token_meter/runtimes/claude.py — skip a record when it has both the pseudo-model name shape (<...>) and no token evidence across input_tokens, output_tokens, cache_read_input_tokens, and cache_creation_input_tokens.

Requiring both conditions is deliberate: a record that does report billable tokens is still priced normally, so the guard cannot silently hide real usage. The guard sits before the input_complete/output_complete tracking so a non-execution cannot degrade completeness flags either.

Tests

I extended the existing test_claude_summary_ignores_zero_usage_synthetic_marker_for_cost_coverage with the model, context, and sample assertions rather than adding a near-duplicate test, plus two new cases:

  • test_claude_recompute_keeps_cost_available_with_synthetic_records — covers the recompute path and the execution count.
  • test_claude_summary_prices_pseudo_model_records_that_report_tokens — asserts a pseudo-model record that does carry tokens is still priced. This one passes both before and after; it is a regression guard on the guard's scope, not a bug demonstration.

Verified red-to-green by running the updated test file against unmodified main:

FAIL: test_claude_summary_ignores_zero_usage_synthetic_marker_for_cost_coverage
AssertionError: '<synthetic>' != 'claude-sonnet-4-6'
FAIL: test_claude_recompute_keeps_cost_available_with_synthetic_records
AssertionError: 2 != 1

Both pass with the fix.

Note on fixtures: the cache fixtures in the touched tests gained the cache_creation ephemeral_5m/1h breakdown. Without it, duration-aware normalization correctly reports cost as unavailable, which masked what these tests were meant to assert. Token totals are unchanged.

Validation

python3 -m unittest discover -s tests          # 963 tests, 16 skipped
PYTHONPYCACHEPREFIX=... python3 -m py_compile meter.py token_meter_mcp.py $(find token_meter -type f -name '*.py' -print)
node -e "...page.html embedded JS..."          # js ok
bash -n scripts/install scripts/install-linux ... scripts/update-linux
swiftc menubar/TokenMeterMenuBar.swift -o /private/tmp/token-meter-menubar
TOKEN_METER_MENUBAR_SMOKE=1 /private/tmp/token-meter-menubar
git diff --check

Installed runtime (macOS 15.6, ./scripts/install):

  • /healthok: true, state_ready: true, inventory_ready: true, 69 sources, no runtime_adapter_failures
  • /menubar → valid compact payload
  • com.token-meter.server and com.token-meter.menubar both running, exit status 0
  • Source/staged-runtime parity confirmed for token_meter/runtimes/claude.py

Limitations

  • One pre-existing failure on main, unrelated to this change: PiDocumentationTests.test_docs_explain_pi_evidence_and_privacy_boundaries asserts "Pi coding-agent sessions" appears in README.md, but that content now lives in specs/USER_GUIDE.md. Reproduced on a clean main checkout before these edits. Left untouched to keep this PR focused — happy to open a separate issue or fix it.
  • No dashboard markup or styling changed, so no screenshot is included. The corrected values surface through existing session fields.
  • macOS only for native verification; Linux tray and Windows NotifyIcon behavior is unverified by me.

Claude Code writes locally generated messages with the pseudo-model name
`<synthetic>` and no token evidence. Both Claude adapter paths treated
these as real model executions, so a session containing one reported:

- `primary_model` as `<synthetic>` instead of the real model
- a phantom `<synthetic>` entry in the session model list
- `context.latest` as 0, collapsing real context into a measured zero
- one extra execution, inflating per-execution derived metrics

Skip records that have both the pseudo-model name shape and no token
evidence. Requiring both conditions keeps a record that does carry
billable tokens priced normally, so the guard cannot hide real usage.

The existing zero-usage synthetic test is extended with the model,
context, and execution assertions rather than duplicated. Its cache
fixtures gain the `cache_creation` duration breakdown now required by
duration-aware normalization.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant