feat(engine): report real per-turn token usage in the envelope and wrapper events

[David Koleczek (DavidKoleczek)]

Summary

amplifier-agent run --output json reported metadata.tokensIn: 0 and tokensOut: 0 on every invocation regardless of actual usage. The CLI read those keys off the engine turn result, but nothing populated them, so the defaults always won.

Token data was reachable only by consuming usage display events off the ndjson stderr stream. That required --display ndjson and left every host to re-derive the totals itself, which is easy to get wrong: the engine emits one usage event per LLM call plus a trailing rollup carrying zero tokens, so a last-value-wins reader silently reports nothing.

The engine now accumulates usage at the display protocol point and reports it directly.

Why that placement

The accumulator sits upstream of the human-facing renderer, so the same numbers appear under --display text, --display ndjson, and --quiet. Usage is no longer a side effect of which renderer happens to be attached. Delegated sub-agent calls flow through the same point and are included in the total.

Envelope

Three new metadata fields:

cacheReadTokens   int
cacheWriteTokens  int
costUsd           str | null

tokensIn is the charged input total: inputTokens + cacheWriteTokens.

Per amplifier-core's docs/contracts/PROVIDER_CONTRACT.md, a provider's input_tokens is the gross total, fresh plus cache_read combined, and cache_write_tokens is the one bucket billed on top of the gross. cacheReadTokens is therefore a reported subset of the gross figure, not an addend, and is exposed for visibility rather than for summing. Cache reads are not added a second time.

costUsd is a decimal string, not a float, because a float loses monetary precision as soon as a host sums turns.

The three envelope paths differ deliberately:

success              reports real spend
in-turn error        reports what the turn spent BEFORE it failed
                     (previously hardcoded 0/0)
pre-boot / argv      still 0/0, and omits the three new fields.
                     No turn ran, so there is nothing to report.

Wrapper SDKs, both in parity

• ResultEvent gains session_id, turn_id, exit_code, usage, stderr_tail. ErrorEvent gains the first four. The parser previously shape-validated the envelope and then discarded everything but reply.
• New Usage type. Deliberate parity exception: cost_usd is a Decimal in Python but stays a string in TypeScript, since a JS number is IEEE-754 and would lose precision. Documented on both sides.
• Synthesized errors that never produced an envelope now carry the session id the wrapper already knows. turn_id stays absent, because the engine assigns turn ids and no envelope came back. Envelope values remain authoritative when one exists.
• New stderr_tail_bytes / stderrTailBytes session option, governing both terminal events:

positive int  ->  last N bytes
None / null   ->  the entire stderr buffer
0             ->  disabled

The default preserves existing behavior.

Bug fixed along the way

The stderr tail measured characters, not bytes. Both wrappers sliced a string, so STDERR_TAIL_BYTES was silently wrong on any non-ASCII stderr. It is now UTF-8 byte-accurate and never splits a codepoint, which means it may return slightly fewer than the requested bytes when trimming to a boundary.

Protocol

PROTOCOL_VERSION moves 0.3.0 to 0.4.0 for the additive envelope fields. Both wrappers, the conformance fixtures, and the documented pins move with it.

The two fixtures pinning the 2099-12-future-vN skew sentinel stay stale on purpose: they exist to prove the engine refuses a foreign protocol version.

Coverage

A new usage e2e suite. It is the first suite to drive a wrapper SDK inside the DTU, installing it from the same mirror the engine comes from so local wrapper changes are actually exercised.

The accuracy case pins the envelope against the provider's own per-call usage, summed out of the session event log. Its independence from UsageAccumulator is deliberate and load-bearing: the two must not share a helper, or the suite could only prove the accumulator summed events correctly and would be structurally blind to the formula itself. That blindness is not hypothetical, see below.

Corrections made during review

Two defects were found and fixed on this branch after the initial push, both worth reading before approving:

1. tokensIn double-counted cache reads. charged_input summed inputTokens + cacheReadTokens + cacheWriteTokens on the assumption the three were disjoint buckets. They are not, per the provider contract quoted above. On a cache-heavy turn, the normal case in an agent loop, this roughly doubled the reported input: one measured turn read 24,700 against a true charged input of 12,788. That is the same class of error this PR set out to fix, inverted, and worse, because a wrong zero looks broken while a plausible wrong number gets persisted downstream.

2. The e2e oracle carried the identical assumption, so tokensIn was being validated against a second implementation of the same wrong formula and passed. Both sides are corrected, and the oracle's independence is now stated as a deliberate property rather than left implicit.

Rebased onto current main

Merged main cleanly, picking up #154, #155, and #156. One interaction required a follow-up commit:

#155 wires up bridge_child_cost in spawn.py. UsageAccumulator's docstring had justified summing per-call cost partly on the grounds that the bridge did not exist, and said to revisit if it ever landed. No behavior changes, since the accumulator never reads sessionCostTotal, which is the only figure the bridge moves. The rationale is now restated on the two grounds that survive: sessionCostTotal is session-scoped rather than turn-scoped, and the bridge only fires on successful delegation, so a failed sub-agent's spend never reaches session.cost while its per-call events still arrive on the display stream.

#154 fixes the same cache-read double-count on the OpenAI wire, independently. Same root cause, different surface.

Verification

make verify             ALL GATES PASSED
  check                 ruff + format + pyright
  verify-codegen        schemas byte-identical to the generator
  verify-parity         10/10 fixtures, Python and TS runners agree
  verify-wrapper        21 files, 157 tests

e2e, post-merge         33 passed  (usage, raw_capture, run, modes)
e2e, pre-merge          82 passed, 4 skipped, 0 failed, fresh DTU

Also exercised by hand in the DTU, via both the CLI envelope and the Python SDK.

Impact

Coordinated cross-component. Engine and both wrapper SDKs must be released together: a wrapper pinning 0.3.0 will refuse a 0.4.0 engine, and vice versa. Merging is safe on its own, since both install paths resolve to the latest tagged release rather than tracking main.

Package versions are intentionally not bumped here; that belongs to the release process.

Reviewer notes

Two things not closed, neither of them regressions:

1. No conformance parity fixture was added. The conformance runners replay scripted JSON-RPC frames against a mock transport and never import a wrapper, and verify-parity.py reads its fixtures from a directory under src/, so the files in wrappers/conformance/fixtures/ are not executed by anything. There is no way to express wrapper-level parity in that harness today without extending it. Parity here is evidenced by byte-identical proof output across both languages and new TS parser cases, but it is not gated. Worth a follow-up.

2. No e2e case covers a turn that spends tokens and then fails. No deterministic trigger was found for an in-turn error after a real LLM call. The fields are present on the error path, but the interesting half of that behavior is untested.

🤖 Generated with Amplifier