feat(engine): report real per-turn token usage in the envelope and wrapper events

CHANGELOG.md

@@ -9,6 +9,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Real per-turn token and cost usage, on the envelope and on both wrapper SDKs.**
+  `metadata.tokensIn` / `metadata.tokensOut` were previously hardcoded to `0` on every path; they
+  now report what the turn actually spent, summed by the engine across every LLM call the turn
+  made (including calls made by delegated sub-agents). Three new `metadata` fields carry the rest
+  of the picture: `cacheReadTokens` and `cacheWriteTokens` (both `int`), and `costUsd` (a decimal
+  STRING, e.g. `"0.00842"`, or `null` when no provider reported a cost -- never a float, since a
+  float cannot hold a decimal money value exactly and a host summing per-turn costs from floats
+  accumulates drift it cannot see). `tokensIn` is the CHARGED input total: the provider's gross
+  input plus `cacheWriteTokens`. Cache reads are already counted inside the gross input per
+  amplifier-core's `PROVIDER_CONTRACT.md`, so `cacheReadTokens` is a subset of `tokensIn` rather
+  than an addend; a host wanting the fresh-only figure derives it as
+  `tokensIn - cacheReadTokens - cacheWriteTokens`. Usage accounting sits upstream of the CLI's
+  display renderer, so the same numbers are reported under `--display text`, `--display ndjson`,
+  and `--quiet` alike.
+
+  The three envelope paths now differ deliberately where they previously didn't:
+  - the success envelope reports real spend, as above.
+  - the in-turn error envelope also reports real spend -- whatever the turn burned before it
+    failed, rather than the previous hardcoded `0`/`0`. A turn that burned tokens and then failed
+    no longer hides them from whoever is paying for them.
+  - the pre-boot (argv-validation) envelope is unchanged: still `0`/`0`, and now also omits all
+    three new fields entirely rather than reporting a zero/null placeholder. No turn ran on that
+    path, so there is nothing to report.
+
+  Both wrapper SDKs surface the same data on their terminal events. `ResultEvent` gains
+  `session_id`, `turn_id`, `exit_code`, `usage`, and `stderr_tail`; `ErrorEvent` gains `session_id`,
+  `turn_id`, `exit_code`, and `usage`. A new `Usage` type carries the five fields above under
+  wrapper-native naming (`input_tokens` / `inputTokens`, etc.), read straight off the envelope with
+  no wrapper-side re-summing. `cost_usd` is the one field where the two wrappers deliberately
+  differ in type: Python parses it into a `Decimal`, TypeScript keeps the exact decimal string,
+  because JavaScript's `number` is an IEEE-754 binary double and parsing a monetary string into one
+  would reintroduce the precision loss the wire format exists to avoid. Synthesized errors that
+  never see an envelope (`spawn_failed`, `engine_hung`, `envelope_missing`, `engine_exit_<N>`) now
+  carry the `session_id` the wrapper already knows from the caller; `turn_id` stays absent on all
+  four, since the engine assigns turn ids and none of these paths involves an engine that returned
+  one. An envelope's own identity fields remain authoritative whenever one exists.
+
+  Both wrappers also gain a `stderr_tail_bytes` option (`stderrTailBytes` in TypeScript) on the
+  session handle: a positive int caps the terminal event's `stderr_tail` to that many UTF-8 bytes
+  (never splitting a codepoint), `None`/`null` returns the entire buffer, and `0` disables capture
+  entirely. It applies uniformly to both `ResultEvent` and `ErrorEvent`, and now also bounds a
+  `stderrTail` the engine itself supplied inside an error envelope, which previously passed through
+  uncapped. The default of 4096 bytes preserves the historical behavior exactly.
+
+  `PROTOCOL_VERSION` moves `0.3.0` -> `0.4.0`, purely additive: no existing field changed shape or
+  meaning, but the minor version moves so a host can gate on the new fields being present. Both
+  wrappers are pinned to the new value; per the usual rule, a wrapper pinned to `0.3.0` and an
+  engine on `0.4.0` fail the handshake loudly (`protocol_version_mismatch`) rather than silently
+  misbehaving, so **upgrading the engine and both wrapper packages together is required.**
+
+### Fixed
+
+- **The wrapper's `stderr_tail` cap now measures real UTF-8 bytes, not characters.** It previously
+  sliced a plain string, so on non-ASCII stderr the cap was wrong by roughly the size of the
+  encoding (a Japanese reply could overshoot a byte budget several times over). The cap is now
+  byte-accurate and never splits a codepoint, which means it may return a handful of bytes fewer
+  than the requested cap when trimming lands mid-character. The exported `STDERR_TAIL_BYTES`
+  constant keeps its name and its value (4096); it simply counts the right unit now.
+
 - **`amplifier-agent run --prompt-file <path>`,** a second transport for the prompt. The
   positional `PROMPT` argument remains valid and unchanged; the two are mutually exclusive.
   File contents are decoded as UTF-8 and delivered verbatim, with no stripping and no newline

docs/INTEGRATION.md

@@ -116,7 +116,7 @@ Model ids on this surface are namespaced per provider, because a single model li
 
 ## Wire protocol
 
-Protocol version **`0.3.0`**, defined in `src/amplifier_agent_lib/protocol/methods.py`. Breaking changes bump it. Wrappers must pass `--protocol-version 0.3.0`; a mismatch returns `protocol_version_mismatch` and exits non-zero rather than silently misbehaving.
+Protocol version **`0.4.0`**, defined in `src/amplifier_agent_lib/protocol/methods.py`. Breaking changes bump it. Wrappers must pass `--protocol-version 0.4.0`; a mismatch returns `protocol_version_mismatch` and exits non-zero rather than silently misbehaving.
 
 The wrapper passes flags as argv. The engine writes one JSON envelope line to stdout on completion.
 
@@ -141,15 +141,16 @@ The wrapper passes flags as argv. The engine writes one JSON envelope line to st
 
 ```json
 {
-  "protocolVersion": "0.3.0",
+  "protocolVersion": "0.4.0",
   "sessionId": "...",
   "turnId": "turn-1",
   "reply": "...",
   "error": null,
   "metadata": {
-    "tokensIn": 0, "tokensOut": 0, "durationMs": 0,
+    "tokensIn": 0, "tokensOut": 0, "cacheReadTokens": 0, "cacheWriteTokens": 0,
+    "costUsd": null, "durationMs": 0,
     "bundleDigest": "...", "engineVersion": "...",
-    "protocolVersion": "0.3.0", "correlationId": "...",
+    "protocolVersion": "0.4.0", "correlationId": "...",
     "activeMode": null
   }
 }

docs/architecture/data-flows.md

@@ -45,7 +45,7 @@ __main__.main -> single_turn.run -> _execute_turn -> Engine -> _runtime.handler
    (`amplifier_agent_lib/protocol_points/defaults_cli.py:46` and `:152`).
 
 7. **Protocol version.** `:771` compares `--protocol-version` against the compiled
-   `PROTOCOL_VERSION` (`protocol/methods.py:11`, currently `0.3.0`). Strict equality
+   `PROTOCOL_VERSION` (`protocol/methods.py:11`, currently `0.4.0`). Strict equality
    unless `host_config.allowProtocolSkew` is set.
 
 8. **Workspace.** `:805` `resolve_workspace(argv, env, cwd)` from
@@ -276,10 +276,12 @@ keep the two honest.
    consumers always see a terminal event.
 
 8. **Failure synthesis.** With no parseable envelope, the wrapper builds an error event
-   from the exit code and the last 4096 bytes of stderr
-   (`run-output-parser.ts:23` `STDERR_TAIL_BYTES`). The classification-to-exit-code
-   table is in `docs/spec/envelope-and-errors.md`; how the wrapper consumes it is in
-   `docs/spec/wrapper-contract.md`.
+   from the exit code and a stderr tail capped at `stderrTailBytes` real UTF-8 bytes,
+   never split mid-codepoint (`run-output-parser.ts:93` `tailStderrBytes`), defaulting
+   to `STDERR_TAIL_BYTES` (4096) when the session handle does not override it. The
+   classification-to-exit-code table is in `docs/spec/envelope-and-errors.md`; how the
+   wrapper consumes it, including the `usage` and identity fields now carried on the
+   terminal event, is in `docs/spec/wrapper-contract.md`.
 
 ## Stream discipline
 

docs/spec/envelope-and-errors.md

@@ -22,18 +22,21 @@ Under `--output text` stdout is left intact so a human sees the reply as it is p
 
 ```json
 {
-  "protocolVersion": "0.3.0",
+  "protocolVersion": "0.4.0",
   "sessionId": "sess-abc-001",
   "turnId": "turn-1",
   "reply": "It is 2:15pm Pacific time.",
   "error": null,
   "metadata": {
     "tokensIn": 1247,
     "tokensOut": 89,
+    "cacheReadTokens": 1024,
+    "cacheWriteTokens": 0,
+    "costUsd": "0.00842",
     "durationMs": 1832,
     "bundleDigest": "",
     "engineVersion": "0.12.0",
-    "protocolVersion": "0.3.0",
+    "protocolVersion": "0.4.0",
     "correlationId": "3f2a1b9c-4d5e-4f60-9a7b-1c2d3e4f5061",
     "activeMode": null
   }
@@ -43,6 +46,21 @@ Under `--output text` stdout is left intact so a human sees the reply as it is p
 - `sessionId` echoes `--session-id` when supplied, else the session id the engine assigned.
 - `turnId` is the engine's turn id, defaulting to `"turn-1"`. One turn is submitted per process,
   so in practice it is always `turn-1`.
+- `tokensIn` is the CHARGED input total for the turn: the provider's gross input plus
+  `cacheWriteTokens`. Per amplifier-core's `PROVIDER_CONTRACT.md`, a provider's `input_tokens` is
+  already the gross total (fresh tokens and cache reads combined), so `cacheReadTokens` is a
+  **reported subset of `tokensIn`, not an addend** -- adding it in would double-count it and
+  roughly double the figure on a cache-heavy turn. `cacheWriteTokens` is the one bucket billed on
+  top of the gross total, so it is added. A caller that wants the fresh-only figure derives it as
+  `tokensIn - cacheReadTokens - cacheWriteTokens`.
+- `tokensOut`, `cacheReadTokens`, `cacheWriteTokens` are turn-scoped sums across every LLM call the
+  turn made, including calls made by delegated sub-agents. They are independent of `--display` and
+  of verbosity (`--quiet`, `-v`, `--debug`): the same numbers are reported no matter which renderer,
+  if any, was attached.
+- `costUsd` is a decimal STRING (e.g. `"0.00842"`), never a JSON number, or `null` when no provider
+  reported a cost for the turn. A float cannot represent a decimal money value exactly, so a caller
+  summing per-turn costs from JSON floats accumulates drift it cannot see; `null` is not the same
+  claim as `"0"` -- it means nobody reported a cost, not that the cost was zero.
 - `durationMs` is wall-clock time measured around the turn by the CLI, not by the engine.
 - `activeMode` echoes `--mode` verbatim, or `null`. The mode is non-sticky: omitting `--mode` on a
   resume returns the field to `null`.
@@ -58,7 +76,7 @@ Emitted for failures raised once the turn is running.
 
 ```json
 {
-  "protocolVersion": "0.3.0",
+  "protocolVersion": "0.4.0",
   "sessionId": "sess-abc-001",
   "turnId": "turn-1",
   "reply": "",
@@ -70,12 +88,15 @@ Emitted for failures raised once the turn is running.
     "message": "unknown approval action 'review'"
   },
   "metadata": {
-    "tokensIn": 0,
-    "tokensOut": 0,
+    "tokensIn": 342,
+    "tokensOut": 18,
+    "cacheReadTokens": 0,
+    "cacheWriteTokens": 0,
+    "costUsd": "0.00051",
     "durationMs": 247,
     "bundleDigest": "",
     "engineVersion": "0.12.0",
-    "protocolVersion": "0.3.0",
+    "protocolVersion": "0.4.0",
     "correlationId": "3f2a1b9c-4d5e-4f60-9a7b-1c2d3e4f5061",
     "activeMode": null
   }
@@ -85,7 +106,10 @@ Emitted for failures raised once the turn is running.
 Error-path invariants:
 
 - `reply` is always `""`.
-- `tokensIn` / `tokensOut` are always `0`.
+- `tokensIn`, `tokensOut`, `cacheReadTokens`, `cacheWriteTokens`, and `costUsd` report whatever the
+  turn actually spent before it failed, using the same accounting as the success envelope. A turn
+  that never reached a provider legitimately reports zero tokens and a `null` cost; a turn that
+  burned tokens and then failed reports them rather than hiding them behind a placeholder `0`.
 - `severity` is always `"error"`. The value `"warning"` exists in the wrapper types; the engine
   never emits it.
 - `activeMode` is always `null`.
@@ -97,7 +121,7 @@ Error-path invariants:
 
 ### Pre-boot (argv-validation) envelope
 
-Failures detected before the turn starts emit the same envelope shape with three differences:
+Failures detected before the turn starts emit the same envelope shape with four differences:
 
 ```
 sessionId, turnId    always ""
@@ -106,6 +130,11 @@ classification       "protocol" for every argv and config failure; "engine" for
                      the caller their mode name is wrong would be a lie
 error.remediation    included when the failure has one
 metadata             omits activeMode entirely, rather than setting it to null
+metadata             tokensIn and tokensOut are still 0, but cacheReadTokens, cacheWriteTokens,
+                     and costUsd are omitted entirely rather than reporting zero/null. No turn
+                     ran on this path, so there is nothing to report for the three new fields --
+                     omission here is a stronger claim than the in-turn error envelope's zero,
+                     which means a turn ran and spent nothing.
 ```
 
 This is the path for config errors, workspace-slug rejection, protocol skew, headless approval,
@@ -245,7 +274,7 @@ One file per turn:
 {
   "argvDigest": "sha256:<hex of the joined argv>",
   "envDigest": "sha256:<hex of a constant placeholder>",
-  "protocolVersion": "0.3.0",
+  "protocolVersion": "0.4.0",
   "exitCode": 0,
   "correlationId": "3f2a1b9c-4d5e-4f60-9a7b-1c2d3e4f5061",
   "startedAt": "2026-07-31T15:00:00.000000+00:00",

docs/spec/install-and-distribution.md

@@ -203,7 +203,7 @@ when a failure must not break the surrounding install.
 ## `version --json`
 
 ```json
-{"version": "0.12.0", "protocolVersion": "0.3.0"}
+{"version": "0.12.0", "protocolVersion": "0.4.0"}
 ```
 
 Exactly two keys. This is the wrapper pre-spawn probe: both SDKs run `<binPath> version --json`
@@ -246,7 +246,7 @@ Four artifacts version independently:
 amplifier-agent          0.12.0    engine, the release truth
 amplifier-agent-ts       0.7.0     TypeScript wrapper SDK
 amplifier-agent-py       0.3.0     Python wrapper SDK
-protocol version         0.3.0     declared by the engine and pinned by each wrapper
+protocol version         0.4.0     declared by the engine and pinned by each wrapper
 ```
 
 Only the protocol version couples them. The engine reports its own version from installed package
@@ -255,7 +255,7 @@ metadata.
 ### The compatibility rule
 
 **Strict string equality on the protocol version.** There is NO support window, NO N-1 policy, and
-NO compatibility matrix. `0.3.0` and `0.3.1` are as incompatible as `0.3.0` and `9.0.0`.
+NO compatibility matrix. `0.4.0` and `0.4.1` are as incompatible as `0.4.0` and `9.0.0`.
 
 Three independent enforcement points:
 

docs/spec/wire-protocol.md

@@ -36,11 +36,11 @@ side channel rather than to the frame dispatcher.
 ## Protocol version
 
 ```
-0.3.0
+0.4.0
 ```
 
 Compared by strict string equality. Semver range matching is not used, and no compatibility window
-exists: `0.3.1` against `0.3.0` is a mismatch.
+exists: `0.4.1` against `0.4.0` is a mismatch.
 
 A mismatch is detected at up to three points, in the order a turn reaches them: