diff --git a/docs/ADRs/0050-distributed-tracing-instrumentation.md b/docs/ADRs/0050-distributed-tracing-instrumentation.md index c004e115b1..685b7afc9e 100644 --- a/docs/ADRs/0050-distributed-tracing-instrumentation.md +++ b/docs/ADRs/0050-distributed-tracing-instrumentation.md @@ -169,3 +169,9 @@ beside telemetry when at least one new score is produced (tool-agnostic). Distin spans only; the root span keeps `fullsend.cost_usd` and `fullsend.tool_calls` (custom-namespaced, not auto-summed by MLflow). This prevents MLflow from double-counting token usage across the trace. + +**2026-09-03 — Tool-call span topology ([ADR 0102](0102-tool-call-span-topology.md)):** +each tool call the runtime reports becomes an `execute_tool` child of its +iteration's `agent` span, metadata only; the message record on the `agent` +span stays the content carrier. Sub-agent nesting (deferred item 1 above) +remains deferred. diff --git a/docs/ADRs/0102-tool-call-span-topology.md b/docs/ADRs/0102-tool-call-span-topology.md new file mode 100644 index 0000000000..9ddc634a6b --- /dev/null +++ b/docs/ADRs/0102-tool-call-span-topology.md @@ -0,0 +1,94 @@ +--- +title: "102. Tool-call span topology" +status: Accepted +relates_to: + - operational-observability +topics: + - observability + - telemetry + - opentelemetry +--- + +# 102. Tool-call span topology + +Date: 2026-09-03 + +## Status + +Accepted + +## Context + +[ADR 0050](0050-distributed-tracing-instrumentation.md) chose OpenTelemetry +and a three-level opt-in but never named the spans: the +`run → sandbox_create → agent` tree lives only in the guides, and granularity +was left to [#294](https://github.com/fullsend-ai/fullsend/issues/294). +Level 3 content ([#6429](https://github.com/fullsend-ai/fullsend/pull/6429), +[#6603](https://github.com/fullsend-ai/fullsend/pull/6603)) put tool calls +and results on the `agent` span as `gen_ai.output.messages` parts — at +semconv v1.37.0 the `execute_tool` span is metadata-only (its content +attributes exist only in the newer GenAI repository, opt-in). Review of +#6603 asked why tool calls are not spans. + +fullsend observes the runtime's stream rather than executing tools. The +normalized `ToolUseEvent`/`ToolResultEvent` pairs carry a call id, a tool +name and an `is_error` flag; `tool_use` lines carry no timestamp and +`tool_result` lines carry a sandbox-clock one; several calls are open at +once (parallel sub-agent dispatch); `parent_tool_use_id` is dropped at +decode; the pi and codex parsers pass no call ids through. + +## Options + +1. **Message record only** (status quo): tool calls stay parts of + `gen_ai.output.messages`; no per-call timing or status in the span tree. +2. **`execute_tool` child spans from the normalized events**: one span per + call under the iteration's `agent` span, metadata only; content stays on + the message record. +3. **The runtime's native OpenTelemetry**: Claude Code emits + `claude_code.tool` spans (beta) and honours inbound `TRACEPARENT` — true + timing, but redaction would leave fullsend's pipeline, the threat model + keeps runtime telemetry out of scope, and the sandbox would need egress. +4. **Per-tool content on the spans**: needs the newer conventions' opt-in + attributes, and the scorers read the message record today. + +## Decision + +Option 2. `toolSpanTracker` (`internal/cli/tool_spans.go`) opens an +`execute_tool ` span (kind Internal) when the runtime reports a +call and ends it when the result arrives. Both timestamps are runner-side +receipt instants — one clock — so the span brackets execution rather than +measuring it, and the start is arguments-complete. Attributes follow semconv +v1.37.0: `gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`, +`gen_ai.tool.call.id`; a result flagged `is_error` sets +`error.type=tool_error` and status Error. Calls that never get a result +close as `error.type=unanswered`, results for calls never reported are +marked `fullsend.tool.unmatched`, and events without a call id (pi, codex, +server-side tools) get no span — the edge cases are specified in the +[dev guide](../guides/dev/tracing.md#execute_tool-spans). Names pass +through the same sanitizer as span content and are bounded; at most 1,024 +spans are recorded per iteration, so an agent-controlled burst cannot fill +the OTLP batch queue and evict the `agent` span, with the overflow counted +in `fullsend.tool_spans.dropped`. The spans are Level 1 metadata, emitted +regardless of the content gate. Tool content — results now, full arguments +next — stays on the `agent` span's `gen_ai.output.messages` record, which +is the scorer contract +([ADR 0087](0087-eval-measurements-online-trace-scoring.md)). + +Option 3 is the candidate to revisit once the runtime's tracing is stable and +a redaction stage outside fullsend is designed; option 4 waits for the +convention bump. + +## Consequences + +- The span tree gains one level; readers that select spans by name + (`evalmeasure`) are unaffected, and the `execute_tool` count can be below + `fullsend.tool_calls`, which counts every reported call, id or not. +- Tool-heavy iterations add hundreds of spans, each one synchronous write to + `run-telemetry.jsonl` and one OTLP batch entry. +- Sub-agent calls are flat children of the `agent` span; nesting them under + the dispatching `Agent` call is a small follow-up now that the parent span + exists while its children run (ADR 0050's deferred item 1 stays deferred). +- `execute_tool` spans are Claude-only until the pi and codex parsers pass + their streams' call ids through. +- This settles the span-granularity question in #294; retention and access + remain open there. diff --git a/docs/architecture.md b/docs/architecture.md index c7128f52dc..7ac5fb6d19 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -368,6 +368,8 @@ Observability is a cross-cutting concern that touches every other component. Eac > **Planned:** portable remote score export via the same OTLP configuration as agent traces ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). Not yet implemented. +- Tool-call span topology: every tool call the runtime reports becomes an `execute_tool` child span of its iteration's `agent` span — semantic-convention metadata only, timed at runner receipt; tool content stays on the `agent` span's message record ([ADR 0102](ADRs/0102-tool-call-span-topology.md)). + **Open questions:** - What signals matter most — cost, latency, token usage, action logs, decision traces, or something else? diff --git a/docs/contributing/runtime-implementation.md b/docs/contributing/runtime-implementation.md index fc6897746e..6a486da196 100644 --- a/docs/contributing/runtime-implementation.md +++ b/docs/contributing/runtime-implementation.md @@ -187,7 +187,7 @@ JSON on stdin; the script's exit code and stdout are the reply. | | Input | Reply | |---|---|---| | **PreToolUse** | `{"tool_name": ..., "tool_input": {...}}` | exit `0` = allow. Blocking scripts exit `1` and print `{"decision":"block","reason":"..."}`; the adapter must stop the tool call and surface the reason | -| **PostToolUse** | the same plus the tool output as `tool_response` (Claude Code; string or structured object such as Bash `{stdout, stderr, interrupted, isImage}`), `tool_result` accepted as a fallback | *Blocking* (standalone `canary_posttool.py`, and `posttool_chain.py` when its canary stage fires): exit `1` + `{"decision":"block",...}`; the adapter drops the result. *Sanitizing* stages (suppress/unicode/redact): always exit `0` and, when they changed something, print `{"hookSpecificOutput":{"hookEventName":"PostToolUse","updatedToolOutput": }, "tool_result": }`. Empty stdout = unchanged | +| **PostToolUse** | the same plus the tool output as `tool_response` (Claude Code; string or structured object such as Bash `{stdout, stderr, interrupted, isImage}`), `tool_result` accepted as a fallback; Claude Code also sends `cwd` (its working directory, which follows the agent's persisted `cd`) — the redact stage uses it to locate the checkout, and an adapter that omits it gets no bare-JWT skip (see *Sanitizer scope*) | *Blocking* (standalone `canary_posttool.py`, and `posttool_chain.py` when its canary stage fires): exit `1` + `{"decision":"block",...}`; the adapter drops the result. *Sanitizing* stages (suppress/unicode/redact): always exit `0` and, when they changed something, print `{"hookSpecificOutput":{"hookEventName":"PostToolUse","updatedToolOutput": }, "tool_result": }`. Empty stdout = unchanged | Shape rules: @@ -207,6 +207,8 @@ The PostToolUse stages exist to remove *controls-relevant* content and nothing e A sweep of 900 fullsend files through the chain rewrites only test files holding token-shaped fakes. +The bare-JWT prefix pattern is skipped when a file-content tool (`Read`, `Grep`, `Edit`, `MultiEdit`, `Write`, `NotebookEdit`, `NotebookRead`) is called with a path inside the checkout — the nearest `.git` ancestor of the hook input's `cwd`, none meaning no skip; the path is normalized, then resolved; `..` segments and `~` paths never skip — because a jwt.io-style fixture is byte-for-byte a valid token and masking it corrupts what the agent edits against, while the runner's own OIDC token file sits beside the checkout and still masks. Bash, WebFetch and MCP output are unaffected. + **Context suppression** condenses the output of exactly one verification command, and only from positive evidence: - Commands: `go test`, `pytest`, `npm test`, `make test`, `pre-commit run`, `gitleaks detect`, `scan-secrets`, with optional setup prefixes (`cd`, `export`, `source`). The command must *start* with the tool, after wrappers that run it (`VAR=...`, `sudo`, `nice`, `timeout `, `env VAR=...`, `uvx`, `npx`, `uv run`, `mise exec --`, stacked; `python3.12 -m pytest` counts). A command that merely mentions it, such as `grep -n scan-secrets hooks.py`, keeps its output. @@ -525,7 +527,7 @@ The Claude-style agent `.md` is parsed by `Bootstrap`: `Bootstrap` installs `security.HookFiles` under `/sandbox/pi-config/hooks/`, writes the `HookPlan` into `fullsend-manifest.json` and loads the embedded `fullsend-hooks.js` extension with `-e` under `--no-extensions` (per pi v0.84.2 `docs/extensions.md`). -`fullsend-hooks.js` sends the scripts `{tool_name, tool_input, tool_result, tool_response}` with Claude tool names (`bash→Bash`, `read→Read`, `write→Write`, `edit→Edit`, `grep→Grep`, `find→Glob`, `ls→LS`; `path` mirrored to `file_path`) and reads back either the v1 `tool_result` or the v2 `hookSpecificOutput.updatedToolOutput` (#6357), so the same extension works before and after the PostToolUse chain lands. +`fullsend-hooks.js` sends the scripts `{tool_name, tool_input, tool_result, tool_response}` (no `cwd`, so the redact stage's checkout-scoped bare-JWT skip is inert under pi — it masks as before) with Claude tool names (`bash→Bash`, `read→Read`, `write→Write`, `edit→Edit`, `grep→Grep`, `find→Glob`, `ls→LS`; `path` mirrored to `file_path`) and reads back either the v1 `tool_result` or the v2 `hookSpecificOutput.updatedToolOutput` (#6357), so the same extension works before and after the PostToolUse chain lands. - PreToolUse groups run in `HookPlan` order and stop at the first block; a script that cannot be spawned blocks; PostToolUse blocks withhold the result and mark it `isError`. - An unreadable manifest, or one without a hook plan, blocks every tool call. diff --git a/docs/guides/dev/tracing.md b/docs/guides/dev/tracing.md index 0a28228826..1cadc2a6c5 100644 --- a/docs/guides/dev/tracing.md +++ b/docs/guides/dev/tracing.md @@ -91,12 +91,13 @@ silently. ## Span lifecycle in run.go -`run.go` creates three span types arranged in a parent-child hierarchy: +`run.go` creates four span types arranged in a parent-child hierarchy: ``` run (root) ├── sandbox_create (gen_ai.operation.name=create_agent) └── agent (one per iteration; gen_ai.operation.name=invoke_agent) + └── execute_tool (one per tool call; gen_ai.operation.name=execute_tool) ``` ### Root span @@ -132,6 +133,35 @@ build the attribute slices. Start attributes: `iteration`, `exit_code`, `gen_ai.system`, model, token counts, `fullsend.cost_usd`, `fullsend.tool_calls`. +### execute_tool spans + +One per tool call the runtime reports, a child of that iteration's agent +span, named `execute_tool `. `toolSpanTracker` +(`internal/cli/tool_spans.go`) starts the span when the `ToolUseEvent` +arrives and ends it when the matching `ToolResultEvent` arrives, so both +timestamps are runner-side receipt instants on one clock — the start is +arguments-complete, not execution start. Attributes: +`gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`, +`gen_ai.tool.call.id`; a result flagged `is_error` sets +`error.type=tool_error` and status Error. A call still open when the +iteration ends — the runtime was stopped, or its result line exceeded the +parser's 1 MiB cap — is closed by `Finish()` as `error.type=unanswered`; a +result with no matching call (its `tool_use` line was skipped) becomes a +near-zero-duration span marked `fullsend.tool.unmatched=true`. Events without +an id — pi and codex emit none, and the parser gives server-side tools +(`server_tool_use`) none because their result never arrives as a +`tool_result` — produce no span, so the child count can be below +`fullsend.tool_calls`. The name passes through `security.OutputPipeline()` +— Unicode normalization, then secret redaction, the same pipeline as span +content — and is bounded to 256 bytes before it becomes the attribute; the +span name keeps at most 128 bytes of it. +The tracker records at most `maxToolSpansPerIteration` (1,024) spans per +iteration and reports the overflow, which `runAgent` records as +`fullsend.tool_spans.dropped` on the agent span — a burst of agent-controlled +calls must not fill the OTLP batch queue and evict the agent span that ends +right after `Finish()`. These spans are metadata: they carry no tool content +and are emitted whether or not the Level 3 gate is on. + ### Level 3 content on agent spans When the content-capture gate is on @@ -143,18 +173,26 @@ spans — and tees the runtime's normalized event stream to it through **The tee trap:** supplying any `OnEvent` replaces the runtime's default console renderer (`internal/runtime/claude.go`), so the handler built by -`contentEventHandler` always calls the renderer first and the collector -second. With the gate off the collector is nil and `contentEventHandler` -returns nil, leaving the default renderer path byte-identical to before -Level 3 existed. +`iterationEventHandler` always calls the renderer first, then the +collector, then the tool-span tracker. The handler is always set — tool +spans are emitted with the gate off — and with the gate off the collector +is nil and inert, so console output stays byte-identical to the default +renderer path. The collector (`internal/cli/content_collector.go`) coalesces contiguous -text/reasoning deltas, maps tool use to `tool_call` parts, redacts every +text/reasoning deltas, maps tool use to `tool_call` parts and tool +results to `tool_call_response` parts (correlated by `id` when the +runtime's stream provides one; the schema's required result field is +`response`), redacts every part through `security.OutputPipeline()` at assembly (redaction runs before the size budget — truncating first could split a secret past recognition), enforces a 256 KiB ordered-suffix budget (the ending survives — the -final answer is what consumers judge) with exact dropped-byte accounting -across content, tool names, and summaries, and emits +final answer is what consumers judge) plus an 8 KiB per-tool-result +bound (tail-kept, redacted before the cut, the part marked +`fullsend.truncated`), with exact dropped-byte +accounting across content, tool names, summaries, responses, and part +ids (a third, earlier boundary — the parser's 1 MiB stream-line cap — +drops oversized lines before any event exists), and emits `gen_ai.output.messages` JSON following the GenAI output-messages schema, including the schema-required `finish_reason` from the iteration outcome. `attachContent` records the content and its marker attributes on the span before either diff --git a/docs/guides/infrastructure/distributed-tracing.md b/docs/guides/infrastructure/distributed-tracing.md index dbb19625c4..000246075e 100644 --- a/docs/guides/infrastructure/distributed-tracing.md +++ b/docs/guides/infrastructure/distributed-tracing.md @@ -12,9 +12,10 @@ For implementation details, see the |-------|-----------------|----------------------| | 1 | `run-telemetry.jsonl` file in the run output directory | None | | 2 | OTLP/HTTP export to a remote backend (metadata only) | `OTEL_EXPORTER_OTLP_*ENDPOINT` | -| 3 | Conversation content (assistant text, reasoning, tool calls) on `agent` spans | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` | +| 3 | Conversation content (assistant text, reasoning, tool calls and results) on `agent` spans | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` | -All levels produce metadata (timing, token counts, tool names, errors). +All levels produce metadata (timing, token counts, tool names, errors), +including one `execute_tool` span per tool call under each `agent` span. Level 3 adds the agent's conversation content to spans — enabled by one environment variable, exactly like Level 2's endpoint. @@ -84,26 +85,31 @@ The variable name and accepted values follow the Fullsend records content on span attributes only, so `event_only` stays off. An unrecognized value disables capture; telemetry never fails a run. -**Captured:** assistant text, reasoning, and tool calls (name plus short -summary) — including any sub-agent activity, unattributed — as the -`gen_ai.output.messages` span attribute: a JSON string following the +**Captured:** assistant text, reasoning, tool calls (name plus short +summary), and tool results — including any sub-agent activity, +unattributed — as the `gen_ai.output.messages` span attribute: a JSON +string following the [GenAI output-messages schema](https://github.com/open-telemetry/semantic-conventions/blob/v1.37.0/docs/gen-ai/gen-ai-output-messages.json) -with a `finish_reason` of `stop` or `error`. +with a `finish_reason` of `stop` or `error`. Tool calls and their +results share a correlating `id` when the runtime's stream provides one +(Claude runs do). **Not captured:** model input (`gen_ai.input.messages`) and pre/post-script content. First-iteration runs have no meaningful runner-side input; retry iterations carry the injected validation feedback, a natural input-capture follow-up. -> **Planned:** Tool results, once a parser extension adds them to the -> normalized event stream — the next change after -> [#6429](https://github.com/fullsend-ai/fullsend/pull/6429). - **Redaction and size:** every part passes through security redaction (Unicode normalization, then secret masking) before reaching the span. -Content is bounded at 256 KiB per iteration, kept as an ordered suffix; -overflow drops the oldest content first. Truncation is marked via -`fullsend.content.truncated`. The SDK's span attribute length cap is +Content is bounded at 256 KiB per iteration — each tool result at 8 KiB — +kept as an ordered suffix; overflow drops the oldest content first. +Truncation is marked via `fullsend.content.truncated` on the span and +`fullsend.truncated` on each cut part. Two cases are absent rather than +truncated: stream lines beyond 1 MiB are skipped whole by the parser, +and results whose content is entirely non-text (for example images) +produce no part. A result that mixed text with non-text blocks keeps +its text and is marked `fullsend.truncated`; a failed call with empty +output survives as a `tool_call_response` part carrying `is_error`. The SDK's span attribute length cap is lifted while capture is on; an explicit `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` still wins and will cut content mid-JSON — fullsend warns on stderr at startup. @@ -124,8 +130,25 @@ of the same trace with identical span IDs. run (root; Consumer when dispatched with TRACEPARENT, else Internal) ├── sandbox_create (gen_ai.operation.name=create_agent) └── agent (one per iteration; gen_ai.operation.name=invoke_agent) + └── execute_tool (one per tool call; gen_ai.operation.name=execute_tool) ``` +`execute_tool` spans are named `execute_tool `. One starts when +the runtime reports a tool call (its arguments complete) and ends when it +reports the result — both are runner-side receipt times, so the span +brackets execution rather than measuring it exactly. A call with no result +by the end of the iteration is closed with `error.type=unanswered`; a +result whose call was never reported (its stream line was skipped) is a +near-zero-duration span marked `fullsend.tool.unmatched`. Runtimes whose parsers +emit no call ids (pi, codex) produce no `execute_tool` spans, and neither +do server-side tools, whose result never arrives as a `tool_result`. Tool +names pass through the same sanitizer as span content (Unicode +normalization, then secret redaction) and are bounded to 256 bytes for the +attribute and 128 for the span name. At most 1,024 `execute_tool` +spans are recorded per iteration; calls past that are counted in +`fullsend.tool_spans.dropped` on the `agent` span. Tool content never rides +these spans — see Content capture. + ### SpanKind | Span | Kind | Condition | @@ -134,6 +157,7 @@ run (root; Consumer when dispatched with TRACEPARENT, else Internal) | `run` | Internal | No inbound `TRACEPARENT` (local/manual invocation) | | `sandbox_create` | Internal | Always | | `agent` | Internal | Always | +| `execute_tool` | Internal | Always | ## Span attributes @@ -144,8 +168,10 @@ and are recognized by LLM-aware backends for GenAI dashboards. | Attribute | Example | Present on | |-----------|---------|------------| -| `gen_ai.operation.name` | `invoke_agent` | `run`, `agent` (`create_agent` on `sandbox_create`) | +| `gen_ai.operation.name` | `invoke_agent` | `run`, `agent` (`create_agent` on `sandbox_create`; `execute_tool` on `execute_tool`) | | `gen_ai.agent.name` | `triage` | `run`, `agent` | +| `gen_ai.tool.name` | `Bash` | `execute_tool` (the runtime's tool name; absent when the call was never reported) | +| `gen_ai.tool.call.id` | `toolu_01…` | `execute_tool` | | `gen_ai.system` / `gen_ai.provider.name` | `anthropic` | `agent` (model vendor; `system` deprecated in OTel GenAI semconv v1.37 — EM-001 accepts either) | | `gen_ai.request.model` | `claude-opus-4-6` | `agent` (resolved model) | | `gen_ai.usage.input_tokens` / `output_tokens` / `cache_*_input_tokens` | `109938` | `agent` | @@ -169,8 +195,10 @@ and are recognized by LLM-aware backends for GenAI dashboards. | `fullsend.transcript_error` | `agent` | Present (`true`) when the agent exited 0 but its transcript reported an error — the span's status is Error while `exit_code` keeps the raw process exit | | `gen_ai.output.messages` | `agent` | Level 3 only: the iteration's conversation content as a JSON string (see Content capture) | | `fullsend.content.truncated` | `agent` | Level 3 only: present (`true`) when the size budget cut or dropped content | -| `fullsend.content.dropped_bytes` | `agent` | Level 3 only: exact content bytes removed by the size budget | +| `fullsend.content.dropped_bytes` | `agent` | Level 3 only: exact part bytes (content and ids) removed by the size budget | | `fullsend.content.redactions` | `agent` | Level 3 only: number of security findings raised while redacting content at assembly (including findings from parts the size budget later dropped) | +| `fullsend.tool.unmatched` | `execute_tool` | Present (`true`) when a result arrived for a call the stream never reported; the span has near-zero duration | +| `fullsend.tool_spans.dropped` | `agent` | Present when the iteration reported more than 1,024 tool calls: the number of calls past the cap that got no `execute_tool` span | ### Common attributes @@ -178,6 +206,7 @@ and are recognized by LLM-aware backends for GenAI dashboards. |-----------|------------|-------------| | `exit_code` | `run`, `agent` | Process exit code | | `iteration` | `agent` | 1-based iteration index | +| `error.type` | `execute_tool` | `tool_error` when the runtime flagged the result `is_error`; `unanswered` when the call had no result by the end of the iteration (the runtime was stopped, or the result line exceeded the 1 MiB stream cap); absent on success | ### Resource attributes @@ -280,7 +309,9 @@ Non-finite float values (NaN, Infinity) are encoded as proto3 JSON strings (`"NaN"`, `"Infinity"`, `"-Infinity"`). The file is written synchronously per span. Spans are flushed to disk as -they complete; the file is the forensic record for crashed runs. +they complete; the file is the forensic record for crashed runs. Every +`execute_tool` span is one such line, so a tool-heavy iteration (a hundred +or more calls) adds that many. ## Cross-run trace correlation diff --git a/docs/guides/user/how-to-emit-traces.md b/docs/guides/user/how-to-emit-traces.md index 0adf07f6ad..16d1b9d045 100644 --- a/docs/guides/user/how-to-emit-traces.md +++ b/docs/guides/user/how-to-emit-traces.md @@ -113,7 +113,9 @@ to a backend like MLflow, Jaeger, Grafana Tempo, etc. ## Capture conversation content Set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` to add the agent's -text, reasoning, and tool calls to each `agent` span, in the local file and +text, reasoning, tool calls, and tool results — when the runtime's stream +provides them; Claude runs do — to each `agent` span, in the +local file and at the endpoint. Content is redacted for secrets and bounded per iteration, but may still contain proprietary code or PII — make sure your backend's access controls fit before enabling it. @@ -122,6 +124,12 @@ access controls fit before enabling it. gh variable set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT --body "true" --repo ``` +With the `claude` runtime, tool calls are visible at every level as +`execute_tool` child spans of each `agent` span (tool name, call id, timing, +error) — that is metadata, not content, and this variable does not affect +it. The pi and codex runtimes emit none yet (see +[Runtimes](../../runtimes.md)). + ## Disable trace export Remove the endpoint variable and header secret from the repository or diff --git a/docs/problems/operational-observability.md b/docs/problems/operational-observability.md index 92a491ff6e..25f3e5bfe3 100644 --- a/docs/problems/operational-observability.md +++ b/docs/problems/operational-observability.md @@ -187,7 +187,7 @@ This works for early experimentation when the volume is low and the operators ar ## Open questions -- What is the right level of trace granularity? Input context is largely already stored in git and GitHub, so traces can reference it by pointer. But full prompt/completion pairs — the novel data — are expensive to store and may contain sensitive content. Is there a middle ground — e.g., capturing token counts and decision summaries by default, with full prompt/completion pairs available on demand? +- What is the right level of trace granularity? Input context is largely already stored in git and GitHub, so traces can reference it by pointer. But full prompt/completion pairs — the novel data — are expensive to store and may contain sensitive content. Is there a middle ground — e.g., capturing token counts and decision summaries by default, with full prompt/completion pairs available on demand? (Span granularity decided in [ADR 0102](../ADRs/0102-tool-call-span-topology.md): one `execute_tool` span per tool call, metadata only, with content opt-in at Level 3 per [ADR 0050](../ADRs/0050-distributed-tracing-instrumentation.md). Retention and access remain open.) - ~~How do we keep the triggering event and the agent run linked in the GitHub Actions UI for debugging?~~ Decided in [ADR 0041](../ADRs/0041-synchronous-workflow-call-event-dispatch.md) (synchronous `workflow_call` dispatch for the event path). - How should trace access be controlled? (JSONL trace exposure decided in [ADR 0021](../ADRs/0021-jsonl-reasoning-trace-exposure.md): owner-scoped storage with credential scanning as defense-in-depth. Broader question of balancing security and transparency for non-JSONL observability data remains open.) - What retention policy applies to traces? Indefinite retention supports audit requirements but increases storage cost and data sensitivity exposure. Time-bounded retention (e.g., 90 days) limits exposure but may lose traces needed for incident investigation. diff --git a/docs/runtimes.md b/docs/runtimes.md index d03c4781d4..b1e3b462f3 100644 --- a/docs/runtimes.md +++ b/docs/runtimes.md @@ -62,6 +62,8 @@ sequenceDiagram | Tools | Native Claude permission syntax | `--tools` (strict) + a first-token Bash allowlist | Shell + `apply_patch` only; `tools:` is recorded, not enforced (the allowlist hook is opt-in) | | Security controls | Full matrix | Full matrix; stricter on failed-call sanitizing | Full matrix; post-tool hooks detect and block but cannot rewrite output | | Cost in `metrics.json` | Reported | Reported | Not reported — codex sends none | +| Content capture (Level 3) | Text, reasoning, tool calls and tool results (correlating ids) | Text, reasoning, tool calls (no correlating ids) — pi's parser emits neither ids nor tool results yet | Text, reasoning, tool calls (no correlating ids) — codex's parser emits neither ids nor tool results | +| Tool spans (`execute_tool`) | One per tool call, a child of the iteration's `agent` span, timed at receipt | None — the parser emits no call ids | None — the parser emits no call ids | All three run unattended in the same sandbox, behind the same egress allowlist. Stay on `claude` when you need sub-agents or a fallback chain. Choose `pi` when you want a non-Anthropic model, or diff --git a/internal/cli/content_collector.go b/internal/cli/content_collector.go index b6adda6a1e..a9caf3edb9 100644 --- a/internal/cli/content_collector.go +++ b/internal/cli/content_collector.go @@ -14,13 +14,37 @@ import ( // maxContentBytes bounds the conversation content attached to one agent // span (one iteration), measured on the raw part bytes before JSON -// encoding. The value is far above what text+reasoning+tool-call -// summaries produce in practice (the full-transcript mean of ~369K chars -// includes tool results, which are not captured yet) and was accepted -// whole by the pilot backend in live validation. Revisit when tool -// results join the stream. +// encoding — the marshaled attribute additionally carries per-part JSON +// syntax and escaping, so a budget-binding iteration serializes larger +// than this value. A 255KB attribute was accepted whole by the pilot +// backend in live validation; beyond that is unproven, so the total +// stays put and tool results are bounded per part instead. const maxContentBytes = 256 * 1024 +// maxToolIDBytes bounds a tool call/result id. The stream decodes ids +// unbounded and Level 3 lifts the SDK attribute cap; real ids run tens +// of bytes, so anything beyond this is malformed and gets dropped — +// never truncated, since a truncated id could falsely collide with +// another call's. Id bytes that pass the bound still count toward the +// size budget like every other serialized part byte. +const maxToolIDBytes = 256 + +// maxToolResultBytes bounds one tool result's response within the +// suffix budget. Measured on three real review-agent MAIN-THREAD +// transcripts (2026-08-25): uncapped results total 222-389KB per +// iteration — overflowing maxContentBytes on two of three runs — while +// an 8KiB cap kept those runs at 127-255KB with 78-89% of results +// untouched (p50 2-3.5KB, p90 9-19KB). The live stream this collector +// consumes also interleaves sub-agent results, so those figures are a +// lower bound on production volume and the eviction-pressure reduction +// is a lower-bound claim. The cap lowers eviction pressure; it does +// not prevent it — a heavier iteration still overflows the total +// budget and evicts oldest-first, marked via the truncated and +// dropped-bytes attributes. A capped response keeps its tail, extending +// the budget's ordered-suffix policy to individual results; no consumer +// requirement has confirmed either direction yet. +const maxToolResultBytes = 8 * 1024 + // newContentCollectorIfEnabled returns a live collector when the Level 3 // gate is on and nil otherwise — nil is the off state and is inert at // every call site, so the gate needs no second check. @@ -31,17 +55,18 @@ func newContentCollectorIfEnabled() *contentCollector { return nil } -// contentEventHandler tees the normalized event stream to the console -// renderer and the collector. A nil collector returns a nil handler so -// the runtime keeps its default renderer path — supplying any OnEvent -// replaces that renderer, and losing it silences CI output. -func contentEventHandler(render func(agentruntime.AgentEvent), c *contentCollector) func(agentruntime.AgentEvent) { - if c == nil { - return nil - } +// iterationEventHandler tees the normalized event stream to the console +// renderer, the Level 3 collector and the tool-span tracker, in that +// order. It is always non-nil: tool spans are metadata and are emitted +// with the content gate off, and supplying any OnEvent replaces the +// runtime's default renderer — losing it silences CI output — so the +// renderer runs first whatever else is off. A nil collector (gate off) +// and a nil tracker are inert. +func iterationEventHandler(render func(agentruntime.AgentEvent), c *contentCollector, t *toolSpanTracker) func(agentruntime.AgentEvent) { return func(evt agentruntime.AgentEvent) { render(evt) c.Handle(evt) + t.Handle(evt) } } @@ -73,22 +98,104 @@ func attachContent(span trace.Span, res contentResult) { // contentPart is one part of the assembled assistant output message, // shaped for the GenAI output-messages JSON schema: TextPart // ({type:"text",content}), the schema's GenericPart extension point -// ({type:"reasoning",content}), and ToolCallRequestPart -// ({type:"tool_call",name}+summary). A tool summary is not the tool's -// arguments, so no arguments field is ever fabricated from it. +// ({type:"reasoning",content}), ToolCallRequestPart +// ({type:"tool_call",id,name}+summary), and ToolCallResponsePart +// ({type:"tool_call_response",id,response}). A tool summary is not the +// tool's arguments, so no arguments field is ever fabricated from it. type contentPart struct { - Type string `json:"type"` - Content string `json:"content,omitempty"` - Name string `json:"name,omitempty"` - Summary string `json:"summary,omitempty"` + Type string `json:"type"` + Content string `json:"content,omitempty"` + ID string `json:"id,omitempty"` + Name string `json:"name,omitempty"` + Summary string `json:"summary,omitempty"` + Response string `json:"response,omitempty"` + // IsError mirrors the wire's is_error on failed tool calls; it is + // content-bearing (an errored empty result is signal, not absence) + // and accounts a fixed footprint. Truncated marks a part whose bulk + // field was cut, so a consumer never reads a fragment as a whole + // result; it is set only by the collector's own cuts and stays + // outside the accounting. + IsError bool `json:"is_error,omitempty"` + Truncated bool `json:"fullsend.truncated,omitempty"` + // bulkScanned records that the bulk field was already redacted at + // accumulation (per-result cap or pre-trim), so Result must not scan + // those bytes again — a re-scan can re-match masked values and + // double-count findings. + bulkScanned bool +} + +// isErrorFootprint is the serialized cost of `"is_error":true,` — the +// bytes an errored-empty part contributes to the attribute. +const isErrorFootprint = 16 + +// MarshalJSON emits the schema-REQUIRED response key on +// tool_call_response parts even when the response is empty; omitempty +// on the shared struct would drop it. +func (p contentPart) MarshalJSON() ([]byte, error) { + type alias contentPart + if p.Type != "tool_call_response" { + return json.Marshal(alias(p)) + } + a := alias(p) + a.Response = "" // serialized by the outer field below instead + return json.Marshal(struct { + alias + Response string `json:"response"` + }{a, p.Response}) +} + +// contentBytes counts a part's content-bearing bytes. A part with none +// contributes nothing to the output and is never kept. An error flag is +// content: a failed call with empty output must survive, so it counts +// its serialized footprint. +func contentBytes(p contentPart) int { + n := len(p.Content) + len(p.Name) + len(p.Summary) + len(p.Response) + if p.IsError { + n += isErrorFootprint + } + return n } +// partSize is the part's budget footprint: content-bearing bytes plus +// id bytes, since the id serializes into the attribute like everything +// else. JSON syntax and escaping added at marshal time remain uncounted +// — the budget is measured on raw part bytes, as documented on +// maxContentBytes. func partSize(p contentPart) int { - return len(p.Content) + len(p.Name) + len(p.Summary) + return contentBytes(p) + len(p.ID) +} + +// boundedID drops an id that exceeds maxToolIDBytes; the part survives +// without correlation rather than carrying a malformed identifier. +func boundedID(id string) string { + if len(id) > maxToolIDBytes { + return "" + } + return id } // contentMessage is one message in the gen_ai.output.messages array. // finish_reason is REQUIRED by the schema's OutputMessage definition. +// +// The whole iteration is deliberately shaped as ONE assistant message, +// a knowing deviation from the convention on two separate counts: +// +// - role placement: the convention's worked example puts +// client-executed tool results under a role:"tool" message in +// gen_ai.input.messages; here they ride in the assistant output +// record. Schema-valid — ToolCallResponsePart is admitted in +// output messages. +// - cardinality: the registry note on gen_ai.output.messages says +// each message corresponds to exactly one generation +// (choice/candidate); this record packs an iteration's many +// generations into one message, which that note forbids +// independently of part-type admission. +// +// Rationale for both: parts keep stream order, and the iteration has +// exactly one meaningful finish_reason — per-generation messages would +// each require a finish_reason with no independent meaning. If a +// consumer ever needs per-generation messages, that is a deliberate +// carrier change, not a reinterpretation of this shape. type contentMessage struct { Role string `json:"role"` Parts []contentPart `json:"parts"` @@ -101,8 +208,8 @@ type contentResult struct { // OutputMessages is the gen_ai.output.messages JSON string, empty // when the iteration produced no content. OutputMessages string - // DroppedBytes counts raw content bytes removed by the size budget - // (content, tool name, and summary bytes alike). + // DroppedBytes counts raw part bytes removed by the size budget + // (content, tool name, summary, tool response, and id bytes alike). DroppedBytes int // Truncated reports whether the budget cut or dropped anything. Truncated bool @@ -143,8 +250,8 @@ func newContentCollector(maxBytes int) *contentCollector { // Handle consumes one normalized event. Contiguous text and reasoning // deltas of the same kind coalesce into a single part (the Claude parser -// emits per-delta); tool calls are discrete parts. All other event kinds -// carry no conversation content and are ignored. +// emits per-delta); tool calls and tool results are discrete parts. All +// other event kinds carry no conversation content and are ignored. func (c *contentCollector) Handle(evt agentruntime.AgentEvent) { if c == nil { return @@ -155,10 +262,41 @@ func (c *contentCollector) Handle(evt agentruntime.AgentEvent) { case agentruntime.ThinkingEvent: c.appendText("reasoning", e.Text) case agentruntime.ToolUseEvent: - c.parts = append(c.parts, contentPart{Type: "tool_call", Name: e.Name, Summary: e.Summary}) - c.total += len(e.Name) + len(e.Summary) - c.evictOverflow() + c.appendPart(contentPart{Type: "tool_call", ID: boundedID(e.ID), Name: e.Name, Summary: e.Summary}) + case agentruntime.ToolResultEvent: + p := contentPart{Type: "tool_call_response", ID: boundedID(e.ID), Response: e.Result, IsError: e.IsError} + // A parser-side partial flatten (non-text blocks skipped) is a + // cut like any other: the part must not read as a whole result. + p.Truncated = e.Partial + if len(p.Response) > maxToolResultBytes { + // Redact before the cap cut — the same invariant as every + // other cut: trimming raw bytes first could split a secret at + // the boundary past recognition. Redaction alone can shrink + // the response under the cap; that is not a cut. + p.Response = c.redact(p.Response, &c.findings) + p.bulkScanned = true + kept := tailToRuneBoundary(p.Response, maxToolResultBytes) + c.evicted += len(p.Response) - len(kept) + if len(kept) < len(p.Response) { + p.Truncated = true + } + p.Response = kept + } + c.appendPart(p) + } +} + +// appendPart admits one discrete part. Parts with no content-bearing +// bytes are refused: they would contribute nothing to the output (an +// empty result produces no part) yet accumulate unboundedly, invisible +// to the size-based eviction. +func (c *contentCollector) appendPart(p contentPart) { + if contentBytes(p) == 0 { + return } + c.parts = append(c.parts, p) + c.total += partSize(p) + c.evictOverflow() } func (c *contentCollector) appendText(kind, text string) { @@ -167,6 +305,10 @@ func (c *contentCollector) appendText(kind, text string) { } if n := len(c.parts); n > 0 && c.parts[n-1].Type == kind { c.parts[n-1].Content += text + // The appended bytes are unscanned; a secret can straddle the + // old/new boundary, so the WHOLE field must rescan — clear the + // pre-trim's scanned flag rather than tracking a prefix. + c.parts[n-1].bulkScanned = false } else { c.parts = append(c.parts, contentPart{Type: kind, Content: text}) } @@ -191,23 +333,54 @@ func (c *contentCollector) evictOverflow() { if c.total-head < c.maxBytes { break } - c.redact(c.parts[0].Content, &c.findings) - c.redact(c.parts[0].Name, &c.findings) - c.redact(c.parts[0].Summary, &c.findings) + hp := &c.parts[0] + hb := bulkField(hp) + if !hp.bulkScanned { + c.redact(*hb, &c.findings) + } + if hb != &hp.Content { + c.redact(hp.Content, &c.findings) + } + if hb != &hp.Response { + c.redact(hp.Response, &c.findings) + } + c.redact(hp.Name, &c.findings) + c.redact(hp.Summary, &c.findings) + c.redact(hp.ID, &c.findings) c.evicted += head c.total -= head c.parts = c.parts[1:] } if len(c.parts) == 1 && c.parts[0].Type != "tool_call" && c.total > 2*c.maxBytes { - p := &c.parts[0] - before := len(p.Content) - p.Content = c.redact(p.Content, &c.findings) - c.total -= before - len(p.Content) - kept := tailToRuneBoundary(p.Content, c.maxBytes) - c.evicted += len(p.Content) - len(kept) - c.total -= len(p.Content) - len(kept) - p.Content = kept + bulk := bulkField(&c.parts[0]) + before := len(*bulk) + // Deliberately unconditional: a re-fired pre-trim means bytes + // coalesced since the last scan, and a straddling secret needs + // the whole field visible — redact-before-cut outranks avoiding + // a rare re-match of already-masked values. + *bulk = c.redact(*bulk, &c.findings) + c.parts[0].bulkScanned = true + c.total -= before - len(*bulk) + kept := tailToRuneBoundary(*bulk, c.maxBytes) + c.evicted += len(*bulk) - len(kept) + c.total -= len(*bulk) - len(kept) + if len(kept) < len(*bulk) { + c.parts[0].Truncated = true + } + *bulk = kept + } +} + +// bulkField returns the part's dominant content-bearing field, the one +// size cuts operate on: response for tool_call_response parts, content +// for text and reasoning. tool_call parts have no bulk field — a partial +// name or summary would misrepresent the call, so they are never cut, +// only dropped whole. +func bulkField(p *contentPart) *string { + if p.Type == "tool_call_response" { + return &p.Response } + return &p.Content } // Result assembles the redacted, size-bounded output messages for one @@ -231,10 +404,20 @@ func (c *contentCollector) Result(finishReason string) contentResult { redacted := make([]contentPart, 0, len(c.parts)) for _, p := range c.parts { - p.Content = c.redact(p.Content, &res.Findings) + bulk := bulkField(&p) + if !p.bulkScanned { + *bulk = c.redact(*bulk, &res.Findings) + } + if bulk != &p.Content { + p.Content = c.redact(p.Content, &res.Findings) + } + if bulk != &p.Response { + p.Response = c.redact(p.Response, &res.Findings) + } p.Name = c.redact(p.Name, &res.Findings) p.Summary = c.redact(p.Summary, &res.Findings) - if partSize(p) == 0 { + p.ID = c.redactID(p.ID, &res.Findings) + if contentBytes(p) == 0 { continue // sanitized away entirely; the finding is recorded } redacted = append(redacted, p) @@ -257,14 +440,22 @@ func (c *contentCollector) Result(finishReason string) contentResult { continue } res.Truncated = true - if !full && p.Type != "tool_call" && remaining > 0 { - tail := tailToRuneBoundary(p.Content, remaining) - res.DroppedBytes += size - len(tail) - if tail != "" { - p.Content = tail - kept = append(kept, p) - } + bulk := bulkField(&p) + // Bytes the part carries besides its bulk field (its id, for + // tool_call_response parts) must fit before any bulk tail can. + nonBulk := size - len(*bulk) + tail := "" + if !full && p.Type != "tool_call" && remaining > nonBulk { + tail = tailToRuneBoundary(*bulk, remaining-nonBulk) + } + if tail != "" { + res.DroppedBytes += size - nonBulk - len(tail) + *bulk = tail + p.Truncated = true + kept = append(kept, p) } else { + // The part drops whole — every byte it carried is charged, + // id included (an empty rune-boundary tail lands here too). res.DroppedBytes += size } full = true @@ -273,6 +464,16 @@ func (c *contentCollector) Result(finishReason string) contentResult { if len(kept) == 0 { return res } + for _, p := range kept { + // A kept part marked truncated (parser-side partial flatten, or + // a cap cut in an otherwise under-budget iteration) must surface + // on the span marker too — it is the only cheap filter for + // affected spans. No byte count is fabricated for it. + if p.Truncated { + res.Truncated = true + break + } + } for i, j := 0, len(kept)-1; i < j; i, j = i+1, j-1 { kept[i], kept[j] = kept[j], kept[i] } @@ -305,6 +506,21 @@ func (c *contentCollector) redact(text string, findings *[]security.Finding) str return text } +// redactID scans a part id like every other stream-derived string; on +// any finding the id is dropped entirely — a substituted id could +// falsely collide with another call's. +func (c *contentCollector) redactID(id string, findings *[]security.Finding) string { + if id == "" { + return id + } + scanned := c.pipeline.Scan(id) + *findings = append(*findings, scanned.Findings...) + if len(scanned.Findings) > 0 { + return "" + } + return id +} + // tailToRuneBoundary keeps at most the last n bytes of s, starting on a // rune boundary. func tailToRuneBoundary(s string, n int) string { diff --git a/internal/cli/content_collector_test.go b/internal/cli/content_collector_test.go index 1904e63d64..96638e00a5 100644 --- a/internal/cli/content_collector_test.go +++ b/internal/cli/content_collector_test.go @@ -30,6 +30,10 @@ func decodeOutputMessages(t *testing.T, raw string) []map[string]any { part, ok := p.(map[string]any) require.True(t, ok) require.Contains(t, part, "type", "schema requires type on every part") + if part["type"] == "tool_call_response" { + require.Contains(t, part, "response", + "the schema's ToolCallResponsePart.required is [\"type\",\"response\"] — the key must be present even for an empty result") + } } } return msgs @@ -89,6 +93,39 @@ func TestContentCollector_ToolUseBecomesToolCallPart(t *testing.T) { "a summary is not the tool's arguments; do not fabricate them") } +func TestContentCollector_ToolCallPartCarriesID(t *testing.T) { + c := newContentCollector(4096) + c.Handle(agentruntime.ToolUseEvent{ID: "toolu_09qrs", Name: "Read", Summary: "/src/main.go"}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + part := partAt(t, msgs, 0) + assert.Equal(t, "toolu_09qrs", part["id"], + "tool_call parts carry the id that correlates them with tool_call_response parts") +} + +func TestContentCollector_IDlessToolCallOmitsIDKey(t *testing.T) { + // Runtimes without wire-format call ids (and the assistant fallback + // path before ids existed) emit ID-less events; the schema's id is + // optional, so the key is omitted rather than serialized empty. + c := newContentCollector(4096) + c.Handle(agentruntime.ToolUseEvent{Name: "Bash", Summary: "ls"}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + assert.NotContains(t, partAt(t, msgs, 0), "id") +} + +func TestContentCollector_ToolResultBecomesToolCallResponsePart(t *testing.T) { + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_01abc", Result: "main.go\nutil.go\n"}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + part := partAt(t, msgs, 0) + assert.Equal(t, "tool_call_response", part["type"]) + assert.Equal(t, "toolu_01abc", part["id"]) + assert.Equal(t, "main.go\nutil.go\n", part["response"], + "the schema's result field is named response, not result") +} + func TestContentCollector_RedactsSecretsAndSurfacesFindings(t *testing.T) { secret := "ghp_" + strings.Repeat("a", 36) c := newContentCollector(4096) @@ -111,6 +148,42 @@ func TestContentCollector_RedactsToolCallNameAndSummary(t *testing.T) { "tool_call name and summary are captured content and must be redacted") } +func TestContentCollector_RedactsToolResultResponse(t *testing.T) { + // Tool results are the highest-secret-density field in the stream — + // they carry file contents and command output verbatim. + secret := "ghp_" + strings.Repeat("d", 36) + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_06sec", Result: "config dump: " + secret}) + + res := c.Result("stop") + assert.NotContains(t, res.OutputMessages, secret, + "a secret inside a tool result must not survive assembly") + assert.NotEmpty(t, res.Findings) +} + +func TestContentCollector_EmptyToolResultProducesNoPart(t *testing.T) { + // An empty result carries no content-bearing bytes; like text + // sanitized to empty, it produces no part — which also means no + // tool_call_response part ever omits its schema-required response key. + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_07empty", Result: ""}) + + assert.Empty(t, c.Result("stop").OutputMessages) +} + +func TestContentCollector_ToolResultsStayDiscrete(t *testing.T) { + // Unlike text/reasoning deltas, tool results never coalesce. + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_a", Result: "one"}) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_b", Result: "two"}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + parts := msgs[0]["parts"].([]any) + require.Len(t, parts, 2) + assert.Equal(t, "toolu_a", parts[0].(map[string]any)["id"]) + assert.Equal(t, "toolu_b", parts[1].(map[string]any)["id"]) +} + func TestContentCollector_PreservesCleanText(t *testing.T) { c := newContentCollector(4096) c.Handle(agentruntime.TextEvent{Text: "nothing secret here"}) @@ -279,6 +352,373 @@ func TestContentCollector_EvictedPartsAreStillScanned(t *testing.T) { "findings inside evicted parts must still be counted") } +func TestContentCollector_EvictedToolResultsAreStillScanned(t *testing.T) { + // Response bytes in parts evicted during accumulation must be scanned + // exactly like Content/Name/Summary — findings count even when the + // budget drops the part. + secret := "ghp_" + strings.Repeat("e", 36) + c := newContentCollector(30) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_ev", Result: "leak: " + secret}) + c.Handle(agentruntime.ThinkingEvent{Text: strings.Repeat("z", 30)}) + c.Handle(agentruntime.TextEvent{Text: "the end"}) + + require.LessOrEqual(t, len(c.parts), 2, + "the secret-bearing tool result must have been evicted during accumulation") + + res := c.Result("stop") + assert.NotContains(t, res.OutputMessages, secret) + assert.NotEmpty(t, res.Findings, + "findings inside evicted tool results must still be counted") +} + +func TestContentCollector_BoundaryToolResultKeepsResponseTail(t *testing.T) { + // A tool_call_response at the suffix boundary is tail-cut on its + // response — like text, and unlike tool_call (whose name+summary + // would be misrepresented by a cut). The id survives the trim and + // its 9 bytes occupy budget first: "final" (5) leaves 15, the id + // takes 9, so 6 response bytes fit. + c := newContentCollector(20) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_cut", Result: "0123456789ABCDEFGHIJ"}) + c.Handle(agentruntime.TextEvent{Text: "final"}) + + res := c.Result("stop") + require.True(t, res.Truncated) + assert.Equal(t, 14, res.DroppedBytes, + "exactly the response bytes that did not fit next to the id are dropped") + + msgs := decodeOutputMessages(t, res.OutputMessages) + part := partAt(t, msgs, 0) + assert.Equal(t, "tool_call_response", part["type"]) + assert.Equal(t, "toolu_cut", part["id"]) + assert.Equal(t, "EFGHIJ", part["response"]) + assert.Equal(t, "final", partAt(t, msgs, 1)["content"]) +} + +func TestContentCollector_DroppedToolResultCountsResponseBytes(t *testing.T) { + // When no budget remains at the boundary, the whole response part + // drops and its response and id bytes land in DroppedBytes exactly. + c := newContentCollector(5) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_drop", Result: "0123456789"}) + c.Handle(agentruntime.TextEvent{Text: "final"}) + + res := c.Result("stop") + require.True(t, res.Truncated) + assert.Equal(t, 20, res.DroppedBytes) + msgs := decodeOutputMessages(t, res.OutputMessages) + require.Len(t, msgs[0]["parts"].([]any), 1) + assert.Equal(t, "final", partAt(t, msgs, 0)["content"]) +} + +func TestContentCollector_PreTrimRedactsGiantToolResult(t *testing.T) { + // The over-double-budget pre-trim must operate on a lone + // tool_call_response's response field with the same + // redact-before-cut invariant as text content. + secret := "ghp_" + strings.Repeat("f", 36) + tail := strings.Repeat("! ", 35) + c := newContentCollector(100) + c.Handle(agentruntime.ToolResultEvent{ + ID: "toolu_giant", + Result: strings.Repeat("x", 140) + secret + tail, + }) + + require.LessOrEqual(t, c.total, 2*c.maxBytes, + "a lone giant tool result must be pre-trimmed to bound memory") + + res := c.Result("stop") + assert.NotContains(t, res.OutputMessages, strings.Repeat("f", 10), + "no fragment of a boundary-straddling secret may survive the pre-trim") + assert.NotEmpty(t, res.Findings) + + msgs := decodeOutputMessages(t, res.OutputMessages) + part := partAt(t, msgs, 0) + assert.Equal(t, "toolu_giant", part["id"]) + assert.True(t, strings.HasSuffix(part["response"].(string), tail), + "the response ending must survive the pre-trim") +} + +func TestContentCollector_ErrorToolResultCarriesErrorKey(t *testing.T) { + // A failed call's part carries is_error; successful parts omit the + // key. additionalProperties permits the sibling, same latitude as + // tool_call's summary. + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_err", Result: "exit 1", IsError: true}) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_ok", Result: "done"}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + failed := partAt(t, msgs, 0) + assert.Equal(t, true, failed["is_error"]) + assert.NotContains(t, partAt(t, msgs, 1), "is_error") +} + +func TestContentCollector_CutPartsCarryTruncatedMarker(t *testing.T) { + // Every cut part says so: a scorer must not read a fragment as a + // whole result. The marker is structural (outside the accounting) + // and absent on untouched parts. + c := newContentCollector(maxContentBytes) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_big", Result: strings.Repeat("a", maxToolResultBytes+100)}) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_ok", Result: "small"}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + assert.Equal(t, true, partAt(t, msgs, 0)["fullsend.truncated"], + "a cap-cut response must be marked") + assert.NotContains(t, partAt(t, msgs, 1), "fullsend.truncated") +} + +func TestContentCollector_BoundaryTrimMarksPart(t *testing.T) { + c := newContentCollector(20) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_cut", Result: "0123456789ABCDEFGHIJ"}) + c.Handle(agentruntime.TextEvent{Text: "final"}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + assert.Equal(t, true, partAt(t, msgs, 0)["fullsend.truncated"], + "a boundary-trimmed part must be marked") + assert.NotContains(t, partAt(t, msgs, 1), "fullsend.truncated", + "the intact ending stays unmarked") +} + +func TestContentCollector_BareOAuthTokenInResultRedacted(t *testing.T) { + // Tool results carry raw command stdout; a bare GCP bearer token — + // the credential class WIF-provisioned runs actually handle — must + // not survive to the span. + token := "ya29.a0AfB_byDEMOtoken1234567890abcdefghij" + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_gcp", Result: "access token: " + token}) + + res := c.Result("stop") + assert.NotContains(t, res.OutputMessages, "a0AfB_byDEMO") + assert.NotEmpty(t, res.Findings) +} + +func TestContentCollector_RedactionUnderCapDoesNotMarkTruncated(t *testing.T) { + // Redaction can shrink an over-cap response below the cap; nothing + // is cut then, so neither the part marker nor the span truncation + // state may fire. The 40-byte secret masks to 7, shrinking an + // 8,212-byte response to 8,179 — under the 8,192 cap. + secret := "ghp_" + strings.Repeat("j", 36) + c := newContentCollector(maxContentBytes) + c.Handle(agentruntime.ToolResultEvent{ + ID: "toolu_shrink", + Result: strings.Repeat("x", maxToolResultBytes+20-len(secret)) + secret, + }) + + res := c.Result("stop") + assert.False(t, res.Truncated, "nothing was cut — redaction shrink is not truncation") + assert.Zero(t, res.DroppedBytes) + assert.NotEmpty(t, res.Findings) + msgs := decodeOutputMessages(t, res.OutputMessages) + assert.NotContains(t, partAt(t, msgs, 0), "fullsend.truncated") +} + +func TestContentCollector_CappedResultScannedOnce(t *testing.T) { + // The cap path already scans the response; Result must not scan the + // same bytes again — a re-scan re-matches masked db-URL passwords + // (supe... still fits the 4+-char capture) and double-counts the + // finding. + c := newContentCollector(maxContentBytes) + c.Handle(agentruntime.ToolResultEvent{ + ID: "toolu_db", + Result: strings.Repeat("x", maxToolResultBytes) + " postgres://user:supersecret@host", + }) + + res := c.Result("stop") + assert.NotContains(t, res.OutputMessages, "supersecret") + assert.Len(t, res.Findings, 1, + "one secret must yield exactly one finding, not one per scan") +} + +func TestContentCollector_CoalescedAfterPreTrimStillScanned(t *testing.T) { + // The pre-trim marks its part's bulk as scanned; deltas that coalesce + // into that part afterwards are NOT scanned yet, so the flag must + // clear on coalesce or the appended bytes reach the span raw — the + // tail-kept suffix keeps exactly the newest bytes. + secret := "ghp_" + strings.Repeat("k", 36) + c := newContentCollector(1024) + c.Handle(agentruntime.TextEvent{Text: strings.Repeat("x", 3000)}) + c.Handle(agentruntime.TextEvent{Text: " leak: " + secret}) + + res := c.Result("stop") + assert.NotContains(t, res.OutputMessages, secret, + "bytes coalesced after a pre-trim must still be redacted") + assert.NotEmpty(t, res.Findings) +} + +func TestContentCollector_PartialResultCarriesTruncatedMarker(t *testing.T) { + // A result whose non-text blocks were skipped at the parser is a + // fragment; the part reuses the same marker every other cut sets. + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_mix", Result: "the text half", Partial: true}) + + res := c.Result("stop") + msgs := decodeOutputMessages(t, res.OutputMessages) + part := partAt(t, msgs, 0) + assert.Equal(t, true, part["fullsend.truncated"], + "a partial result must not read as a whole one") + assert.Equal(t, "the text half", part["response"]) + assert.True(t, res.Truncated, + "the span-level marker must fire too — it is the only cheap filter for affected spans") + assert.Zero(t, res.DroppedBytes, + "the parser never measured the skipped blocks; no byte count is fabricated") +} + +func TestContentCollector_ErroredEmptyResultKept(t *testing.T) { + // A failed call with empty output is signal, not absence: the part + // survives with is_error and its schema-required response key, even + // empty. Successful empty results still produce no part. + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_errempty", Result: "", IsError: true}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + part := partAt(t, msgs, 0) + assert.Equal(t, "tool_call_response", part["type"]) + assert.Equal(t, true, part["is_error"]) + assert.Equal(t, "", part["response"], + "the schema-required response key is present even when empty") + assert.Equal(t, "toolu_errempty", part["id"]) +} + +func TestContentCollector_EmptyTailDropChargesWholePart(t *testing.T) { + // When the boundary window lands inside a trailing multi-byte rune, + // the tail is empty and the part drops whole — id included — so the + // whole part must be charged, not just its response bytes. Budget 16: + // "final" (5) leaves 11; the id (9) leaves a 2-byte window inside €. + c := newContentCollector(16) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_cut", Result: "0123456789ABCDEFG€"}) + c.Handle(agentruntime.TextEvent{Text: "final"}) + + res := c.Result("stop") + require.True(t, res.Truncated) + assert.Equal(t, 29, res.DroppedBytes, + "a whole-dropped part is charged in full: 20 response + 9 id bytes") + + msgs := decodeOutputMessages(t, res.OutputMessages) + parts := msgs[0]["parts"].([]any) + require.Len(t, parts, 1) + assert.Equal(t, "final", parts[0].(map[string]any)["content"]) +} + +func TestContentCollector_IDBytesCountTowardBudget(t *testing.T) { + // Ids are serialized into the attribute, so they count toward the + // budget like every other part byte — uncounted ids would let the + // attribute grow past the budget in aggregate. + c := newContentCollector(30) + c.Handle(agentruntime.ToolResultEvent{ID: "12345678901234567890", Result: "0123456789"}) + c.Handle(agentruntime.TextEvent{Text: "end"}) + + res := c.Result("stop") + require.True(t, res.Truncated) + assert.Equal(t, 3, res.DroppedBytes, + "the id's 20 bytes count: only 7 response bytes fit next to it") + + msgs := decodeOutputMessages(t, res.OutputMessages) + part := partAt(t, msgs, 0) + assert.Equal(t, "3456789", part["response"]) + assert.Equal(t, "12345678901234567890", part["id"], + "a kept boundary part keeps its id intact") +} + +func TestContentCollector_SecretBearingIDDropped(t *testing.T) { + // Ids pass the same redaction scan as every other stream-derived + // string; a finding drops the id entirely — never substitutes, since + // a rewritten id could falsely collide. + secret := "ghp_" + strings.Repeat("h", 36) + c := newContentCollector(4096) + c.Handle(agentruntime.ToolResultEvent{ID: secret, Result: "clean output"}) + + res := c.Result("stop") + assert.NotContains(t, res.OutputMessages, secret) + assert.NotEmpty(t, res.Findings) + msgs := decodeOutputMessages(t, res.OutputMessages) + part := partAt(t, msgs, 0) + assert.NotContains(t, part, "id", "a secret-bearing id is dropped, not rewritten") + assert.Equal(t, "clean output", part["response"]) +} + +func TestContentCollector_ZeroContentPartsDoNotAccumulate(t *testing.T) { + // Parts with no content-bearing bytes are refused at Handle: they + // would contribute nothing to the output (the empty-result rule) yet + // accumulate unboundedly, invisible to the size-based eviction. + c := newContentCollector(4096) + for i := 0; i < 100; i++ { + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_x", Result: ""}) + c.Handle(agentruntime.ToolUseEvent{ID: "toolu_y"}) + } + assert.Empty(t, c.parts, "zero-content parts must not accumulate") +} + +func TestContentCollector_OversizedIDDropped(t *testing.T) { + // The stream decodes ids unbounded and Level 3 lifts the SDK + // attribute cap, so an id beyond any legitimate format is treated as + // malformed and dropped — never truncated, since a truncated id + // could falsely collide. The part itself survives, uncorrelated. + huge := strings.Repeat("x", maxToolIDBytes+1) + c := newContentCollector(4096) + c.Handle(agentruntime.ToolUseEvent{ID: huge, Name: "Bash", Summary: "ls"}) + c.Handle(agentruntime.ToolResultEvent{ID: huge, Result: "output"}) + + msgs := decodeOutputMessages(t, c.Result("stop").OutputMessages) + parts := msgs[0]["parts"].([]any) + require.Len(t, parts, 2) + for _, p := range parts { + assert.NotContains(t, p.(map[string]any), "id", + "an oversized id must be dropped, not serialized") + } +} + +func TestContentCollector_CapsOversizedToolResult(t *testing.T) { + // A single tool result larger than maxToolResultBytes keeps only its + // tail — measured on real review runs, uncapped results overflow the + // total budget and evict whole older parts; the cap lowers that + // pressure. Capped bytes land in DroppedBytes exactly. + c := newContentCollector(maxContentBytes) + oversized := strings.Repeat("a", maxToolResultBytes) + strings.Repeat("b", 100) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_cap", Result: oversized}) + + res := c.Result("stop") + require.True(t, res.Truncated) + assert.Equal(t, 100, res.DroppedBytes, + "exactly the bytes beyond the per-result cap are dropped") + + msgs := decodeOutputMessages(t, res.OutputMessages) + part := partAt(t, msgs, 0) + response := part["response"].(string) + assert.Len(t, response, maxToolResultBytes) + assert.True(t, strings.HasSuffix(response, strings.Repeat("b", 100)), + "the cap keeps the tail — the budget's suffix policy extended per-result") + assert.Equal(t, "toolu_cap", part["id"]) +} + +func TestContentCollector_CapRedactsBeforeCutting(t *testing.T) { + // The redaction-before-truncation invariant applies to the per-result + // cap exactly as to every other cut: a secret straddling the cap + // boundary must be redacted before the head is discarded. + // The cap keeps the tail, so the head-cut point for this 8262-byte + // result lands at byte 70 — inside the secret spanning [50:90]. A + // raw-first cut would keep an unrecognizable 20-byte fragment. + secret := "ghp_" + strings.Repeat("g", 36) + c := newContentCollector(maxContentBytes) + c.Handle(agentruntime.ToolResultEvent{ + ID: "toolu_capsec", + Result: strings.Repeat("x", 50) + secret + strings.Repeat("! ", 4086), + }) + + res := c.Result("stop") + assert.NotContains(t, res.OutputMessages, strings.Repeat("g", 10), + "no fragment of a cap-straddling secret may survive") + assert.NotEmpty(t, res.Findings) +} + +func TestContentCollector_SubCapToolResultUntouched(t *testing.T) { + c := newContentCollector(maxContentBytes) + within := strings.Repeat("c", maxToolResultBytes) + c.Handle(agentruntime.ToolResultEvent{ID: "toolu_fit", Result: within}) + + res := c.Result("stop") + assert.False(t, res.Truncated) + assert.Zero(t, res.DroppedBytes) + msgs := decodeOutputMessages(t, res.OutputMessages) + assert.Equal(t, within, partAt(t, msgs, 0)["response"]) +} + func TestContentCollector_EvictsWholeOldPartsExactly(t *testing.T) { // Long sessions must not accumulate unbounded content: parts older // than the suffix budget are evicted during Handle, and every diff --git a/internal/cli/run.go b/internal/cli/run.go index 85503a3596..17f32a8fd4 100644 --- a/internal/cli/run.go +++ b/internal/cli/run.go @@ -2167,8 +2167,11 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep agentCtx, agentSpan := tracer.Start(ctx, "agent", trace.WithAttributes(agentSpanStartAttrs(iteration, agentName)...)) // One collector per iteration: iteration and agent span are 1:1, so // a run-scoped collector would repeat earlier iterations' content on - // later spans. Nil when the Level 3 gate is off; nil is inert. + // later spans. Nil when the Level 3 gate is off; nil is inert. The + // tool-span tracker is per iteration for the same reason and is not + // gated: execute_tool spans are metadata. collector := newContentCollectorIfEnabled() + toolSpans := newToolSpanTracker(tracer, agentCtx) var metrics agentruntime.RunMetrics hooksSettings := "" if h.SecurityEnabled() { @@ -2190,7 +2193,7 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep Prompt: agentPrompt, Forge: forgePlatform, ModelAliases: configModelAliases, - OnEvent: contentEventHandler(agentruntime.NewEventRenderer(printer).Handle, collector), + OnEvent: iterationEventHandler(agentruntime.NewEventRenderer(printer).Handle, collector, toolSpans), }, printer, agentStart, &metrics) close(heartbeatDone) lastIterElapsed = time.Since(agentStart) @@ -2214,6 +2217,7 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep if runErr != nil { attachIterationContent("error") + recordToolSpanOverflow(agentSpan, toolSpans.Finish()) finalizeAgentSpan(agentSpan, runErr, iteration, exitCode, rt.System(), rt.Name(), &metrics, "") printer.StepFail("Agent execution failed") // Record the real exit code (rt.Run returns -1 when the agent never @@ -2261,6 +2265,7 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep contentFinishReason = "error" } attachIterationContent(contentFinishReason) + recordToolSpanOverflow(agentSpan, toolSpans.Finish()) finalizeAgentSpan(agentSpan, nil, iteration, exitCode, rt.System(), rt.Name(), &metrics, transcriptErrMsg) printer.Blank() @@ -3489,6 +3494,15 @@ func finalizeRootSpan(span trace.Span, runErr error, exitCode int, validationPas span.End() } +// recordToolSpanOverflow marks an agent span whose iteration reported more +// tool calls than the tracker records (maxToolSpansPerIteration), so a +// consumer can tell a partial execute_tool set from a complete one. +func recordToolSpanOverflow(span trace.Span, dropped int) { + if dropped > 0 { + span.SetAttributes(attribute.Int("fullsend.tool_spans.dropped", dropped)) + } +} + // finalizeSandboxSpan records the sandbox-create outcome and ends the // span. On failure the create error — which embeds raw supervisor/ // gateway/container logs — gets the same treatment as the agent and root @@ -4329,7 +4343,8 @@ func scanOutputFiles(outputDir, traceID string, printer *ui.Printer) error { } // Skip the telemetry JSONL: it is still open for append, and any // Level 3 conversation content in it was already redacted at - // assembly (contentCollector) before reaching a span, so it needs + // assembly (contentCollector) before reaching a span, as were the + // tool names on execute_tool spans (toolSpanTracker), so it needs // no post-hoc sweep. if path == filepath.Join(outputDir, telemetry.TelemetryFile) { return nil diff --git a/internal/cli/telemetry_run_test.go b/internal/cli/telemetry_run_test.go index 501cb96fc6..a8c1f378d4 100644 --- a/internal/cli/telemetry_run_test.go +++ b/internal/cli/telemetry_run_test.go @@ -1018,15 +1018,10 @@ func TestAgentSpanEndAttrs_ModelBoundedWithoutSDKCap(t *testing.T) { t.Fatal("gen_ai.request.model attribute not found") } -func TestContentEventHandler_NilCollectorKeepsDefaultRenderer(t *testing.T) { - assert.Nil(t, contentEventHandler(func(agentruntime.AgentEvent) {}, nil), - "gate off must leave OnEvent nil so the runtime's default renderer runs") -} - -func TestContentEventHandler_TeesToRendererAndCollector(t *testing.T) { +func TestIterationEventHandler_TeesToRendererAndCollector(t *testing.T) { var rendered []agentruntime.AgentEvent c := newContentCollector(4096) - handler := contentEventHandler(func(e agentruntime.AgentEvent) { rendered = append(rendered, e) }, c) + handler := iterationEventHandler(func(e agentruntime.AgentEvent) { rendered = append(rendered, e) }, c, nil) require.NotNil(t, handler) handler(agentruntime.TextEvent{Text: "hello"}) @@ -1092,8 +1087,8 @@ func TestContentCapture_EndToEndFileSink(t *testing.T) { } // TestContentCapture_GateOffProducesNoContent is the negative control: with -// the gate off the collector is nil, OnEvent stays nil (default renderer), -// and nothing content-shaped reaches the file sink. +// the gate off the collector is nil and nothing content-shaped reaches the +// file sink (tool spans are metadata and are emitted regardless). func TestContentCapture_GateOffProducesNoContent(t *testing.T) { t.Setenv("OTEL_SDK_DISABLED", "") t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "") diff --git a/internal/cli/tool_spans.go b/internal/cli/tool_spans.go new file mode 100644 index 0000000000..f5ffbf3437 --- /dev/null +++ b/internal/cli/tool_spans.go @@ -0,0 +1,184 @@ +package cli + +import ( + "context" + "strings" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/codes" + "go.opentelemetry.io/otel/trace" + + agentruntime "github.com/fullsend-ai/fullsend/internal/runtime" + "github.com/fullsend-ai/fullsend/internal/security" +) + +// maxToolSpansPerIteration bounds how many execute_tool spans one iteration +// may record. The count of calls is under the sandboxed agent's control, +// and Finish ends every open call in a burst right before the agent span +// ends: the OTLP batch processor's queue (2048 by default) drops newest +// spans when full, so an unbounded burst would evict the agent span — the +// one carrying the iteration's content. Half the queue leaves room for the +// rest of the trace; real review iterations run 117-255 calls. +const maxToolSpansPerIteration = 1024 + +// maxToolNameBytes bounds gen_ai.tool.name. The name comes from the +// sandboxed agent's stream; real names (mcp__server__tool included) are +// well under 100 bytes. +const maxToolNameBytes = 256 + +// maxToolSpanNameBytes bounds the tool-name part of an execute_tool span +// name. A span name is not an attribute, so no SDK limit ever applied to +// it. +const maxToolSpanNameBytes = 128 + +// toolSpanTracker turns the tool calls one iteration's runtime stream +// reports into execute_tool child spans of that iteration's agent span +// (semconv v1.37.0: gen_ai.operation.name, gen_ai.tool.name, +// gen_ai.tool.call.id, error.type on failure; span kind Internal). Tool +// content stays on the agent span's gen_ai.output.messages record — these +// spans are Level 1 metadata and are emitted whether or not the content +// gate is on. +// +// A span starts when the ToolUseEvent arrives and ends when the matching +// ToolResultEvent arrives, so both timestamps are runner-side receipt +// instants on one clock: tool_use arrival is arguments-complete rather +// than execution start, and tool_result arrival trails execution end by +// the pipe latency. Calls are keyed by id because the stream interleaves +// several open calls (parallel sub-agent dispatch); a call that never gets +// a result — the runtime was stopped, or its result line exceeded the +// parser's 1 MiB cap — is ended by Finish as error.type=unanswered, and a +// result whose call was never seen (its tool_use line was skipped) becomes +// a marked span of near-zero duration. Events without an id — pi and codex +// emit none, and server-side tools get none because their result never +// arrives as a tool_result — produce no span. The name goes through the +// same output pipeline as span content (Unicode normalization, then secret +// redaction) and is bounded before it reaches the span: it lands on a Level +// 1 span in the telemetry file the output scan exempts on the strength of +// that treatment. Delivery is synchronous on one goroutine, so no lock. +type toolSpanTracker struct { + tracer trace.Tracer + ctx context.Context + pipeline *security.Pipeline + open map[string]trace.Span + created int + dropped int +} + +func newToolSpanTracker(tracer trace.Tracer, agentCtx context.Context) *toolSpanTracker { + return &toolSpanTracker{ + tracer: tracer, + ctx: agentCtx, + pipeline: security.OutputPipeline(), + open: map[string]trace.Span{}, + } +} + +// Handle opens a span for a tool call and ends it on the call's result. A +// nil tracker is inert. +func (t *toolSpanTracker) Handle(evt agentruntime.AgentEvent) { + if t == nil { + return + } + switch e := evt.(type) { + case agentruntime.ToolUseEvent: + id := boundedID(e.ID) + if id == "" || !t.allow() { + return + } + if prev, ok := t.open[id]; ok { + endUnanswered(prev) + } + t.open[id] = t.start(id, e.Name) + case agentruntime.ToolResultEvent: + id := boundedID(e.ID) + if id == "" { + return + } + span, ok := t.open[id] + if ok { + delete(t.open, id) + } else { + if !t.allow() { + return + } + span = t.start(id, "") + span.SetAttributes(attribute.Bool("fullsend.tool.unmatched", true)) + } + finalizeToolSpan(span, e.IsError) + } +} + +// Finish ends every call still open as unanswered and returns how many +// calls got no span because the iteration passed maxToolSpansPerIteration; +// call it before the agent span ends. A nil tracker is inert. +func (t *toolSpanTracker) Finish() int { + if t == nil { + return 0 + } + for id, span := range t.open { + endUnanswered(span) + delete(t.open, id) + } + return t.dropped +} + +// allow reports whether one more span may be created this iteration. +func (t *toolSpanTracker) allow() bool { + if t.created >= maxToolSpansPerIteration { + t.dropped++ + return false + } + t.created++ + return true +} + +// safeName sanitizes the stream-derived tool name and bounds it — in that +// order, so a cut can never split a credential past recognition. An empty +// result with findings means the whole name was sanitized away (the same +// reading contentCollector.redact applies), so nothing is shown. +func (t *toolSpanTracker) safeName(name string) string { + scanned := t.pipeline.Scan(name) + if scanned.Sanitized != "" { + name = scanned.Sanitized + } else if len(scanned.Findings) > 0 { + return "" + } + return strings.ToValidUTF8(truncateStatusMsgTo(name, maxToolNameBytes), "") +} + +func (t *toolSpanTracker) start(id, name string) trace.Span { + attrs := []attribute.KeyValue{ + attribute.String("gen_ai.operation.name", "execute_tool"), + stringAttr("gen_ai.tool.call.id", id), + } + spanName := "execute_tool" + if name != "" { + name = t.safeName(name) + } + if name != "" { + attrs = append(attrs, attribute.String("gen_ai.tool.name", name)) + spanName += " " + strings.ToValidUTF8(truncateStatusMsgTo(name, maxToolSpanNameBytes), "") + } + _, span := t.tracer.Start(t.ctx, spanName, + trace.WithSpanKind(trace.SpanKindInternal), trace.WithAttributes(attrs...)) + return span +} + +// finalizeToolSpan ends a tool span from the wire's is_error flag — the +// only failure signal the stream carries, so error.type is a fixed value +// and no exception event is recorded. +func finalizeToolSpan(span trace.Span, isError bool) { + if isError { + span.SetAttributes(attribute.String("error.type", "tool_error")) + span.SetStatus(codes.Error, "tool reported is_error") + } else { + span.SetStatus(codes.Ok, "") + } + span.End() +} + +func endUnanswered(span trace.Span) { + span.SetAttributes(attribute.String("error.type", "unanswered")) + span.SetStatus(codes.Error, "no tool_result before the iteration ended") + span.End() +} diff --git a/internal/cli/tool_spans_test.go b/internal/cli/tool_spans_test.go new file mode 100644 index 0000000000..3a5a77f160 --- /dev/null +++ b/internal/cli/tool_spans_test.go @@ -0,0 +1,350 @@ +package cli + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "unicode/utf8" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/codes" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" + "go.opentelemetry.io/otel/trace" + + agentruntime "github.com/fullsend-ai/fullsend/internal/runtime" + "github.com/fullsend-ai/fullsend/internal/telemetry" +) + +// toolSpanFixture opens an agent span on a recording provider and returns +// a tracker parented under it, the recorder, and the agent span. +func toolSpanFixture(t *testing.T) (*toolSpanTracker, *tracetest.SpanRecorder, trace.Span) { + t.Helper() + rec := tracetest.NewSpanRecorder() + tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(rec)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + tracer := tp.Tracer("test") + agentCtx, agentSpan := tracer.Start(context.Background(), "agent") + return newToolSpanTracker(tracer, agentCtx), rec, agentSpan +} + +// endedToolSpans returns every ended span except the agent span, in end +// order. +func endedToolSpans(rec *tracetest.SpanRecorder) []tracetest.SpanStub { + var out []tracetest.SpanStub + for _, s := range rec.Ended() { + if s.Name() != "agent" { + out = append(out, tracetest.SpanStubFromReadOnlySpan(s)) + } + } + return out +} + +func toolSpanAttrs(s tracetest.SpanStub) map[attribute.Key]attribute.Value { + out := map[attribute.Key]attribute.Value{} + for _, kv := range s.Attributes { + out[kv.Key] = kv.Value + } + return out +} + +func TestToolSpanTracker_PairEmitsExecuteToolChild(t *testing.T) { + tr, rec, agentSpan := toolSpanFixture(t) + + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_01", Name: "Bash", Summary: "ls"}) + require.Empty(t, rec.Ended(), "the span stays open until its result arrives") + + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_01", Result: "ok"}) + spans := endedToolSpans(rec) + require.Len(t, spans, 1) + s := spans[0] + assert.Equal(t, "execute_tool Bash", s.Name) + assert.Equal(t, agentSpan.SpanContext().SpanID(), s.Parent.SpanID(), + "execute_tool must be a child of the iteration's agent span") + assert.Equal(t, trace.SpanKindInternal, s.SpanKind) + assert.False(t, s.EndTime.Before(s.StartTime)) + + attrs := toolSpanAttrs(s) + assert.Equal(t, "execute_tool", attrs["gen_ai.operation.name"].AsString()) + assert.Equal(t, "Bash", attrs["gen_ai.tool.name"].AsString()) + assert.Equal(t, "toolu_01", attrs["gen_ai.tool.call.id"].AsString()) + assert.NotContains(t, attrs, attribute.Key("error.type"), "success carries no error.type") + assert.NotContains(t, attrs, attribute.Key("fullsend.tool.unmatched")) + assert.Equal(t, codes.Ok, s.Status.Code) + assert.Empty(t, tr.open, "an answered call must not stay open (Finish would re-end it, and the map would grow per call)") +} + +func TestToolSpanTracker_ErrorResultSetsErrorTypeAndStatus(t *testing.T) { + tr, rec, _ := toolSpanFixture(t) + + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_02", Name: "Edit"}) + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_02", Result: "no such file", IsError: true}) + + spans := endedToolSpans(rec) + require.Len(t, spans, 1) + assert.Equal(t, "tool_error", toolSpanAttrs(spans[0])["error.type"].AsString()) + assert.Equal(t, codes.Error, spans[0].Status.Code) + assert.NotEmpty(t, spans[0].Status.Description) + assert.Empty(t, spans[0].Events, "the wire carries no error text, so no exception event") +} + +func TestToolSpanTracker_UnansweredCallEndsAtFinish(t *testing.T) { + tr, rec, _ := toolSpanFixture(t) + + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_03", Name: "Bash"}) + require.Empty(t, rec.Ended()) + + tr.Finish() + spans := endedToolSpans(rec) + require.Len(t, spans, 1) + assert.Equal(t, "unanswered", toolSpanAttrs(spans[0])["error.type"].AsString()) + assert.Equal(t, codes.Error, spans[0].Status.Code) + + tr.Finish() + assert.Len(t, endedToolSpans(rec), 1, "Finish is idempotent") + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_03"}) + assert.Len(t, endedToolSpans(rec), 2, "a result after Finish is an orphan, never a second end") +} + +func TestToolSpanTracker_OrphanResultIsMarkedUnmatched(t *testing.T) { + tr, rec, agentSpan := toolSpanFixture(t) + + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_04", Result: "late"}) + + spans := endedToolSpans(rec) + require.Len(t, spans, 1) + s := spans[0] + assert.Equal(t, "execute_tool", s.Name, "no tool_use was seen, so no name is known") + assert.Equal(t, agentSpan.SpanContext().SpanID(), s.Parent.SpanID()) + attrs := toolSpanAttrs(s) + assert.True(t, attrs["fullsend.tool.unmatched"].AsBool()) + assert.Equal(t, "toolu_04", attrs["gen_ai.tool.call.id"].AsString()) + assert.NotContains(t, attrs, attribute.Key("gen_ai.tool.name")) + assert.Equal(t, codes.Ok, s.Status.Code) + + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_05", IsError: true}) + spans = endedToolSpans(rec) + require.Len(t, spans, 2) + assert.Equal(t, "tool_error", toolSpanAttrs(spans[1])["error.type"].AsString()) + assert.Equal(t, codes.Error, spans[1].Status.Code) +} + +func TestToolSpanTracker_EventsWithoutIDProduceNoSpan(t *testing.T) { + // pi and codex emit ToolUseEvent without an id and never a result. + tr, rec, _ := toolSpanFixture(t) + + tr.Handle(agentruntime.ToolUseEvent{Name: "bash", Summary: "ls"}) + tr.Handle(agentruntime.ToolResultEvent{Result: "stray"}) + tr.Finish() + + assert.Empty(t, endedToolSpans(rec)) +} + +func TestToolSpanTracker_MalformedIDProducesNoSpan(t *testing.T) { + tr, rec, _ := toolSpanFixture(t) + long := strings.Repeat("i", maxToolIDBytes+1) + + tr.Handle(agentruntime.ToolUseEvent{ID: long, Name: "Bash"}) + tr.Handle(agentruntime.ToolResultEvent{ID: long}) + tr.Finish() + + assert.Empty(t, endedToolSpans(rec), "an id beyond the bound is malformed: no span, not a truncated one") +} + +func TestToolSpanTracker_NameIsBoundedAndRepaired(t *testing.T) { + tr, rec, _ := toolSpanFixture(t) + name := strings.Repeat("n", telemetry.MaxSpanAttrValueLen*2) + "\xff" + + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_06", Name: name}) + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_06"}) + + spans := endedToolSpans(rec) + require.Len(t, spans, 1) + s := spans[0] + attr := toolSpanAttrs(s)["gen_ai.tool.name"].AsString() + assert.LessOrEqual(t, len(attr), maxToolNameBytes) + assert.True(t, utf8.ValidString(attr)) + assert.True(t, strings.HasPrefix(s.Name, "execute_tool ")) + assert.LessOrEqual(t, len(s.Name), len("execute_tool ")+maxToolSpanNameBytes) + assert.True(t, utf8.ValidString(s.Name)) +} + +func TestToolSpanTracker_DuplicateIDSupersedesOpenCall(t *testing.T) { + tr, rec, _ := toolSpanFixture(t) + + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_07", Name: "Read"}) + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_07", Name: "Read"}) + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_07"}) + + spans := endedToolSpans(rec) + require.Len(t, spans, 2) + assert.Equal(t, "unanswered", toolSpanAttrs(spans[0])["error.type"].AsString(), + "the earlier open call is closed as unanswered when its id is reused") + assert.Equal(t, codes.Ok, spans[1].Status.Code) + tr.Finish() + assert.Len(t, endedToolSpans(rec), 2) +} + +func TestToolSpanTracker_CapsSpansPerIteration(t *testing.T) { + // An agent-controlled burst of calls must not fill the OTLP batch queue + // and evict the agent span that follows; past the cap nothing is + // tracked or started, and the overflow is reported. + tr, rec, _ := toolSpanFixture(t) + for i := 0; i < maxToolSpansPerIteration+5; i++ { + tr.Handle(agentruntime.ToolUseEvent{ID: fmt.Sprintf("toolu_%05d", i), Name: "Bash"}) + } + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_orphan"}) + + dropped := tr.Finish() + assert.Equal(t, 6, dropped, "five calls and one orphan past the cap") + assert.Len(t, endedToolSpans(rec), maxToolSpansPerIteration) + assert.Equal(t, 6, tr.Finish(), "the count is stable across calls") + assert.Equal(t, 0, newToolSpanTracker(nil, context.Background()).Finish()) +} + +func TestToolSpanTracker_NameIsRedactedAndBounded(t *testing.T) { + // The name is stream-derived and lands on a Level 1 span the output + // scan exempts, so it is redacted like the console summary and bounded + // far below the SDK cap. + tr, rec, _ := toolSpanFixture(t) + secret := "ghp_" + strings.Repeat("A1b2C3d4", 5) + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_09", Name: "mcp__vault__" + secret}) + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_09"}) + long := strings.Repeat("n", maxToolNameBytes*3) + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_10", Name: long}) + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_10"}) + + spans := endedToolSpans(rec) + require.Len(t, spans, 2) + name := toolSpanAttrs(spans[0])["gen_ai.tool.name"].AsString() + assert.NotContains(t, name, secret) + assert.NotContains(t, spans[0].Name, secret) + assert.True(t, strings.HasPrefix(name, "mcp__vault__"), "the non-secret part survives") + assert.LessOrEqual(t, len(toolSpanAttrs(spans[1])["gen_ai.tool.name"].AsString()), maxToolNameBytes) +} + +func TestToolSpanTracker_NameGoesThroughTheOutputSanitizer(t *testing.T) { + // The name lands on a Level 1 span in the telemetry file the output + // scan exempts, so it must get the same treatment as span content: + // Unicode normalization (escapes, NUL, zero-width and bidi overrides) + // and then secret redaction — not the secret pass alone. + tr, rec, _ := toolSpanFixture(t) + cases := map[string]string{ + "toolu_11": "Bash\x1b[31mred\x1b[0m", + "toolu_12": "Bash\u200b\u202eevil", + "toolu_13": "Bash\x00nul", + } + for id, name := range cases { + tr.Handle(agentruntime.ToolUseEvent{ID: id, Name: name}) + tr.Handle(agentruntime.ToolResultEvent{ID: id}) + } + spans := endedToolSpans(rec) + require.Len(t, spans, 3) + for _, s := range spans { + name := toolSpanAttrs(s)["gen_ai.tool.name"].AsString() + for _, bad := range []string{"\x1b", "\u200b", "\u202e", "\x00"} { + assert.NotContains(t, name, bad, "attribute must be normalized") + assert.NotContains(t, s.Name, bad, "span name must be normalized") + } + assert.True(t, strings.HasPrefix(name, "Bash"), "the visible part survives: %q", name) + } + + // A name that sanitizes to nothing shows nothing: no attribute, and the + // bare operation as the span name — never the unsanitized bytes. + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_14", Name: "\u200b\u202e"}) + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_14"}) + spans = endedToolSpans(rec) + require.Len(t, spans, 4) + assert.Equal(t, "execute_tool", spans[3].Name) + assert.NotContains(t, toolSpanAttrs(spans[3]), attribute.Key("gen_ai.tool.name")) +} + +func TestRecordToolSpanOverflow_MarksAgentSpanOnlyWhenDropped(t *testing.T) { + rec := tracetest.NewSpanRecorder() + tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(rec)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + _, clean := tp.Tracer("test").Start(context.Background(), "agent") + recordToolSpanOverflow(clean, 0) + clean.End() + _, over := tp.Tracer("test").Start(context.Background(), "agent") + recordToolSpanOverflow(over, 3) + over.End() + + ended := rec.Ended() + require.Len(t, ended, 2) + assert.NotContains(t, toolSpanAttrs(tracetest.SpanStubFromReadOnlySpan(ended[0])), attribute.Key("fullsend.tool_spans.dropped")) + assert.Equal(t, int64(3), toolSpanAttrs(tracetest.SpanStubFromReadOnlySpan(ended[1]))["fullsend.tool_spans.dropped"].AsInt64()) +} + +func TestToolSpanTracker_NilIsInert(t *testing.T) { + var tr *toolSpanTracker + assert.NotPanics(t, func() { + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_08", Name: "Bash"}) + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_08"}) + tr.Finish() + }) +} + +func TestIterationEventHandler_RendersThenCollectsThenTracks(t *testing.T) { + tr, rec, _ := toolSpanFixture(t) + c := newContentCollector(4096) + var rendered []agentruntime.AgentEvent + handler := iterationEventHandler(func(e agentruntime.AgentEvent) { rendered = append(rendered, e) }, c, tr) + require.NotNil(t, handler) + + handler(agentruntime.ToolUseEvent{ID: "toolu_09", Name: "Bash", Summary: "ls"}) + handler(agentruntime.ToolResultEvent{ID: "toolu_09", Result: "ok"}) + + assert.Len(t, rendered, 2, "the console renderer sees every event") + res := c.Result("stop") + assert.Contains(t, res.OutputMessages, `"tool_call_response"`, "the collector still receives every event") + assert.Len(t, endedToolSpans(rec), 1, "the tracker receives every event") +} + +func TestIterationEventHandler_NilCollectorAndTrackerStillRender(t *testing.T) { + // Tool spans are Level 1 metadata, so OnEvent is always set; supplying + // it replaces the runtime's default renderer, which must therefore be + // called here whatever else is off. + var rendered []agentruntime.AgentEvent + handler := iterationEventHandler(func(e agentruntime.AgentEvent) { rendered = append(rendered, e) }, nil, nil) + require.NotNil(t, handler) + handler(agentruntime.TextEvent{Text: "hello"}) + handler(agentruntime.ToolUseEvent{ID: "toolu_10", Name: "Bash"}) + assert.Len(t, rendered, 2) +} + +func TestToolSpans_EndToEndFileSink(t *testing.T) { + t.Setenv("OTEL_SDK_DISABLED", "") + t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "") + t.Setenv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", "") + t.Setenv("OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT", "") + t.Setenv("OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT", "") + t.Setenv(telemetry.ContentCaptureEnvVar, "") + + dir := t.TempDir() + tracer, cleanup := telemetry.Setup(dir, "test") + agentCtx, agentSpan := tracer.Start(context.Background(), "agent") + tr := newToolSpanTracker(tracer, agentCtx) + tr.Handle(agentruntime.ToolUseEvent{ID: "toolu_11", Name: "Grep", Summary: "TODO"}) + tr.Handle(agentruntime.ToolResultEvent{ID: "toolu_11", Result: "3 hits"}) + tr.Finish() + agentSpan.End() + cleanup(context.Background()) + + raw, err := os.ReadFile(filepath.Join(dir, telemetry.TelemetryFile)) + require.NoError(t, err) + content := string(raw) + assert.Equal(t, 1, strings.Count(content, `"execute_tool Grep"`)) + parent, err := json.Marshal(agentSpan.SpanContext().SpanID().String()) + require.NoError(t, err) + assert.Contains(t, content, `"parentSpanId":`+string(parent), + "the execute_tool span must be parented under the agent span in the file sink") + assert.NotContains(t, content, "3 hits", "tool spans carry no content, gate off or on") +} diff --git a/internal/runtime/claude_progress.go b/internal/runtime/claude_progress.go index 72142d01bf..b850e2a2d3 100644 --- a/internal/runtime/claude_progress.go +++ b/internal/runtime/claude_progress.go @@ -39,6 +39,7 @@ type innerEvent struct { type contentBlock struct { Type string `json:"type"` + ID string `json:"id"` Name string `json:"name"` } @@ -66,6 +67,28 @@ type assistantMessage struct { } `json:"message"` } +// userMessage contains tool_result blocks from user messages. As with +// assistantMessage, Claude Code's stream-json nests the content array +// under "message"; older/flat shapes put content at the top level. We +// accept both. +type userMessage struct { + Type string `json:"type"` + Content json.RawMessage `json:"content"` + Message struct { + Content json.RawMessage `json:"content"` + } `json:"message"` +} + +// userContentItem is one content block within a user message. Only +// tool_result blocks are consumed; the block's content arrives either as +// a plain string or as an array of text blocks. +type userContentItem struct { + Type string `json:"type"` + ToolUseID string `json:"tool_use_id"` + Content json.RawMessage `json:"content"` + IsError bool `json:"is_error"` +} + // systemEvent is Claude Code's initial "system"/"init" event, which carries the // resolved model name. The result event does not include the model. type systemEvent struct { @@ -81,6 +104,7 @@ type systemEvent struct { type contentItem struct { Type string `json:"type"` + ID string `json:"id"` Name string `json:"name"` Text string `json:"text"` Thinking string `json:"thinking"` @@ -114,6 +138,7 @@ func parseClaudeStream(r io.Reader, onEvent func(AgentEvent)) error { var ( seenStreamEvent bool currentToolName string + currentToolID string toolInputJSON strings.Builder // per-message token tracking for throttled TokensEvent totalInput int @@ -162,6 +187,10 @@ func parseClaudeStream(r io.Reader, onEvent func(AgentEvent)) error { return err } if isPrefix { + // Lines beyond streamBufSize are skipped whole. For user + // lines this loses any tool_result they carry (e.g. results + // holding base64 image blocks) — the event is never emitted + // and content capture cannot mark the loss. for isPrefix && err == nil { _, isPrefix, err = br.ReadLine() } @@ -216,6 +245,13 @@ func parseClaudeStream(r io.Reader, onEvent func(AgentEvent)) error { } if cb.Type == "tool_use" || cb.Type == "server_tool_use" { currentToolName = cb.Name + // A server-side tool's result arrives inside the assistant + // message, never as a user tool_result, so an id could never + // be matched: leave it empty. + currentToolID = "" + if cb.Type == "tool_use" { + currentToolID = cb.ID + } toolInputJSON.Reset() } @@ -238,10 +274,12 @@ func parseClaudeStream(r io.Reader, onEvent func(AgentEvent)) error { case "content_block_stop": if currentToolName != "" { onEvent(ToolUseEvent{ + ID: currentToolID, Name: currentToolName, Summary: extractSafeContext(currentToolName, json.RawMessage(toolInputJSON.String())), }) currentToolName = "" + currentToolID = "" toolInputJSON.Reset() } @@ -359,13 +397,73 @@ func parseClaudeStream(r io.Reader, onEvent func(AgentEvent)) error { } case "tool_use": onEvent(ToolUseEvent{ + ID: item.ID, Name: item.Name, Summary: extractSafeContext(item.Name, item.Input), }) } } + + case "user": + var msg userMessage + if err := json.Unmarshal(line, &msg); err != nil { + continue + } + content := msg.Message.Content + if len(content) == 0 { + content = msg.Content + } + var items []userContentItem + if err := json.Unmarshal(content, &items); err != nil { + continue + } + for _, item := range items { + if item.Type != "tool_result" { + continue + } + text, partial := toolResultText(item.Content) + onEvent(ToolResultEvent{ + ID: item.ToolUseID, + Result: text, + IsError: item.IsError, + Partial: partial, + }) + } + } + } +} + +// toolResultText flattens a tool_result block's content. The wire +// carries it either as a plain JSON string or as an array of content +// blocks, of which only text blocks contribute; they are joined with +// newlines. partial reports that non-text blocks (or undecodable +// content) were skipped — the returned text is a fragment of what the +// wire carried. +func toolResultText(raw json.RawMessage) (text string, partial bool) { + if len(raw) == 0 { + // An absent content key carried nothing to skip. + return "", false + } + var s string + if err := json.Unmarshal(raw, &s); err == nil { + return s, false + } + var blocks []struct { + Type string `json:"type"` + Text string `json:"text"` + } + if err := json.Unmarshal(raw, &blocks); err != nil { + return "", true + } + var texts []string + for _, b := range blocks { + if b.Type == "text" { + texts = append(texts, b.Text) + } else { + partial = true } } + return strings.Join(texts, "\n"), partial } // progressParser reads NDJSON from Claude Code's stream-json output and emits diff --git a/internal/runtime/claude_progress_test.go b/internal/runtime/claude_progress_test.go index 3aed0969ae..3842a09f64 100644 --- a/internal/runtime/claude_progress_test.go +++ b/internal/runtime/claude_progress_test.go @@ -787,6 +787,187 @@ func TestParseClaudeStreamToolUse(t *testing.T) { } } +func TestParseClaudeStreamToolUseCarriesID(t *testing.T) { + lines := []string{ + `{"type":"stream_event","event":{"type":"content_block_start","index":0,"content_block":{"type":"tool_use","id":"toolu_01AbCdEf","name":"Read"}}}`, + `{"type":"stream_event","event":{"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"{\"file_path\":\"/src/main.go\"}"}}}`, + `{"type":"stream_event","event":{"type":"content_block_stop","index":0}}`, + } + events := collectEvents(t, strings.Join(lines, "\n")) + var tools []ToolUseEvent + for _, e := range events { + if te, ok := e.(ToolUseEvent); ok { + tools = append(tools, te) + } + } + if len(tools) != 1 { + t.Fatalf("expected 1 tool event, got %d", len(tools)) + } + if tools[0].ID != "toolu_01AbCdEf" { + t.Errorf("expected tool ID toolu_01AbCdEf, got %q", tools[0].ID) + } +} + +func TestParseClaudeStreamAssistantFallbackToolUseCarriesID(t *testing.T) { + input := `{"type":"assistant","message":{"content":[{"type":"tool_use","id":"toolu_02XyZ","name":"Bash","input":{"command":"ls"}}]}}` + events := collectEvents(t, input) + var tools []ToolUseEvent + for _, e := range events { + if te, ok := e.(ToolUseEvent); ok { + tools = append(tools, te) + } + } + if len(tools) != 1 { + t.Fatalf("expected 1 tool event, got %d", len(tools)) + } + if tools[0].ID != "toolu_02XyZ" { + t.Errorf("expected tool ID toolu_02XyZ, got %q", tools[0].ID) + } +} + +func collectToolResults(t *testing.T, input string) []ToolResultEvent { + t.Helper() + var results []ToolResultEvent + for _, e := range collectEvents(t, input) { + if tr, ok := e.(ToolResultEvent); ok { + results = append(results, tr) + } + } + return results +} + +func TestParseClaudeStreamToolResultStringContent(t *testing.T) { + input := `{"type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01DULm","type":"tool_result","content":"main.go\nutil.go\n"}]}}` + results := collectToolResults(t, input) + if len(results) != 1 { + t.Fatalf("expected 1 tool result event, got %d", len(results)) + } + if results[0].ID != "toolu_01DULm" { + t.Errorf("expected ID toolu_01DULm, got %q", results[0].ID) + } + if results[0].Result != "main.go\nutil.go\n" { + t.Errorf("expected raw result text, got %q", results[0].Result) + } +} + +func TestParseClaudeStreamToolResultArrayContent(t *testing.T) { + input := `{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_03Arr","content":[{"type":"text","text":"first block"},{"type":"image","source":{"type":"base64","data":"aGk="}},{"type":"text","text":"second block"}]}]}}` + results := collectToolResults(t, input) + if len(results) != 1 { + t.Fatalf("expected 1 tool result event, got %d", len(results)) + } + if results[0].ID != "toolu_03Arr" { + t.Errorf("expected ID toolu_03Arr, got %q", results[0].ID) + } + if results[0].Result != "first block\nsecond block" { + t.Errorf("expected text blocks joined by newline with image skipped, got %q", results[0].Result) + } + if !results[0].Partial { + t.Errorf("skipping the image block loses content — the event must say so") + } +} + +func TestParseClaudeStreamToolResultPureTextNotPartial(t *testing.T) { + for name, input := range map[string]string{ + "string content": `{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_s","content":"plain"}]}}`, + "text-only array": `{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_a","content":[{"type":"text","text":"only text"}]}]}}`, + "absent content key": `{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_n","is_error":true}]}}`, + } { + results := collectToolResults(t, input) + if len(results) != 1 { + t.Fatalf("%s: expected 1 event, got %d", name, len(results)) + } + if results[0].Partial { + t.Errorf("%s: nothing was skipped; Partial must be false", name) + } + } +} + +func TestParseClaudeStreamToolResultFlatContent(t *testing.T) { + // Older/flat shape: content at the top level, no "message" nesting — + // the same dual-shape contract assistantMessage supports. + input := `{"type":"user","content":[{"type":"tool_result","tool_use_id":"toolu_04Flat","content":"flat shape"}]}` + results := collectToolResults(t, input) + if len(results) != 1 { + t.Fatalf("expected 1 tool result event, got %d", len(results)) + } + if results[0].ID != "toolu_04Flat" { + t.Errorf("expected ID toolu_04Flat, got %q", results[0].ID) + } + if results[0].Result != "flat shape" { + t.Errorf("expected flat-shape result, got %q", results[0].Result) + } +} + +func TestParseClaudeStreamUserTextContentIgnored(t *testing.T) { + // User messages can carry plain text (e.g. runner-composed feedback + // prompts); only tool_result blocks produce events. + input := `{"type":"user","message":{"role":"user","content":[{"type":"text","text":"please fix the validation errors"}]}}` + events := collectEvents(t, input) + if len(events) != 0 { + t.Fatalf("expected 0 events for text-only user message, got %d: %#v", len(events), events) + } +} + +func TestParseClaudeStreamUserMalformedShapesIgnored(t *testing.T) { + // Defensive branches: a user line whose message is not an object, and + // one whose content is a plain string (a real wire shape for user + // turns) carry no tool_result blocks — both are skipped without error. + lines := []string{ + `{"type":"user","message":5}`, + `{"type":"user","message":{"role":"user","content":"just text, not an array"}}`, + } + events := collectEvents(t, strings.Join(lines, "\n")) + if len(events) != 0 { + t.Fatalf("expected 0 events for malformed/plain user lines, got %d: %#v", len(events), events) + } +} + +func TestParseClaudeStreamToolResultNonTextContent(t *testing.T) { + // A tool_result whose content is neither a string nor a block array + // (e.g. an object) flattens to empty; the event still carries the ID. + input := `{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_08obj","content":{"unexpected":true}}]}}` + results := collectToolResults(t, input) + if len(results) != 1 { + t.Fatalf("expected 1 tool result event, got %d", len(results)) + } + if results[0].ID != "toolu_08obj" { + t.Errorf("expected ID toolu_08obj, got %q", results[0].ID) + } + if results[0].Result != "" { + t.Errorf("expected empty result for non-text content, got %q", results[0].Result) + } +} + +func TestParseClaudeStreamToolResultIsError(t *testing.T) { + // Failed tool calls carry is_error on the wire; the event surfaces it + // so consumers can tell an errored call from a successful one. + input := `{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_09err","content":"command not found","is_error":true}]}}` + results := collectToolResults(t, input) + if len(results) != 1 { + t.Fatalf("expected 1 tool result event, got %d", len(results)) + } + if !results[0].IsError { + t.Errorf("expected IsError=true for an is_error tool_result") + } +} + +func TestParseClaudeStreamToolResultEmptyContent(t *testing.T) { + // A tool_result with empty content still marks completion; the event + // is emitted with its ID and an empty Result. + input := `{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_05Empty","content":""}]}}` + results := collectToolResults(t, input) + if len(results) != 1 { + t.Fatalf("expected 1 tool result event, got %d", len(results)) + } + if results[0].ID != "toolu_05Empty" { + t.Errorf("expected ID toolu_05Empty, got %q", results[0].ID) + } + if results[0].Result != "" { + t.Errorf("expected empty result, got %q", results[0].Result) + } +} + func TestParseClaudeStreamUnknownToolShowsNameNoContext(t *testing.T) { lines := []string{ `{"type":"stream_event","event":{"type":"content_block_start","index":0,"content_block":{"type":"tool_use","name":"Skill"}}}`, @@ -1165,3 +1346,33 @@ func TestParseClaudeStreamFinalTokensEventOnCancel(t *testing.T) { t.Errorf("expected 500 output tokens, got %d", tokens[0].OutputTokens) } } + +func TestParseClaudeStream_ServerToolUseCarriesNoID(t *testing.T) { + // A server-side tool's result arrives inside the assistant message, never + // as a user tool_result, so the call must not carry an id that a result + // could match — an id-less ToolUseEvent gets no execute_tool span. + lines := []string{ + `{"type":"stream_event","event":{"type":"content_block_start","index":0,"content_block":{"type":"server_tool_use","id":"srvtoolu_01","name":"web_search"}}}`, + `{"type":"stream_event","event":{"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"{\"query\":\"otel\"}"}}}`, + `{"type":"stream_event","event":{"type":"content_block_stop","index":0}}`, + `{"type":"stream_event","event":{"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_01","name":"Read"}}}`, + `{"type":"stream_event","event":{"type":"content_block_stop","index":1}}`, + } + var uses []ToolUseEvent + if err := parseClaudeStream(strings.NewReader(strings.Join(lines, "\n")), func(e AgentEvent) { + if u, ok := e.(ToolUseEvent); ok { + uses = append(uses, u) + } + }); err != nil { + t.Fatalf("parseClaudeStream: %v", err) + } + if len(uses) != 2 { + t.Fatalf("expected 2 tool-use events, got %d: %+v", len(uses), uses) + } + if uses[0].Name != "web_search" || uses[0].ID != "" { + t.Errorf("server_tool_use must keep its name and carry no id, got %+v", uses[0]) + } + if uses[1].Name != "Read" || uses[1].ID != "toolu_01" { + t.Errorf("client tool_use must keep its id, got %+v", uses[1]) + } +} diff --git a/internal/runtime/event.go b/internal/runtime/event.go index d03d4f683c..eeb9bd1465 100644 --- a/internal/runtime/event.go +++ b/internal/runtime/event.go @@ -33,16 +33,36 @@ type TextEvent struct { func (TextEvent) agentEvent() {} // ToolUseEvent is emitted when a tool invocation completes. +// ID is the tool call identifier from the runtime stream; it is empty +// for runtimes whose wire format does not carry one. // Name is the raw tool name from the runtime stream. // Summary is a one-line context string from extractSafeContext; it is // empty for tools not recognized by that function. type ToolUseEvent struct { + ID string Name string Summary string } func (ToolUseEvent) agentEvent() {} +// ToolResultEvent carries the result of a completed tool invocation. +// ID is the tool call identifier from the runtime stream (matches +// ToolUseEvent.ID); Result is the raw result text; IsError reports a +// failed call, as set on the wire; Partial reports that non-text +// content (for example image blocks) was skipped while flattening, so +// Result is a fragment of what the wire carried. Only the Claude +// runtime emits it today — see the runtime support matrix in +// docs/runtimes.md. +type ToolResultEvent struct { + ID string + Result string + IsError bool + Partial bool +} + +func (ToolResultEvent) agentEvent() {} + // TokensEvent carries incremental token usage counters. type TokensEvent struct { InputTokens int diff --git a/internal/runtime/event_test.go b/internal/runtime/event_test.go index 690d1c1ff5..d98182c12a 100644 --- a/internal/runtime/event_test.go +++ b/internal/runtime/event_test.go @@ -14,8 +14,9 @@ func TestAgentEventInterfaceSatisfied(t *testing.T) { ResultEvent{NumTurns: 5, TotalCostUSD: 0.42}, ErrorEvent{ErrorType: "overloaded", Message: "rate limited"}, RetryEvent{Attempt: 1, MaxRetries: 3, DelayMs: 1000, Error: "timeout"}, + ToolResultEvent{ID: "toolu_01abc", Result: "file contents"}, ) - if len(events) != 8 { - t.Errorf("expected 8 event types, got %d", len(events)) + if len(events) != 9 { + t.Errorf("expected 9 event types, got %d", len(events)) } } diff --git a/internal/runtime/pi_extension/fullsend-hooks.js b/internal/runtime/pi_extension/fullsend-hooks.js index 777162e583..5522adee89 100644 --- a/internal/runtime/pi_extension/fullsend-hooks.js +++ b/internal/runtime/pi_extension/fullsend-hooks.js @@ -11,6 +11,8 @@ // // Contract with the scripts (v1 and v2 — fullsend#6357): // stdin {"tool_name", "tool_input", "tool_result", "tool_response"} +// (no "cwd": the redact stage's checkout-scoped bare-JWT skip +// stays inert under pi, which masks as before) // stdout PreToolUse: exit != 0 or {"decision":"block","reason"} blocks. // PostToolUse: {"hookSpecificOutput":{"updatedToolOutput": }} // (v2) or {"tool_result": } (v1) replaces the result text; diff --git a/internal/runtime/renderer_test.go b/internal/runtime/renderer_test.go index 5caede006a..e83d23732f 100644 --- a/internal/runtime/renderer_test.go +++ b/internal/runtime/renderer_test.go @@ -55,6 +55,19 @@ func TestRendererToolUseEvent(t *testing.T) { } } +func TestRendererToolResultEventIgnored(t *testing.T) { + // Tool results feed the Level 3 content collector, not the console: + // the renderer intentionally produces no output for them. + var buf bytes.Buffer + r := newTestRenderer(&buf) + + r.Handle(ToolResultEvent{ID: "toolu_01abc", Result: "output text"}) + + if output := buf.String(); output != "" { + t.Errorf("expected no output for ToolResultEvent, got: %s", output) + } +} + func TestRendererToolUseEventCI(t *testing.T) { t.Setenv("GITHUB_ACTIONS", "true") diff --git a/internal/security/hooks/posttool_chain.py b/internal/security/hooks/posttool_chain.py index 30ec4a84b4..286392bc63 100644 --- a/internal/security/hooks/posttool_chain.py +++ b/internal/security/hooks/posttool_chain.py @@ -231,13 +231,14 @@ def _handle_failure(hook_input: dict[str, Any]) -> None: try: redact_mod = _load_stage("redact") if redact_mod is not None: - _, findings = redact_mod.redact_text(text) + skips = redact_mod.content_skips(hook_input) + _, findings = redact_mod.redact_text(text, skip=skips) # Same obfuscation cover as the success path: a fullwidth or # mark-split credential is only visible in detection form. seen = {(f["pattern"], f["masked"]) for f in findings} normalized = hook_io._detection_form(text) if normalized != text: - _, extra = redact_mod.redact_text(normalized) + _, extra = redact_mod.redact_text(normalized, skip=skips) findings += [f for f in extra if (f["pattern"], f["masked"]) not in seen] if findings: for f in findings: @@ -397,19 +398,20 @@ def _sanitize(text: str) -> str: try: redact_mod = _load_stage("redact") if redact_mod is not None: + skips = redact_mod.content_skips(hook_input) def _redact(text: str) -> str: if not text: return text try: - cleaned, findings = redact_mod.redact_text(text) + cleaned, findings = redact_mod.redact_text(text, skip=skips) normalized = hook_io.nfkc(text) if normalized != text: # Fullwidth/compatibility obfuscation: the unicode # stage keeps such text, so scan a normalized copy # and, only when that finds more, emit the # normalized+redacted field instead. - cleaned_n, findings_n = redact_mod.redact_text(normalized) + cleaned_n, findings_n = redact_mod.redact_text(normalized, skip=skips) seen = {(f["pattern"], f["masked"]) for f in findings} if len(findings_n) > len(findings) or any( (f["pattern"], f["masked"]) not in seen for f in findings_n diff --git a/internal/security/hooks/posttool_chain_test.py b/internal/security/hooks/posttool_chain_test.py index bdcf662212..7308f89933 100644 --- a/internal/security/hooks/posttool_chain_test.py +++ b/internal/security/hooks/posttool_chain_test.py @@ -3,11 +3,13 @@ from __future__ import annotations +import contextlib import io import json import os import subprocess import sys +import tempfile import unittest from pathlib import Path from unittest import mock @@ -18,6 +20,14 @@ CHAIN_HOOK = str(HOOKS_DIR / "posttool_chain.py") PLAIN_PAT = "ghp_FAKEtesttoken000000000000000000000000" +# Segments concatenated so the fixture does not trip gitleaks. +JWT = ( + "eyJhbGciOiJSUzI1NiJ9" + + "." + + "eyJzdWIiOiIxMjM0NTY3ODkwIn0" + + "." + + "dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk" +) def obfuscate_with_char(text: str, char: str) -> str: @@ -43,10 +53,13 @@ def run_hook( env_extra: dict[str, str] | None = None, tool_name: str = "Read", tool_input: dict | None = None, + cwd: str | None = None, ) -> tuple[int, str, str]: body: dict = {"tool_name": tool_name, key: payload} if tool_input is not None: body["tool_input"] = tool_input + if cwd is not None: + body["cwd"] = cwd env = {k: v for k, v in os.environ.items() if k != "FULLSEND_CANARY_TOKEN"} env.update(env_extra or {}) proc = subprocess.run( @@ -276,7 +289,7 @@ def fake_load(token: str): mod = real_load(token) if token == "redact" and mod is not None: - def boom(_text: str): + def boom(_text: str, **_kw): raise RuntimeError("boom") mod.redact_text = boom @@ -569,6 +582,17 @@ def read_payload(content: str) -> dict: } +@contextlib.contextmanager +def checkout(): + """A checkout (with .git) beside a runner token file, as the sandbox + lays them out; yields (repo, token_path).""" + with tempfile.TemporaryDirectory() as tmp: + ws = os.path.realpath(tmp) + repo = os.path.join(ws, "repo") + os.makedirs(os.path.join(repo, ".git")) + yield repo, os.path.join(ws, ".gcp-oidc-token") + + class TestContentPreservedAndRewriteNotes(unittest.TestCase): """What the agent reads must be what is on disk unless a control fired, and when one fires the agent is told (additionalContext).""" @@ -593,6 +617,33 @@ def test_cjk_read_not_rewritten(self): self.assertEqual(rc, 0) self.assertNotIn("updatedToolOutput", stdout) + def test_jwt_fixture_in_read_not_rewritten(self): + with checkout() as (repo, _): + rc, stdout, _ = run_hook( + CHAIN_HOOK, + read_payload(f'\t{{name: "valid", input: "{JWT}"}},\n'), + key="tool_response", + tool_input={"file_path": f"{repo}/x_test.go"}, + cwd=repo, + ) + self.assertEqual(rc, 0) + self.assertEqual(stdout, "") + + def test_jwt_in_runner_token_file_read_still_masked(self): + # The runner's OIDC token file sits beside the checkout, not in it. + with checkout() as (repo, token): + rc, stdout, _ = run_hook( + CHAIN_HOOK, + read_payload(f"{JWT}\n"), + key="tool_response", + tool_input={"file_path": token}, + cwd=repo, + ) + self.assertEqual(rc, 0) + out = json.loads(stdout) + self.assertNotIn(JWT, json.dumps(out["hookSpecificOutput"]["updatedToolOutput"])) + self.assertIn("masked", out["hookSpecificOutput"]["additionalContext"]) + def test_redaction_adds_additional_context(self): rc, stdout, _ = run_hook( CHAIN_HOOK, @@ -693,6 +744,22 @@ def test_failed_call_output_is_never_rewritten(self): self.assertNotIn("updatedToolOutput", stdout) self.assertNotIn("tool_result", stdout) + def test_jwt_in_failed_read_not_flagged(self): + body = self._body(f'parse error near: tok := "{JWT}"') + body["tool_name"] = "Read" + with checkout() as (repo, _): + body["tool_input"] = {"file_path": f"{repo}/f"} + body["cwd"] = repo + rc, stdout, _ = run_raw(body) + self.assertEqual(rc, 0) + self.assertEqual(stdout, "") + + def test_jwt_in_failed_bash_call_is_flagged(self): + rc, stdout, _ = run_raw(self._body(f"Exit code 1\nexchange with {JWT} failed")) + self.assertEqual(rc, 0) + out = json.loads(stdout) + self.assertIn("credential-like", out["hookSpecificOutput"]["additionalContext"]) + def test_credential_in_a_failed_call_is_detected_and_flagged(self): aws = "AKIA" + "IOSFODNN7EXAMPLE" rc, stdout, _ = run_raw(self._body(f"Exit code 1\nusing {aws} failed")) diff --git a/internal/security/hooks/secret_redact_posttool.py b/internal/security/hooks/secret_redact_posttool.py index 0575ed43ab..7d380d77b4 100644 --- a/internal/security/hooks/secret_redact_posttool.py +++ b/internal/security/hooks/secret_redact_posttool.py @@ -1,13 +1,16 @@ #!/usr/bin/env python3 """Claude Code PostToolUse hook for secret redaction. -Intercepts tool results (Bash, WebFetch, Read) and redacts secrets +Intercepts tool results (Bash, WebFetch, Read, ...) and redacts secrets before they enter the LLM context window. This prevents the agent from -seeing or leaking credentials in subsequent output. - -Protocol: reads JSON from stdin (``tool_response`` preferred, ``tool_result`` -fallback). Writes ``hookSpecificOutput.updatedToolOutput`` (and ``tool_result``) -when secrets are found. Exit code 0 always (never blocks). +seeing or leaking credentials in subsequent output. The bare-JWT pattern +is skipped when a file-content tool is called with a path inside the +checkout — see ``content_skips``. + +Protocol: reads JSON from stdin (``tool_name``, ``tool_input``, ``cwd`` and +``tool_response`` preferred, ``tool_result`` fallback). Writes +``hookSpecificOutput.updatedToolOutput`` (and ``tool_result``) when secrets +are found. Exit code 0 always (never blocks). """ from __future__ import annotations @@ -26,6 +29,11 @@ _PREFIX_PATTERNS: list[tuple[str, re.Pattern]] = [ ("openai_key", re.compile(r"sk-(?:proj-)?[A-Za-z0-9_-]{20,}")), + # Before github_pat: dots in the class cover the 2026 JWT-wrapped + # installation-token format whole (mirrors the Go redactor's + # github_server_token); the combined pattern below would otherwise + # stop at the first dot and leave payload+signature in the clear. + ("github_server_token", re.compile(r"ghs_[A-Za-z0-9_.\-]{36,}")), ("github_pat", re.compile(r"(?:ghp|github_pat|gho|ghu|ghs|ghr)_[A-Za-z0-9_]{16,}")), ("slack_token", re.compile(r"xox[baprs]-[A-Za-z0-9\-]{10,}")), ("google_api_key", re.compile(r"AIza[A-Za-z0-9_-]{35}")), @@ -38,7 +46,14 @@ ("stripe_key", re.compile(r"(?:sk|pk|rk)_(?:live|test)_[A-Za-z0-9]{10,}")), ("sendgrid_key", re.compile(r"SG\.[A-Za-z0-9_-]{22}\.[A-Za-z0-9_-]{43}")), ("gitlab_pat", re.compile(r"gl(?:pat|rt|ptt|dt|ft|soat|cs)-[A-Za-z0-9_-]{20,}")), - ("google_oauth_token", re.compile(r"ya29\.[A-Za-z0-9_-]{30,}")), + # Mirrors the Go redactor: the literal c. alternative covers + # service-account tokens, whose one-char first segment would + # otherwise defeat the length quantifier. + ("google_oauth_token", re.compile(r"ya29\.(?:c\.)?[A-Za-z0-9_-]{20,}")), + # Bare three-segment JWTs (and OIDC/WIF STS tokens) carry no + # surrounding context for the structural patterns to anchor on. + # Skipped for file content inside the checkout — see content_skips. + ("jwt", re.compile(r"eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}")), ("aws_sts_key", re.compile(r"ASIA[A-Z0-9]{16}")), ("hf_token", re.compile(r"hf_[A-Za-z0-9]{20,}")), ("npm_token", re.compile(r"npm_[A-Za-z0-9]{36}")), @@ -55,6 +70,97 @@ ), ] +# Tools whose response is file content, with the tool_input key naming the +# path they read. A bare JWT has no fixture-shaped escape — a jwt.io example +# in a test file is byte-for-byte a valid token — so masking it there +# rewrites what the agent reads and edits against (the failure mode +# described under the structural patterns below) without catching a live +# credential: the STS/OIDC/WIF tokens the pattern exists for surface in +# Bash, WebFetch and MCP output, which stays covered. The skip is limited to +# paths inside the checkout because the runner delivers its own OIDC token +# file to the sandbox workspace, beside the checkout, and a Read or Grep of +# that must still mask. The checkout is found from the hook input's cwd: +# the runtime starts there, and cwd follows the agent's persisted cd, so +# the nearest ancestor holding .git is the root, and no .git means no +# skip (a cd through a symlink out of the checkout resolves to a +# directory without one). A cwd inside a submodule narrows the root to +# the submodule, so superproject fixtures mask again there — the safe +# direction. A path is normalized before it is resolved, because the +# runtime opens the normalized path while a bare realpath would follow a +# symlink before applying '..'; and any '..' segment refuses the skip +# outright, as no fixture needs one. A copy or hard link of an outside +# file into the checkout, or a .git planted above the checkout with cwd +# moved under it, is a deliberate agent action the hook does not defend +# against — the same class as a Bash transform of the token (base64, +# cut), which it never caught. An adapter that sends no cwd (pi) gets no +# skip, i.e. masks as before. Adapters translate tool names to Claude's +# vocabulary before the chain runs, so the names here apply everywhere. +_TOOL_PATH_KEY = { + "Read": "file_path", + "Edit": "file_path", + "MultiEdit": "file_path", + "Write": "file_path", + "NotebookEdit": "notebook_path", + "NotebookRead": "notebook_path", + "Grep": "path", +} +_CHECKOUT_SKIPS = frozenset({"jwt"}) + + +def _checkout_root(cwd: str) -> str | None: + """The nearest ancestor of cwd (itself included) holding a .git entry, + or None when there is none.""" + probe = cwd + while True: + if os.path.lexists(os.path.join(probe, ".git")): + return probe + parent = os.path.dirname(probe) + if parent == probe: + return None + probe = parent + + +def content_skips(hook_input: dict) -> frozenset[str]: + """Patterns redact_text skips for this call: the bare-JWT pattern when a + file-content tool is called with a path inside the checkout, nothing + otherwise. + Anything malformed or unresolvable means no skip, never an exception.""" + tool_name = hook_input.get("tool_name") + cwd = hook_input.get("cwd") + if not isinstance(tool_name, str) or tool_name not in _TOOL_PATH_KEY: + return frozenset() + if not isinstance(cwd, str) or not os.path.isabs(cwd): + return frozenset() + tool_input = hook_input.get("tool_input") + if isinstance(tool_input, str): + try: + tool_input = json.loads(tool_input) + except (json.JSONDecodeError, TypeError): + return frozenset() + if not isinstance(tool_input, dict): + return frozenset() + path = tool_input.get(_TOOL_PATH_KEY[tool_name]) + if path is None: + # Grep searches cwd by default; every other tool here names its path. + if tool_name != "Grep": + return frozenset() + path = cwd + if not isinstance(path, str) or not path: + return frozenset() + if path.startswith("~") or ".." in path.split("/"): + return frozenset() + try: + base = os.path.realpath(cwd) + root = _checkout_root(base) + if root is None: + return frozenset() + target = os.path.realpath(os.path.normpath(os.path.join(base, path))) + inside = os.path.commonpath([root, target]) == root + except (ValueError, OSError): + return frozenset() + return _CHECKOUT_SKIPS if inside else frozenset() + + # --- Structural patterns --- # # env_secret / json_secret match *shape*: a secret-bearing name assigned a @@ -433,11 +539,13 @@ def mask_token(token: str) -> str: return f"{token[:4]}..." -def redact_text(text: str) -> tuple[str, list[dict]]: +def redact_text(text: str, *, skip: frozenset[str] = frozenset()) -> tuple[str, list[dict]]: findings: list[dict] = [] result = text for name, pattern in _PREFIX_PATTERNS: + if name in skip: + continue for match in pattern.finditer(result): token = match.group(0) masked = mask_token(token) @@ -476,13 +584,14 @@ def main(): sys.exit(0) original = hook_io.payload(hook_input) + skips = content_skips(hook_input) findings: list[dict] = [] def _redact_field(text: str) -> str: if not text: return text try: - redacted, field_findings = redact_text(text) + redacted, field_findings = redact_text(text, skip=skips) except Exception as e: log_finding("redaction_error", f"Redaction failed (passing original): {e}") return text diff --git a/internal/security/hooks/secret_redact_posttool_test.py b/internal/security/hooks/secret_redact_posttool_test.py index aa7f4f7bd2..02c221eadf 100644 --- a/internal/security/hooks/secret_redact_posttool_test.py +++ b/internal/security/hooks/secret_redact_posttool_test.py @@ -2,6 +2,7 @@ """Unit tests for secret_redact_posttool.py hook.""" import json +import os import subprocess import sys import unittest @@ -319,6 +320,45 @@ def test_real_vendor_tokens_with_known_prefixes_redacted(self): for leaked in (glpat, glrt, ya29, asia): self.assertNotIn(leaked, text) + def test_service_account_token_redacted(self): + # WIF-provisioned runs mint ya29.c. tokens; the one-char c + # segment must not defeat the match (mirrors the Go redactor). + ya29c = "ya29.c." + "b0Aaekm1K8sVq9dNfP2xJ3hT7wY5uZ4rQ6mE8oL1iC0aS" + _, stdout, _ = run_hook(f"got token {ya29c} from the metadata server\n") + self.assertTrue(stdout) + self.assertNotIn(ya29c, json.loads(stdout)["tool_result"]) + + def test_ghs_wrapped_jwt_fully_redacted(self): + # GitHub's 2026 installation-token format wraps a JWT: the whole + # token must mask, not just the ghs_ prefix and header segment + # (mirrors the Go redactor's github_server_token). + token = ( + "ghs_12345_" + + "eyJhbGciOiJSUzI1NiJ9" + + "." + + "eyJzdWIiOiIxMjM0NTY3ODkwIn0" + + "." + + "dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk" + ) + _, stdout, _ = run_hook(f"Token: {token}\n") + self.assertTrue(stdout) + text = json.loads(stdout)["tool_result"] + self.assertNotIn("eyJzdWIiOiIxMjM0NTY3ODkwIn0", text) + self.assertNotIn("dBjftJeZ4CVP", text) + + def test_bare_jwt_redacted(self): + # Segments concatenated so the fixture does not trip gitleaks. + jwt = ( + "eyJhbGciOiJSUzI1NiJ9" + + "." + + "eyJzdWIiOiIxMjM0NTY3ODkwIn0" + + "." + + "dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk" + ) + _, stdout, _ = run_hook(f"curl output: {jwt}\n") + self.assertTrue(stdout) + self.assertNotIn(jwt, json.loads(stdout)["tool_result"]) + def test_short_prefixed_fakes_still_untouched(self): _, stdout, _ = run_hook( 'Token: "ghs_maskable"\n"token": "glpat-new"\nGitToken: "ghp_test123"\n' @@ -353,6 +393,251 @@ def test_aws_pagination_tokens_untouched(self): self.assertEqual(stdout, "") +class TestBareJwtToolScope(unittest.TestCase): + """A bare JWT has no fixture-shaped escape, so the pattern skips file + content inside the checkout: a committed jwt.io example is not the live + STS/OIDC token the pattern exists to catch, and masking it hands the + agent text that is not in the file. Anything outside the checkout — the + runner's own token file included — still masks. The layout mirrors the + sandbox: a checkout with .git beside the runner's token file.""" + + # Segments concatenated so the fixture does not trip gitleaks. + JWT = ( + "eyJhbGciOiJSUzI1NiJ9" + + "." + + "eyJzdWIiOiIxMjM0NTY3ODkwIn0" + + "." + + "dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk" + ) + + def setUp(self): + import tempfile + + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.ws = os.path.realpath(tmp.name) + self.repo = os.path.join(self.ws, "repo") + os.makedirs(os.path.join(self.repo, ".git")) + os.makedirs(os.path.join(self.repo, "pkg")) + os.makedirs(os.path.join(self.repo, "internal", "cli")) + self.fixture = os.path.join(self.repo, "pkg", "x_test.go") + Path(self.fixture).write_text("x") + self.token = os.path.join(self.ws, ".gcp-oidc-token") + Path(self.token).write_text("x") + # The outside-the-checkout cases need nothing above the tempdir to + # hold .git; a TMPDIR inside a checkout would fail them spuriously. + import secret_redact_posttool as sr + + self.assertIsNone(sr._checkout_root(self.ws), f"{self.ws} lies under a checkout") + + def _hook_input(self, tool_name, path=None, *, cwd=None, tool_input=None) -> dict: + body: dict = {"tool_name": tool_name, "cwd": self.repo if cwd is None else cwd} + if tool_input is not None: + body["tool_input"] = tool_input + elif path is not None: + body["tool_input"] = {"file_path": path} + return body + + def test_skip_for_checkout_paths(self): + import secret_redact_posttool as sr + + for tool in ("Read", "Edit", "MultiEdit", "Write"): + with self.subTest(tool=tool): + self.assertEqual(sr.content_skips(self._hook_input(tool, self.fixture)), {"jwt"}) + for tool in ("NotebookEdit", "NotebookRead"): + with self.subTest(tool=tool): + body = self._hook_input(tool, tool_input={"notebook_path": self.fixture}) + self.assertEqual(sr.content_skips(body), {"jwt"}) + pkg = os.path.join(self.repo, "pkg") + grep = self._hook_input("Grep", tool_input={"pattern": "eyJ", "path": pkg}) + self.assertEqual(sr.content_skips(grep), {"jwt"}) + self.assertEqual(sr.content_skips(self._hook_input("Read", "pkg/x_test.go")), {"jwt"}) + grep = self._hook_input("Grep", tool_input={"pattern": "eyJ"}) + self.assertEqual(sr.content_skips(grep), {"jwt"}) + + def test_no_skip_outside_checkout(self): + import secret_redact_posttool as sr + + plain = os.path.join(self.ws, "plain") + os.makedirs(plain) + Path(os.path.join(plain, "f.go")).write_text("x") + cases = [ + self._hook_input("Read", self.token), + self._hook_input("Grep", tool_input={"pattern": "eyJ", "path": self.ws}), + # No .git anywhere above cwd: no checkout, no skip. + self._hook_input("Read", os.path.join(plain, "f.go"), cwd=plain), + self._hook_input("Bash", tool_input={"command": "cat x"}), + self._hook_input("WebFetch", tool_input={"url": "https://x"}), + ] + for body in cases: + with self.subTest(body=body): + self.assertEqual(sr.content_skips(body), frozenset()) + + def test_traversal_and_tilde_never_skip(self): + # The runtime opens the normalized path; realpath would follow a + # committed symlink first, so link/../../token lands outside while + # looking inside. No fixture needs '..', so refuse it outright. + import secret_redact_posttool as sr + + dotted = os.path.join(self.repo, "pkg", "..", "pkg", "x_test.go") + for path in (dotted, "pkg/../pkg/x_test.go", "~/x_test.go"): + with self.subTest(path=path): + self.assertEqual(sr.content_skips(self._hook_input("Read", path)), frozenset()) + + def test_symlink_out_of_checkout_is_resolved(self): + import secret_redact_posttool as sr + + link = os.path.join(self.repo, "token.txt") + os.symlink(self.token, link) + self.assertEqual(sr.content_skips(self._hook_input("Read", link)), frozenset()) + deep = os.path.join(self.repo, "link") + os.symlink("a/b", deep) + escape = f"{deep}/../../.gcp-oidc-token" + self.assertEqual(sr.content_skips(self._hook_input("Read", escape)), frozenset()) + self.assertEqual(sr.content_skips(self._hook_input("Read", self.fixture)), {"jwt"}) + + def test_symlinked_cwd_never_reaches_outside(self): + # A cd through a symlink resolves to the real directory: outward, + # nothing there holds .git, so no skip whatever the path; inward, the + # checkout is found. + import secret_redact_posttool as sr + + ws_link = os.path.join(self.repo, "ws") + os.symlink(self.ws, ws_link) + cases = [ + self._hook_input("Read", self.token, cwd=ws_link), + self._hook_input("Read", ".gcp-oidc-token", cwd=ws_link), + self._hook_input("Grep", cwd=ws_link, tool_input={"pattern": "eyJ"}), + ] + up = os.path.join(self.repo, "up") + os.symlink("..", up) + cases.append(self._hook_input("Read", self.token, cwd=up)) + for body in cases: + with self.subTest(body=body): + self.assertEqual(sr.content_skips(body), frozenset()) + inward = os.path.join(self.ws, "inward") + os.symlink(os.path.join(self.repo, "internal"), inward) + self.assertEqual( + sr.content_skips(self._hook_input("Read", self.fixture, cwd=inward)), {"jwt"} + ) + + def test_root_is_nearest_git_ancestor_of_cwd(self): + # cwd follows the agent's persisted cd; the checkout is still the root. + import secret_redact_posttool as sr + + cwd = os.path.join(self.repo, "internal", "cli") + self.assertEqual(sr.content_skips(self._hook_input("Read", self.fixture, cwd=cwd)), {"jwt"}) + self.assertEqual( + sr.content_skips(self._hook_input("Read", self.token, cwd=cwd)), frozenset() + ) + grep = self._hook_input("Grep", cwd=cwd, tool_input={"pattern": "eyJ", "path": self.repo}) + self.assertEqual(sr.content_skips(grep), {"jwt"}) + # A submodule cwd narrows the root to the submodule: its own files + # skip, superproject fixtures mask again — the safe direction. + dep = os.path.join(self.repo, "vendor", "dep") + os.makedirs(dep) + Path(os.path.join(dep, ".git")).write_text("gitdir: ../../.git/modules/dep\n") + dep_file = os.path.join(dep, "x.go") + Path(dep_file).write_text("x") + self.assertEqual(sr.content_skips(self._hook_input("Read", dep_file, cwd=dep)), {"jwt"}) + self.assertEqual( + sr.content_skips(self._hook_input("Read", self.fixture, cwd=dep)), frozenset() + ) + + def test_grep_uses_its_own_path_key(self): + import secret_redact_posttool as sr + + stray = {"pattern": "eyJ", "path": self.ws, "file_path": self.fixture} + self.assertEqual(sr.content_skips(self._hook_input("Grep", tool_input=stray)), frozenset()) + self.assertEqual(sr.content_skips(self._hook_input("Grep", tool_input={})), {"jwt"}) + self.assertEqual(sr.content_skips({"tool_name": "Grep", "cwd": self.repo}), frozenset()) + + def test_malformed_input_means_mask_not_error(self): + import secret_redact_posttool as sr + + read = {"file_path": self.fixture} + cases = [ + {"tool_name": ["Read"], "cwd": self.repo, "tool_input": read}, + {"tool_name": "Read", "tool_input": read}, + {"tool_name": "Read", "cwd": None, "tool_input": read}, + {"tool_name": "Read", "cwd": "repo", "tool_input": read}, + {"tool_name": "Read", "cwd": self.repo}, + {"tool_name": "Read", "cwd": self.repo, "tool_input": "not json"}, + {"tool_name": "Read", "cwd": self.repo, "tool_input": {"file_path": [self.fixture]}}, + {"tool_name": "Read", "cwd": self.repo, "tool_input": {"file_path": ""}}, + { + "tool_name": "Read", + "cwd": self.repo, + "tool_input": {"file_path": self.fixture + "\x00"}, + }, + ] + for body in cases: + with self.subTest(body=body): + self.assertEqual(sr.content_skips(body), frozenset()) + text, findings = sr.redact_text(f"tok {self.JWT}\n", skip=sr.content_skips(cases[0])) + self.assertNotIn(self.JWT, text) + self.assertEqual([f["pattern"] for f in findings], ["jwt"]) + + def test_tool_input_as_json_string(self): + import secret_redact_posttool as sr + + body = { + "tool_name": "Read", + "cwd": self.repo, + "tool_input": json.dumps({"file_path": self.fixture}), + } + self.assertEqual(sr.content_skips(body), {"jwt"}) + + def test_skip_is_jwt_only(self): + import secret_redact_posttool as sr + + ya29c = "ya29.c." + "b0Aaekm1K8sVq9dNfP2xJ3hT7wY5uZ4rQ6mE8oL1iC0aS" + ghs = "ghs_12345_" + self.JWT + text, findings = sr.redact_text( + f"a = {ya29c}\nb = {ghs}\nc = {self.JWT}\n", skip=frozenset({"jwt"}) + ) + self.assertNotIn(ya29c, text) + self.assertNotIn(ghs, text) + self.assertIn(self.JWT, text) + self.assertEqual( + sorted(f["pattern"] for f in findings), ["github_server_token", "google_oauth_token"] + ) + + def test_named_assignment_still_masked_by_structural_pattern(self): + # The skip covers the context-free pattern only: an assignment whose + # name says it is a token is masked by env_secret on every tool, as + # before this pattern existed. + import secret_redact_posttool as sr + + text, findings = sr.redact_text(f'var testToken = "{self.JWT}"\n', skip=frozenset({"jwt"})) + self.assertNotIn(self.JWT, text) + self.assertEqual([f["pattern"] for f in findings], ["env_secret"]) + + def test_script_honours_checkout_scope(self): + def run(path: str, tool_result: str) -> str: + body = { + "tool_name": "Read", + "cwd": self.repo, + "tool_input": {"file_path": path}, + "tool_result": tool_result, + } + proc = subprocess.run( + [sys.executable, HOOK], + input=json.dumps(body), + capture_output=True, + text=True, + timeout=10, + ) + self.assertEqual(proc.returncode, 0, proc.stderr) + return proc.stdout + + fixture = f'\t{{name: "valid", input: "{self.JWT}"}},\n' + self.assertEqual(run(self.fixture, fixture), "") + stdout = run(self.token, f"{self.JWT}\n") + self.assertTrue(stdout) + self.assertNotIn(self.JWT, json.loads(stdout)["tool_result"]) + + class TestPaginationVsSecretNames(unittest.TestCase): def test_pagination_words_only_veto_token_names(self): import secret_redact_posttool as sr diff --git a/internal/security/redactor.go b/internal/security/redactor.go index eb2d1dc12d..f6918e0028 100644 --- a/internal/security/redactor.go +++ b/internal/security/redactor.go @@ -207,6 +207,11 @@ func defaultPrefixPatterns() []secretPattern { {"github_refresh_token", `ghr_[a-zA-Z0-9_]{36,}`}, {"slack_token", `xox[baprs]-[a-zA-Z0-9-]{10,}`}, {"google_api_key", `AIza[a-zA-Z0-9_-]{35}`}, + // No dot in the class (like google_api_key above, a dot would run + // the match through punctuation into adjacent prose); the literal + // c. alternative covers service-account tokens, whose 1-char + // first segment would otherwise defeat the {20,} quantifier. + {"google_oauth_token", `ya29\.(?:c\.)?[a-zA-Z0-9_\-]{20,}`}, {"aws_access_key", `AKIA[A-Z0-9]{16}`}, {"stripe_live", `sk_live_[a-zA-Z0-9]{24,}`}, {"stripe_test", `sk_test_[a-zA-Z0-9]{24,}`}, @@ -217,6 +222,9 @@ func defaultPrefixPatterns() []secretPattern { {"gitlab_pat", `glpat-[a-zA-Z0-9_-]{20,}`}, {"vault_token", `hvs\.[a-zA-Z0-9_-]{24,}`}, {"age_secret_key", `AGE-SECRET-KEY-[A-Z0-9]{59}`}, + // Bare three-segment JWTs (and OIDC/WIF STS tokens) carry no + // surrounding context for the structural patterns to anchor on. + {"jwt", `eyJ[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}`}, } result := make([]secretPattern, len(patterns)) diff --git a/internal/security/scanner_test.go b/internal/security/scanner_test.go index 5f49ead20a..9eeae45c2c 100644 --- a/internal/security/scanner_test.go +++ b/internal/security/scanner_test.go @@ -140,6 +140,43 @@ func TestSecretRedactor(t *testing.T) { assert.True(t, hasFinding(result, "github_server_token")) }) + t.Run("google oauth access token redacted", func(t *testing.T) { + // Bare ya29. bearer tokens are what WIF-provisioned runs print + // from gcloud auth print-access-token — no surrounding context. + result := r.Scan("token: ya29.a0AfB_byDEMOtoken1234567890abcdefghij") + assert.False(t, result.Safe) + assert.NotContains(t, result.Sanitized, "a0AfB_byDEMO") + assert.True(t, hasFinding(result, "google_oauth_token")) + }) + + t.Run("google service-account token redacted", func(t *testing.T) { + // WIF-provisioned runs mint ya29.c. tokens; the 1-char c + // segment must not defeat the match. + result := r.Scan("got ya29.c.b0Aaekm1K8sVq9dNfP2xJ3hT7wY5uZ4rQ6mE8oL1iC0aS") + assert.False(t, result.Safe) + assert.NotContains(t, result.Sanitized, "b0Aaekm1K8sVq9dNfP2xJ3hT") + assert.True(t, hasFinding(result, "google_oauth_token")) + }) + + t.Run("google oauth token does not swallow adjacent prose", func(t *testing.T) { + // A dot inside the character class would run the match through + // sentence punctuation into the following word. + result := r.Scan("Use ya29.a0AfB_byDEMOtoken1234567890abcdefghij.Endpoint next") + assert.False(t, result.Safe) + assert.Contains(t, result.Sanitized, "Endpoint", + "the match must stop at the token, not consume dot-joined prose") + }) + + t.Run("bare JWT redacted", func(t *testing.T) { + // Segments concatenated so the fixture itself does not trip the + // gitleaks pre-commit hook. + jwt := "eyJhbGciOiJSUzI1NiJ9" + "." + "eyJzdWIiOiIxMjM0NTY3ODkwIn0" + "." + "dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk" + result := r.Scan("curl -H output: " + jwt) + assert.False(t, result.Safe) + assert.NotContains(t, result.Sanitized, "eyJzdWIiOiIxMjM0NTY3ODkwIn0") + assert.True(t, hasFinding(result, "jwt")) + }) + t.Run("openai key redacted", func(t *testing.T) { result := r.Scan("key=sk-proj-abc123def456ghi789jkl012mno345pqr678") assert.False(t, result.Safe)