feat(telemetry): capture tool results in Level 3 content and emit execute_tool spans

docs/ADRs/0050-distributed-tracing-instrumentation.md

@@ -169,3 +169,9 @@ beside telemetry when at least one new score is produced (tool-agnostic). Distin
 spans only; the root span keeps `fullsend.cost_usd` and `fullsend.tool_calls`
 (custom-namespaced, not auto-summed by MLflow). This prevents MLflow from
 double-counting token usage across the trace.
+
+**2026-09-03 — Tool-call span topology ([ADR 0102](0102-tool-call-span-topology.md)):**
+each tool call the runtime reports becomes an `execute_tool` child of its
+iteration's `agent` span, metadata only; the message record on the `agent`
+span stays the content carrier. Sub-agent nesting (deferred item 1 above)
+remains deferred.

docs/ADRs/0102-tool-call-span-topology.md

@@ -0,0 +1,94 @@
+---
+title: "102. Tool-call span topology"
+status: Accepted
+relates_to:
+  - operational-observability
+topics:
+  - observability
+  - telemetry
+  - opentelemetry
+---
+
+# 102. Tool-call span topology
+
+Date: 2026-09-03
+
+## Status
+
+Accepted
+
+## Context
+
+[ADR 0050](0050-distributed-tracing-instrumentation.md) chose OpenTelemetry
+and a three-level opt-in but never named the spans: the
+`run → sandbox_create → agent` tree lives only in the guides, and granularity
+was left to [#294](https://github.com/fullsend-ai/fullsend/issues/294).
+Level 3 content ([#6429](https://github.com/fullsend-ai/fullsend/pull/6429),
+[#6603](https://github.com/fullsend-ai/fullsend/pull/6603)) put tool calls
+and results on the `agent` span as `gen_ai.output.messages` parts — at
+semconv v1.37.0 the `execute_tool` span is metadata-only (its content
+attributes exist only in the newer GenAI repository, opt-in). Review of
+#6603 asked why tool calls are not spans.
+
+fullsend observes the runtime's stream rather than executing tools. The
+normalized `ToolUseEvent`/`ToolResultEvent` pairs carry a call id, a tool
+name and an `is_error` flag; `tool_use` lines carry no timestamp and
+`tool_result` lines carry a sandbox-clock one; several calls are open at
+once (parallel sub-agent dispatch); `parent_tool_use_id` is dropped at
+decode; the pi and codex parsers pass no call ids through.
+
+## Options
+
+1. **Message record only** (status quo): tool calls stay parts of
+   `gen_ai.output.messages`; no per-call timing or status in the span tree.
+2. **`execute_tool` child spans from the normalized events**: one span per
+   call under the iteration's `agent` span, metadata only; content stays on
+   the message record.
+3. **The runtime's native OpenTelemetry**: Claude Code emits
+   `claude_code.tool` spans (beta) and honours inbound `TRACEPARENT` — true
+   timing, but redaction would leave fullsend's pipeline, the threat model
+   keeps runtime telemetry out of scope, and the sandbox would need egress.
+4. **Per-tool content on the spans**: needs the newer conventions' opt-in
+   attributes, and the scorers read the message record today.
+
+## Decision
+
+Option 2. `toolSpanTracker` (`internal/cli/tool_spans.go`) opens an
+`execute_tool <tool name>` span (kind Internal) when the runtime reports a
+call and ends it when the result arrives. Both timestamps are runner-side
+receipt instants — one clock — so the span brackets execution rather than
+measuring it, and the start is arguments-complete. Attributes follow semconv
+v1.37.0: `gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`,
+`gen_ai.tool.call.id`; a result flagged `is_error` sets
+`error.type=tool_error` and status Error. Calls that never get a result
+close as `error.type=unanswered`, results for calls never reported are
+marked `fullsend.tool.unmatched`, and events without a call id (pi, codex,
+server-side tools) get no span — the edge cases are specified in the
+[dev guide](../guides/dev/tracing.md#execute_tool-spans). Names pass
+through the same sanitizer as span content and are bounded; at most 1,024
+spans are recorded per iteration, so an agent-controlled burst cannot fill
+the OTLP batch queue and evict the `agent` span, with the overflow counted
+in `fullsend.tool_spans.dropped`. The spans are Level 1 metadata, emitted
+regardless of the content gate. Tool content — results now, full arguments
+next — stays on the `agent` span's `gen_ai.output.messages` record, which
+is the scorer contract
+([ADR 0087](0087-eval-measurements-online-trace-scoring.md)).
+
+Option 3 is the candidate to revisit once the runtime's tracing is stable and
+a redaction stage outside fullsend is designed; option 4 waits for the
+convention bump.
+
+## Consequences
+
+- The span tree gains one level; readers that select spans by name
+  (`evalmeasure`) are unaffected, and the `execute_tool` count can be below
+  `fullsend.tool_calls`, which counts every reported call, id or not.
+- Tool-heavy iterations add hundreds of spans, each one synchronous write to
+  `run-telemetry.jsonl` and one OTLP batch entry.
+- Sub-agent calls are flat children of the `agent` span; nesting them under
+  the dispatching `Agent` call is a small follow-up now that the parent span
+  exists while its children run (ADR 0050's deferred item 1 stays deferred).
+- `execute_tool` spans are Claude-only until the pi and codex parsers pass
+  their streams' call ids through.
+- This settles the span-granularity question in #294; retention and access
+  remain open there.

docs/architecture.md

@@ -368,6 +368,8 @@ Observability is a cross-cutting concern that touches every other component. Eac
 
   > **Planned:** portable remote score export via the same OTLP configuration as agent traces ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). Not yet implemented.
 
+- Tool-call span topology: every tool call the runtime reports becomes an `execute_tool` child span of its iteration's `agent` span — semantic-convention metadata only, timed at runner receipt; tool content stays on the `agent` span's message record ([ADR 0102](ADRs/0102-tool-call-span-topology.md)).
+
 **Open questions:**
 
 - What signals matter most — cost, latency, token usage, action logs, decision traces, or something else?

docs/contributing/runtime-implementation.md

@@ -182,7 +182,7 @@ JSON on stdin; the script's exit code and stdout are the reply.
 | | Input | Reply |
 |---|---|---|
 | **PreToolUse** | `{"tool_name": ..., "tool_input": {...}}` | exit `0` = allow. Blocking scripts exit `1` and print `{"decision":"block","reason":"..."}`; the adapter must stop the tool call and surface the reason |
-| **PostToolUse** | the same plus the tool output as `tool_response` (Claude Code; string or structured object such as Bash `{stdout, stderr, interrupted, isImage}`), `tool_result` accepted as a fallback | *Blocking* (standalone `canary_posttool.py`, and `posttool_chain.py` when its canary stage fires): exit `1` + `{"decision":"block",...}`; the adapter drops the result. *Sanitizing* stages (suppress/unicode/redact): always exit `0` and, when they changed something, print `{"hookSpecificOutput":{"hookEventName":"PostToolUse","updatedToolOutput": <same shape as the input value>}, "tool_result": <scan text>}`. Empty stdout = unchanged |
+| **PostToolUse** | the same plus the tool output as `tool_response` (Claude Code; string or structured object such as Bash `{stdout, stderr, interrupted, isImage}`), `tool_result` accepted as a fallback; Claude Code also sends `cwd` (its working directory, which follows the agent's persisted `cd`) — the redact stage uses it to locate the checkout, and an adapter that omits it gets no bare-JWT skip (see *Sanitizer scope*) | *Blocking* (standalone `canary_posttool.py`, and `posttool_chain.py` when its canary stage fires): exit `1` + `{"decision":"block",...}`; the adapter drops the result. *Sanitizing* stages (suppress/unicode/redact): always exit `0` and, when they changed something, print `{"hookSpecificOutput":{"hookEventName":"PostToolUse","updatedToolOutput": <same shape as the input value>}, "tool_result": <scan text>}`. Empty stdout = unchanged |
 
 Shape rules:
 
@@ -202,6 +202,8 @@ The PostToolUse stages exist to remove *controls-relevant* content and nothing e
 
 A sweep of 900 fullsend files through the chain rewrites only test files holding token-shaped fakes.
 
+The bare-JWT prefix pattern is skipped when a file-content tool (`Read`, `Grep`, `Edit`, `MultiEdit`, `Write`, `NotebookEdit`, `NotebookRead`) is called with a path inside the checkout — the nearest `.git` ancestor of the hook input's `cwd`, none meaning no skip; the path is normalized, then resolved; `..` segments and `~` paths never skip — because a jwt.io-style fixture is byte-for-byte a valid token and masking it corrupts what the agent edits against, while the runner's own OIDC token file sits beside the checkout and still masks. Bash, WebFetch and MCP output are unaffected.
+
 **Context suppression** condenses the output of exactly one verification command, and only from positive evidence:
 
 - Commands: `go test`, `pytest`, `npm test`, `make test`, `pre-commit run`, `gitleaks detect`, `scan-secrets`, with optional setup prefixes (`cd`, `export`, `source`). The command must *start* with the tool, after wrappers that run it (`VAR=...`, `sudo`, `nice`, `timeout <n>`, `env VAR=...`, `uvx`, `npx`, `uv run`, `mise exec --`, stacked; `python3.12 -m pytest` counts). A command that merely mentions it, such as `grep -n scan-secrets hooks.py`, keeps its output.
@@ -504,7 +506,7 @@ The Claude-style agent `.md` is parsed by `Bootstrap`:
 
 `Bootstrap` installs `security.HookFiles` under `/sandbox/pi-config/hooks/`, writes the `HookPlan` into `fullsend-manifest.json` and loads the embedded `fullsend-hooks.js` extension with `-e` under `--no-extensions` (per pi v0.84.2 `docs/extensions.md`).
 
-`fullsend-hooks.js` sends the scripts `{tool_name, tool_input, tool_result, tool_response}` with Claude tool names (`bash→Bash`, `read→Read`, `write→Write`, `edit→Edit`, `grep→Grep`, `find→Glob`, `ls→LS`; `path` mirrored to `file_path`) and reads back either the v1 `tool_result` or the v2 `hookSpecificOutput.updatedToolOutput` (#6357), so the same extension works before and after the PostToolUse chain lands.
+`fullsend-hooks.js` sends the scripts `{tool_name, tool_input, tool_result, tool_response}` (no `cwd`, so the redact stage's checkout-scoped bare-JWT skip is inert under pi — it masks as before) with Claude tool names (`bash→Bash`, `read→Read`, `write→Write`, `edit→Edit`, `grep→Grep`, `find→Glob`, `ls→LS`; `path` mirrored to `file_path`) and reads back either the v1 `tool_result` or the v2 `hookSpecificOutput.updatedToolOutput` (#6357), so the same extension works before and after the PostToolUse chain lands.
 
 - PreToolUse groups run in `HookPlan` order and stop at the first block; a script that cannot be spawned blocks; PostToolUse blocks withhold the result and mark it `isError`.
 - An unreadable manifest, or one without a hook plan, blocks every tool call.

docs/guides/dev/tracing.md

@@ -91,12 +91,13 @@ silently.
 
 ## Span lifecycle in run.go
 
-`run.go` creates three span types arranged in a parent-child hierarchy:
+`run.go` creates four span types arranged in a parent-child hierarchy:
 
 ```
 run (root)
 ├── sandbox_create    (gen_ai.operation.name=create_agent)
 └── agent             (one per iteration; gen_ai.operation.name=invoke_agent)
+    └── execute_tool  (one per tool call; gen_ai.operation.name=execute_tool)
 ```
 
 ### Root span
@@ -132,6 +133,35 @@ build the attribute slices. Start attributes: `iteration`,
 `exit_code`, `gen_ai.system`, model, token counts, `fullsend.cost_usd`,
 `fullsend.tool_calls`.
 
+### execute_tool spans
+
+One per tool call the runtime reports, a child of that iteration's agent
+span, named `execute_tool <tool name>`. `toolSpanTracker`
+(`internal/cli/tool_spans.go`) starts the span when the `ToolUseEvent`
+arrives and ends it when the matching `ToolResultEvent` arrives, so both
+timestamps are runner-side receipt instants on one clock — the start is
+arguments-complete, not execution start. Attributes:
+`gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`,
+`gen_ai.tool.call.id`; a result flagged `is_error` sets
+`error.type=tool_error` and status Error. A call still open when the
+iteration ends — the runtime was stopped, or its result line exceeded the
+parser's 1 MiB cap — is closed by `Finish()` as `error.type=unanswered`; a
+result with no matching call (its `tool_use` line was skipped) becomes a
+near-zero-duration span marked `fullsend.tool.unmatched=true`. Events without
+an id — pi and codex emit none, and the parser gives server-side tools
+(`server_tool_use`) none because their result never arrives as a
+`tool_result` — produce no span, so the child count can be below
+`fullsend.tool_calls`. The name passes through `security.OutputPipeline()`
+— Unicode normalization, then secret redaction, the same pipeline as span
+content — and is bounded to 256 bytes before it becomes the attribute; the
+span name keeps at most 128 bytes of it.
+The tracker records at most `maxToolSpansPerIteration` (1,024) spans per
+iteration and reports the overflow, which `runAgent` records as
+`fullsend.tool_spans.dropped` on the agent span — a burst of agent-controlled
+calls must not fill the OTLP batch queue and evict the agent span that ends
+right after `Finish()`. These spans are metadata: they carry no tool content
+and are emitted whether or not the Level 3 gate is on.
+
 ### Level 3 content on agent spans
 
 When the content-capture gate is on
@@ -143,18 +173,26 @@ spans — and tees the runtime's normalized event stream to it through
 
 **The tee trap:** supplying any `OnEvent` replaces the runtime's default
 console renderer (`internal/runtime/claude.go`), so the handler built by
-`contentEventHandler` always calls the renderer first and the collector
-second. With the gate off the collector is nil and `contentEventHandler`
-returns nil, leaving the default renderer path byte-identical to before
-Level 3 existed.
+`iterationEventHandler` always calls the renderer first, then the
+collector, then the tool-span tracker. The handler is always set — tool
+spans are emitted with the gate off — and with the gate off the collector
+is nil and inert, so console output stays byte-identical to the default
+renderer path.
 
 The collector (`internal/cli/content_collector.go`) coalesces contiguous
-text/reasoning deltas, maps tool use to `tool_call` parts, redacts every
+text/reasoning deltas, maps tool use to `tool_call` parts and tool
+results to `tool_call_response` parts (correlated by `id` when the
+runtime's stream provides one; the schema's required result field is
+`response`), redacts every
 part through `security.OutputPipeline()` at assembly (redaction runs
 before the size budget — truncating first could split a secret past
 recognition), enforces a 256 KiB ordered-suffix budget (the ending survives — the
-final answer is what consumers judge) with exact dropped-byte accounting
-across content, tool names, and summaries, and emits
+final answer is what consumers judge) plus an 8 KiB per-tool-result
+bound (tail-kept, redacted before the cut, the part marked
+`fullsend.truncated`), with exact dropped-byte
+accounting across content, tool names, summaries, responses, and part
+ids (a third, earlier boundary — the parser's 1 MiB stream-line cap —
+drops oversized lines before any event exists), and emits
 `gen_ai.output.messages` JSON following the GenAI output-messages schema,
 including the schema-required `finish_reason` from the iteration outcome. `attachContent` records the
 content and its marker attributes on the span before either