feat(telemetry): implement ADR 0050 Level 3 content capture

[dhshah13]

Implements ADR 0050's Level 3 exactly as accepted: when the org sets OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, conversation content is included in spans. One environment variable, the Level 2 enablement pattern — no new ADR, no additional consent surface.

The agent runtime's native content telemetry is never enabled. Content is assembled by fullsend's own runner from the normalized event stream the console already renders, redacted through security.OutputPipeline() at assembly, and attached to the per-iteration agent span. Fullsend reads the gate variable itself — it is a GenAI-instrumentation convention documented by opentelemetry-python-contrib; the pinned semconv v1.37.0 release does not define it, and Claude Code does not read it (0 occurrences in the shipped binary). Fullsend never sets the runtime's own OTEL_LOG_* variables, so there is no second export pipeline and no redaction bypass.

What ships

Piece	Where	Summary
Gate	internal/telemetry/content.go	Value contract below; unrecognized values are off — telemetry never fails a run
Collector	internal/cli/content_collector.go	Assembles text + reasoning + tool calls into one schema-conformant gen_ai.output.messages attribute per iteration
SDK limits	internal/telemetry/telemetry.go	Attribute cap lifted only while gated; free-text attrs bounded at call sites via boundedStringAttr; Setup warns if a finite operator limit would cut content JSON
Wiring	internal/cli/run.go	One collector per iteration, teed through RunParams.OnEvent with the console renderer preserved; gate off = OnEvent nil = the pre-change path, byte-identical
Docs	guides + threat model	One-line L3 enablement parallel to L2; consumer contract; pipeline-scope boundary sentence

Gate values

Enables capture	Stays off
true, span_only, span_and_event (case-insensitive)	unset, false, NO_CONTENT, event_only (we cannot honor "only"), anything unrecognized

Content shape. One assistant message per iteration, conformant to the v1.37.0 output-messages JSON schema including the required finish_reason (stop/error from the iteration outcome). Parts: text, reasoning (the schema's GenericPart extension point), and tool_call with name + summary — no fabricated arguments.

Redaction and size. Redaction runs before any cut (truncating first could split a secret); hits are masked, counted in fullsend.content.redactions, and warned on the console. The 256 KiB budget keeps an ordered suffix — the final answer survives a cut — with exact dropped-byte accounting surfaced as fullsend.content.truncated / fullsend.content.dropped_bytes, and early eviction keeps long sessions memory-bounded.

Deliberately not captured (documented as dispositions)

• gen_ai.input.messages — first-iteration runs have no meaningful runner-side input; retry iterations now carry the injected validation feedback (Validation feedback is injected into the agent prompt unframed and unsanitized (feedback_mode: append) #6502), which makes input capture a natural follow-up in this series.
• Tool results — not in the normalized event stream; follows as a parser extension in the next PR of this series.
• Sub-agent attribution — the stream carries no provenance; sub-agent activity appears unattributed exactly as in the console (ADR 0050's deferred sub-agent item).
• Pre/post-script content.

Next in this series (PR B, starts after this merges)

Tool results complete the captured record. One PR, three parts:

1. Parser extension — a new ToolResultEvent in internal/runtime, emitted from the tool_result blocks the Claude stream parser currently discards, as a runtime-agnostic addition to the normalized AgentEvent contract.
2. Collector mapping — one new case mapping it to the schema's tool_call_response part ({type, id, result}); the gate, redaction, suffix budget, and markers in this PR handle it with no new surface (still one env var).
3. Budget re-derivation — the 256 KiB bound is deliberately revisited then: tool results are where content volume concentrates, and the guide carries a note saying so.

The parser half ships together with its consumer rather than ahead of it, so no event type lands without production callers.

Evidence (pilot MLflow, experiment 1)

Trace	Proves
tr-5802e956713c146edef284bca8a0d338	Live gated run (exit 0, validation passed): captured conversation with finish_reason=stop, byte-identical in run-telemetry.jsonl and on the backend; console rendered normally through the tee
tr-a1703820bb276f1121a9f808d0bde07e	MLflow translates the semconv content attributes into its Request/Response preview (root-span-derived; fullsend content lives in the trace's span view — deliberately not duplicated onto the root, the #5788 double-count anti-pattern)
tr-4a4ce9d96307f239880d5e2ba1802f72	A 255,082-byte content attribute arrived byte-complete — nothing silently dropped at budget scale
gate-off control	Deterministic e2e test: zero content bytes anywhere; event-handler path identical to main

Test plan

• go test ./internal/telemetry/ ./internal/cli/ -race — green. (The two TestDummyRuntime_* failures on the author's machine are pre-existing environment flakes present on bare main.)
• Unit coverage on all new functions: value-contract table, schema structural assertions, suffix-budget / rune-boundary / exact-accounting cases, eviction (including redact-before-evict), sanitize-to-empty, tee, marker-only attach, bounded attributes, operator-limit warning.
• e2e through the real telemetry.Setup() file sink for both gate states.
• Live validation per the evidence table above.

Notes for review

• Single-gate model: docs(adr): specify Level 3 content capture activation and sandbox env denylist #5947 established that additional gates collapsed to the same repo-write actor (repo variables shadow org variables; no CODEOWNERS surface; ADR 0044 removes org mode), so this PR claims only what ADR 0050 claims — the org that sets the gate owns its backend's access controls (0050's own allocation). Backend ACL adequacy for a given deployment is an operational sign-off, not a code property.
• The OTel spec's external-storage-with-references pattern is the recommended production shape for high-volume content; Export agent traces to persistent GCP bucket for cross-team querying #6410's bucket export is its natural future home and is cross-referenced in the guide.

Relates to #5361 (content capture is the last telemetry level; the enrichment legs remain open). Relates to #6036 (eval scorers read run-telemetry.jsonl; content-aware scorers become possible once this lands). Relates to #294 (retention; unchanged by this PR).

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[github-actions]

Site preview

Preview: https://eca2d23e-site.fullsend-ai.workers.dev

Commit: 94c3c4b4619a461f546e7738142ded008e34a796

[codecov]

Codecov Report

❌ Patch coverage is 98.69281% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/cli/content_collector.go	98.43%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[qodo-code-review]

PR Summary by Qodo

feat(telemetry): implement Level 3 content capture on agent spans

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Add opt-in Level 3 gate to attach assistant output content to agent spans.
• Assemble redacted, size-bounded gen_ai.output.messages per iteration from normalized events.
• Lift SDK attribute cap only when gated; warn on operator limits that corrupt JSON.

Diagram

graph TD
  ENV{{"OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"}} --> GATE["telemetry.ContentCaptureEnabled()"] --> RUN["cli.runAgent"] --> COLLECT["contentCollector (per iteration)"] --> REDACT["security.OutputPipeline()"] --> SPAN(["agent span attrs: gen_ai.output.messages + markers"]) --> EXPORT["Telemetry sinks (jsonl + OTLP)"]
  GATE --> LIMITS["telemetry.spanLimits()"] --> EXPORT
  subgraph Legend
    direction LR
    _ext{{"Env var"}} ~~~ _fn["Function/Component"] ~~~ _span(["Span data"])
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Enable runtime-native OTel content telemetry (OTEL_LOG_*)

• ➕ Less custom code for message assembly/schema conformance
• ➕ Potentially captures more raw runtime details (e.g., tool I/O)
• ➖ Creates a second content export path with different redaction guarantees
• ➖ Harder to enforce “redact before truncate” and consistent byte budgeting
• ➖ Contradicts stated boundary: fullsend should not enable in-sandbox telemetry

2. Store full transcript externally and reference it from spans

• ➕ Avoids large span attributes and backend ingestion limits
• ➕ Supports stronger access controls and lifecycle management for sensitive data
• ➖ Extra storage/infra and operational complexity
• ➖ Not “ADR 0050 Level 3 exactly as accepted” if it requires new surface/contract
• ➖ Requires consumers to resolve references; reduces portability of spans

Recommendation: Keep the PR’s approach: assembling content from the normalized event stream in the runner preserves a single, redaction-controlled pipeline and matches ADR 0050’s one-env-var opt-in. The added SDK-cap lift + operator-limit warning is the right guardrail to keep gen_ai.output.messages parseable across sinks.

Files changed (12) +1234 / -24

Enhancement (4) +451 / -11

content_collector.goImplement per-iteration content collector with redaction and suffix budget +319/-0

Implement per-iteration content collector with redaction and suffix budget

• Introduces a runtime-agnostic collector that consumes normalized AgentEvents, coalesces deltas, maps tool usage into schema parts, redacts via security.OutputPipeline, and emits schema-conformant 'gen_ai.output.messages' JSON with required finish_reason. Enforces a 256KiB ordered-suffix byte budget with exact dropped-byte accounting and eviction to keep long sessions bounded.

internal/cli/content_collector.go

run.goWire content capture into runAgent and bound free-text attrs +47/-4

Wire content capture into runAgent and bound free-text attrs

• Creates one collector per iteration when gated, tees events to both renderer and collector, and attaches content + markers to the agent span with finish_reason derived from iteration outcome. Adds boundedStringAttr and applies it to previously unbounded free-text attributes to avoid oversized export batches when SDK caps are lifted.

internal/cli/run.go

content.goAdd Level 3 env-var gate parser for content capture +35/-0

Add Level 3 env-var gate parser for content capture

• Defines OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT and implements the accepted value contract, enabling capture only for span-compatible values and defaulting to off for unknown inputs to avoid failing runs.

internal/telemetry/content.go

telemetry.goLift SDK attribute cap under gate and warn on conflicting operator limits +50/-7

Lift SDK attribute cap under gate and warn on conflicting operator limits

• Updates spanLimits to keep attribute values unlimited when Level 3 capture is enabled (unless an operator override exists), avoiding mid-JSON truncation by the SDK. Adds operator limit resolution and a startup stderr warning when a finite operator cap would silently corrupt content JSON.

internal/telemetry/telemetry.go

Tests (5) +654 / -2

content_collector_test.goAdd comprehensive unit tests for collection, redaction, and truncation +323/-0

Add comprehensive unit tests for collection, redaction, and truncation

• Covers delta coalescing, tool_call mapping, schema requirements (finish_reason/type), secret redaction (including tool name/summary), suffix truncation on rune boundaries, exact dropped-byte accounting, and eviction/redaction ordering invariants.

internal/cli/content_collector_test.go

scan_output_telemetry_test.goClarify telemetry JSONL redaction-scan expectations for Level 3 +4/-2

Clarify telemetry JSONL redaction-scan expectations for Level 3

• Updates the test commentary to reflect that Level 3 content is already redacted at assembly, so the host-side scan should continue skipping the telemetry JSONL file.

internal/cli/scan_output_telemetry_test.go

telemetry_run_test.goAdd tests for content attachment, gating, and end-to-end file sink +197/-0

Add tests for content attachment, gating, and end-to-end file sink

• Adds unit tests verifying attachContent marker behavior, boundedStringAttr enforcement, and the OnEvent tee semantics. Adds integration tests ensuring large content survives into run-telemetry.jsonl when gated and that gate-off runs produce no content-shaped attributes.

internal/cli/telemetry_run_test.go

content_test.goTest content gate contract and span-limit lift behavior +69/-0

Test content gate contract and span-limit lift behavior

• Adds table-driven tests for ContentCaptureEnabled’s accepted/ignored values and verifies that spanLimits lifts the default cap only when the gate is on and no operator override applies.

internal/telemetry/content_test.go

telemetry_test.goPin operator-limit warning behavior for gated content capture +61/-0

Pin operator-limit warning behavior for gated content capture

• Extends env pinning and adds tests that capture stderr to ensure Setup warns only when Level 3 capture is enabled and a finite operator attribute-length limit is configured.

internal/telemetry/telemetry_test.go

Documentation (3) +129 / -11

tracing.mdDocument Level 3 collector wiring and consumer contract +37/-0

Document Level 3 collector wiring and consumer contract

• Adds an explanation of per-iteration content collection, the OnEvent tee behavior, and why the default renderer must be preserved. Documents redaction-before-budgeting, ordered-suffix truncation semantics, and how consumers should interpret marker attributes.

docs/guides/dev/tracing.md

distributed-tracing.mdDefine Level 3 content capture semantics and boundaries +91/-11

Define Level 3 content capture semantics and boundaries

• Updates tracing levels to mark Level 3 as implemented and clarifies that content is attached to 'agent' spans. Documents the env-var value contract, captured vs. excluded content, redaction and sizing behavior, SDK cap interactions, and backend/MLflow considerations.

docs/guides/infrastructure/distributed-tracing.md

security-threat-model.mdAdd telemetry content boundary to threat model +1/-0

Add telemetry content boundary to threat model

• Extends the threat model to explicitly scope content-handling guarantees to fullsend’s extraction/redaction pipeline, excluding in-sandbox runtime instrumentation.

docs/problems/security-threat-model.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (1)

1. Unbounded agentName attributes ✓ Resolved 🐞 Bug ☼ Reliability

Description

When content capture is enabled, spanLimits() removes the SDK attribute value-length cap, but
agentName is still recorded via stringAttr on the root span and every per-iteration agent span.
A very large CLI run <agent-name> argument can therefore create oversized span batches and
increase the chance the exporter/collector rejects the whole batch.

Code

internal/telemetry/telemetry.go[R110-116]

+		if ContentCaptureEnabled() {
+			// Level 3 puts JSON-string content attributes on spans. The
+			// default cap would cut such a value mid-string and corrupt
+			// the JSON; the content collector's byte budget is the size
+			// bound, so the SDK cap stays unlimited. An operator's
+			// explicit limit env var still wins above.
+			return limits

Relevance

●●● Strong

Recent telemetry reviews accept bounding untrusted dynamic values when provider-wide span limits are
lifted.

PR-#5944

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR introduces an unlimited SDK attribute value-length cap under Level 3, but agentName
continues to be added to spans via stringAttr (no truncation). Since agentName is taken directly
from the CLI argument, it can be arbitrarily large and will be exported unbounded when content
capture is enabled.

internal/telemetry/telemetry.go[107-120]
internal/cli/run.go[242-251]
internal/cli/run.go[922-928]
internal/cli/run.go[2360-2365]
internal/cli/run.go[2532-2550]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
With Level 3 enabled, `spanLimits()` returns unlimited `AttributeValueLengthLimit` to avoid corrupting `gen_ai.output.messages` JSON mid-value. However, `agentName` is still attached to spans using `stringAttr` (UTF-8 repair only), and `agentName` originates directly from the CLI arg with no length bound. This makes it possible for an arbitrarily-long agent name to bloat multiple spans and risk OTLP batch rejection.

### Issue Context
- `ContentCaptureEnabled()` controls whether `spanLimits()` lifts the SDK cap.
- Root span attributes include `fullsend.agent` and `gen_ai.agent.name`.
- Agent iteration spans include `gen_ai.agent.name`.

### Fix Focus Areas
- internal/telemetry/telemetry.go[107-119]
- internal/cli/run.go[242-251]
- internal/cli/run.go[922-928]
- internal/cli/run.go[2360-2365]

### Suggested fix
1. Replace `stringAttr(..., agentName)` with a bounded helper for all span attributes that take `agentName`, e.g.:
  - root span: `boundedStringAttr("fullsend.agent", agentName)` and `boundedStringAttr("gen_ai.agent.name", agentName)`
  - agent span start attrs: `boundedStringAttr("gen_ai.agent.name", agentName)`
2. Consider using a *non-ellipsis* truncation helper for identifier-like fields (agent names) if consumers treat ellipsis as problematic; otherwise reuse `boundedStringAttr`.
3. (Optional hardening) Validate/limit `agentName` length at CLI parsing time to avoid also creating overly-long `sandboxName`/paths.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Infra guide outside admin/user 📜 Skill insight ⌂ Architecture

Description

The PR modifies guides under docs/guides/ that are currently located in
docs/guides/infrastructure/ and docs/guides/dev/, but guides in this tree must live under either
docs/guides/admin/ or docs/guides/user/. This violates the required documentation directory
structure and indexing conventions.

Code

docs/guides/infrastructure/distributed-tracing.md[15]

+| 3 | Conversation content (assistant text, reasoning, tool calls) on `agent` spans | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` |

Relevance

● Weak

Recent precedents explicitly reject relocating guides from infrastructure/dev despite the same
documented taxonomy concern.

PR-#5944
PR-#5454

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The compliance checklist states that all guide files under docs/guides/ must be placed in either
admin/ or user/; however, the cited changes are in
docs/guides/infrastructure/distributed-tracing.md and docs/guides/dev/tracing.md, demonstrating
that the modified guides reside outside the allowed subdirectories and therefore violate the
placement requirement.

docs/guides/infrastructure/distributed-tracing.md[15-15]
docs/guides/dev/tracing.md[135-138]
Skill: writing-user-docs

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Guides updated by this PR are located under `docs/guides/infrastructure/` and `docs/guides/dev/`, but the documentation rules require that all guides under `docs/guides/` be placed under `docs/guides/admin/` or `docs/guides/user/`; this breaks the required directory structure and indexing conventions.

## Issue Context
Because this PR updates these guide files, they must comply with the guide placement rule; adjust the locations (and any related references/indexing described in the guides README) so the guides reside under the permitted `admin/` or `user/` subtrees.

## Fix Focus Areas
- docs/guides/infrastructure/distributed-tracing.md[12-20]
- docs/guides/dev/tracing.md[135-171]
- docs/guides/README.md[1-200]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Planned feature missing callout ✓ Resolved 📜 Skill insight ≡ Correctness

Description

The guide mentions a not-yet-implemented "future bucket-export pipeline" but does not use the
required > **Planned:** blockquote callout format. This can mislead readers about what is
currently shipped versus planned.

Code

docs/guides/infrastructure/distributed-tracing.md[R142-145]

+recommends external storage with span references for high-volume or
+high-sensitivity production use; that pattern is a natural fit for a
+future bucket-export pipeline
+([#6410](https://github.com/fullsend-ai/fullsend/issues/6410)).

Relevance

● Weak

A closely matching planned-callout request was explicitly rejected when the guide already linked
tracking context.

PR-#3903

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The checklist requires any mention of not-yet-implemented features to use the > **Planned:**
blockquote callout format with an issue link. The added text describes a "future bucket-export
pipeline" (planned) but is written as normal prose instead of the required callout.

docs/guides/infrastructure/distributed-tracing.md[142-145]
Skill: writing-user-docs

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A not-yet-implemented feature is described without the required `> **Planned:**` callout format.

## Issue Context
Docs must clearly distinguish planned functionality from current behavior and include an issue link using the required callout style.

## Fix Focus Areas
- docs/guides/infrastructure/distributed-tracing.md[142-145]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 58 rules

✅ Skills: writing-user-docs, writing-adrs

Tip of the day

💡 Did you know, you can hide the parts of a finding you never read, like the evidence or the agent prompt

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[rh-hemartin]

You need to add the variable to the workflows and add the variable to the user guide, follow the same style and tone of the guide. It shouldn't take more than 7 or 8 lines.

[rh-hemartin]

LGTM but I would want to see a review instrumented trace, as they are very complex.

[dhshah13]

Fixed in 829bd9e — OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT forwarded at all 13 managed agent steps (6 reusable-<stage>.yml + 7 inline steps in reusable-dispatch.yml) on the vars channel like OTEL_SDK_DISABLED, pinned by TestOTELVariableForwarding. User guide: 3b086ad adds "Capture conversation content" to how-to-emit-traces.md (8 lines); dbf888d lists the variable in the reference's GHA configuration table and BYO example.

[dhshah13]

Review-instrumented trace, as requested: tr-7d31c9bc618064343147b91d5478df88 (pilot MLflow, experiment 1) — the shipped review agent run against this PR with capture on, at the pushed head dbf888d.

What it shows: the agent span carries a 42,000-byte gen_ai.output.messages with 107 parts — 19 reasoning, 21 text, 67 tool calls including 11 sub-agent dispatches (tool_call name Agent) — finish_reason=stop, byte-identical in run-telemetry.jsonl and on the backend. fullsend.content.redactions=23: the agent quoted this PR's own ghp_… test fixtures while reviewing and the assembly-time redactor caught every one (none survive into the attribute). Content fit the 256 KiB budget, so no truncation markers; validation passed against the real review-result schema; console output rendered normally through the tee. 884s, $10.57.

[dhshah13]

On the $10.57 for tr-7d31c9bc618064343147b91d5478df88: verified independently from the raw transcript. 38 unique API calls (32 main-thread, 6 sub-agent) priced at Opus list rates ($15/$75 per MTok in/out, $18.75 cache-write, $1.50 cache-read) recompute to $10.55 vs Claude Code's reported total_cost_usd $10.57 — no double-count. Shape: cache reads ~3.9M tokens ≈ $5.9, cache writes ~245K ≈ $4.6, output ~14.5K ≈ $1.1 — the review agent re-reading a large cached context across 32 turns, not generation. The two result events in the transcript ($8.77 after turn 1, $10.57 after turn 7) are a stop-hook continuation with cumulative cost; the final figure is the run's cost.

[rh-hemartin]

Looks good to me

[dhshah13]

Merge-queue dequeue was two unrelated infra flakes, not the diff: functional eval:001-bug-url-encoding lost the parallel-case race ensuring the vertex-ai provider (provider was modified concurrently (resource_version: 2) — four cases start in the same second; siblings passed), and behaviour failed in the pool-org allocation hook (fullsend inference provision halfsend-07/test-repo-01 exit 1) before any scenario logic. Adjacent merge-group runs the same hour passed both suites; neither code path is touched by this PR. Re-queue should clear it.