telemetry: OTLP export silently loses spans — endpoint path handling, terminal flush, and no wire-level tests

[guyoron1]

Upstream: fullsend-ai#5545

What happens

Three related behaviors on main (3c57dda9), introduced or exposed by the fullsend-ai#4510 OTel SDK migration, each verified with live runs against a current-main build.

1. A generic OTEL_EXPORTER_OTLP_ENDPOINT with a path posts to the wrong URL.
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318/otlp POSTs to /otlp (not /otlp/v1/traces); a trailing slash (http://collector:4318/) POSTs to /. Bare host:port works. The OTLP spec appends /v1/traces to the generic var; Setup resolves the env itself and passes the value through otlptracehttp.WithEndpointURL, which overrides the SDK env layer's path join (internal/telemetry/telemetry.go:36-38, :89-94). The operator guide promises the append (docs/guides/infrastructure/distributed-tracing.md:45-46). Conversely, a path-less signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT gets /v1/traces appended where spec and docs say used-as-is. Net effect: any path-prefixed collector/gateway endpoint 404s and every span is dropped; validateEndpoint (telemetry.go:104-113) accepts these values, so there is no warning of any kind.

2. The terminal flush loses the root span on any retryable failure, silently.
The exporter is constructed with SDK-default retry (first backoff ~5s jittered, 10s HTTP timeout, MaxElapsedTime 60s); cleanup flushes under a 5-second context (internal/cli/run.go:858-862). One 503/429/hang at flush time → a single attempt, the budget expires mid-backoff, the tp.Shutdown error is discarded (telemetry.go:82), and the run exits 0 with empty stderr. The root span carries the aggregate gen_ai.usage.*/fullsend.cost_usd/exit_code attributes and is always in the final drain; for runs shorter than the batch tick, the entire trace is. Live reproduction: a 503 sink and a hanging sink both produced exactly one delivery attempt, 5000 ms of cleanup, exit 0, and no output. The pre-fullsend-ai#4510 exporter set WithRetry{1s, 2s, 4s} (comment: SDK defaults are "tuned for long-lived services, not a CLI at exit") and printed StepWarn on export failure; both were dropped in the migration. The guide's "flushed within a 5-second budget" (distributed-tracing.md:64-66) is not met whenever the first jittered backoff (~2.5–7.5 s) outlives the remaining budget — the common case, and the observed one in both live reproductions.

3. The wire path is untested.
fullsend-ai#4510 deleted the ~700-line test matrix that exercised a real exporter against in-process protobuf-decoding servers (endpoint-path pinning, header injection, gzip, timeout, TCP blackhole, 503-retry-then-delivered). The replacement tests stub the newOTLPExporter seam and discard the endpoint argument (internal/telemetry/telemetry_test.go:82, :103), so TestSetup_TracesEndpointPreferred passes even if precedence were reversed, and no test on main sends a span through the real HTTP exporter. Items 1 and 2 shipped precisely because their pinning tests were deleted.

What should happen

• Both endpoint vars follow the OTLP spec and the documented contract: generic = base URL with /v1/traces appended; signal-specific = used verbatim.
• The terminal flush either delivers within its budget or leaves an operator-visible warning; a run whose export failed is distinguishable from one whose export succeeded.
• Wire-level behavior (URL path, headers, retry-within-budget, fail-open) is pinned by tests against a real in-process OTLP sink.

How to reproduce (current main)

1. Build the CLI, run any traced invocation with OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:<port>/otlp pointed at a request-logging sink → observe POST /otlp.
2. Same invocation with the sink returning 503 → observe one attempt, ~5 s cleanup, exit 0, empty stderr, no trace delivered.

Context

Found in a retrospective review of fullsend-ai#4510 ahead of Level 3 content-capture work, which builds on this export path. Verified live 2026-07-23 against main 3c57dda9. Relates to fullsend-ai#2862, PR fullsend-ai#4510.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 1:02 PM UTC · Completed 1:06 PM UTC
Commit: 9908ddc · View workflow run →

[fullsend-ai-triage]

Triage Summary

This is a well-documented, high-severity bug report covering three related regressions in the telemetry export path, all introduced by the OTel SDK migration in PR fullsend-ai#4510. Each was verified with live reproductions against commit 3c57dda9.

Problems identified

1. Endpoint path handling violates OTLP spec — endpointFromEnv() passes both the generic and signal-specific env var values identically to otlptracehttp.WithEndpointURL, which overrides the SDK's internal path-join. The generic OTEL_EXPORTER_OTLP_ENDPOINT with a path (e.g., /otlp) should have /v1/traces appended per the OTLP spec, but currently does not — causing 404s at any path-prefixed collector.

2. Terminal flush silently drops spans — The cleanup closure discards tp.Shutdown errors, and the 5s context budget cannot accommodate the SDK-default retry backoff (~5s jittered). A single retryable failure (503/429/hang) at flush time results in one attempt, budget expiry, exit 0, and no warning. The root span (carrying aggregate usage/cost attributes) is always in the final drain.

3. Wire-level test coverage removed — The ~700-line test matrix exercising a real exporter against in-process sinks was deleted in fullsend-ai#4510. Replacement tests stub the newOTLPExporter seam, so endpoint path construction and retry behavior are untested.

Recommended fix

• Differentiate the two endpoint env vars in endpointFromEnv/Setup: append /v1/traces to the generic var, pass the signal-specific var verbatim.
• Surface tp.Shutdown errors (e.g., StepWarn) and configure CLI-appropriate retry settings (short backoff fitting within the 5s budget).
• Restore wire-level integration tests with an in-process OTLP HTTP sink.

Proposed test cases

1. TestEndpointPath_GenericVar: generic endpoint with path -> assert POST <path>/v1/traces
2. TestEndpointPath_SignalSpecificVar: signal-specific endpoint -> assert POST <exact-path> (no append)
3. TestFlush_RetryableFailure_WarnsOnDrop: 503 sink -> assert stderr warning, no silent drop
4. TestFlush_TransientThenSuccess: 503-then-200 -> assert span delivered within budget
5. TestEndpointPath_BareHostPort: bare host:port -> assert POST /v1/traces

---

Labels: This is a confirmed bug in Go code under internal/telemetry/ with high impact (silent data loss). Matches type/bug and priority/high labels.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[mysticgohan1]

/fs-code

[guyoron1]

/fs-code

[guyoron1]

/fs-code

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 2:53 AM UTC · Completed 3:04 AM UTC
Commit: ba77dbd · View workflow run →

[guyoron1]

Path A complete — closing for Path B run.