feat(correlation): propagate request ID and trace context across all legs

[cdoern]

Problem

A single AI request reaches the backend over two paths:

• forwarded — the client's request, proxied upstream
• delegated — callouts the proxy originates itself while handling it (Files API metadata and content fetches)

Neither carried trace context. The two legs arrived at the backend as unrelated traffic, so a slow or failing request could not be attributed to a layer.

Change

A shared TraceContext resolves x-request-id and a W3C traceparent once per downstream request, and both are stamped onto every leg:

• New trace_context filter propagates them to the forwarded request.
• Delegated callouts inject them unconditionally, outside the operator-configured forward_headers allowlist — correlation that depends on per-filter configuration silently goes missing.

Each leg draws its own span-id from the shared trace-id, which is what keeps delegation latency separable from inference latency.

Notes for review

Identifier resolution reads extra_request_headers, not just downstream headers. The request_id builtin writes its generated ID there rather than back into ctx.request.headers. A downstream-only lookup finds an ID only when the client happened to supply one — and silently no-ops otherwise, which is the common case. There is a test named for it.

Agreement across hops goes through request extensions, not header inspection. Delegated callouts can run in the StreamBuffer pre-read phase, ahead of header-phase filters, where injected values are not visible. Whichever hop resolves first initializes the shared context; later hops reuse it regardless of filter order. An earlier header-based attempt produced two different trace-ids for one request, caught by the integration test in this PR.

Client-supplied traceparent values are validated and discarded when malformed rather than forwarded into the telemetry backend unchecked.

One assertion was deliberately relaxed. The delegated callouts of one request share a span, because correlation is resolved once at the filter boundary — file resolution is one delegation hop regardless of how many files it fetches. The test asserts that instead of per-callout spans, and that the forwarded leg's span is disjoint from it. Happy to switch to per-callout spans if reviewers prefer.

Incidental improvement: the callout header set is now built once per request instead of once per callout, removing repeated allowlist copies for requests that resolve several file references.

Scope

Trace context is propagated but not yet exported — no OpenTelemetry pipeline exists in this repo, so these headers make requests correlatable without producing spans. Wiring an exporter, covering delegated MCP calls, and recording a delegation-latency metric remain to be done.

Testing

• 9 unit tests for identifier resolution and traceparent validation, including a table of malformed inputs
• 7 unit tests for the trace_context filter
• 2 integration tests over the example config asserting both legs share a trace-id under distinct spans, for generated and client-supplied traces
• Example config examples/configs/trace-context.yaml, generated filter docs, full workspace suite and clippy -D warnings clean

[praxis-bot-app]

Unsigned commits: ca5fa2d. Please sign your commits.

[praxis-bot]

Review Summary

Clean, well-documented feature with thorough test coverage. The module-level doc in correlation/mod.rs is excellent -- the resolution precedence and the agreement-across-hops rationale are exactly what a future maintainer needs. Three medium-severity items below.

Severity	Count
Critical	0
Large	0
Medium	3

[leseb]

Good direction, thanks @cdoern

[leseb]

My AI review:

Findings

[MAJOR] request_id can overwrite the shared ID
File: filters/src/trace_context/mod.rs:96
What: With trace_context before request_id and no client ID, this filter generates ID A; request_id later generates ID B. Praxis applies pending headers sequentially, so the forwarded request carries B while delegated calls and the response retain A.
Why it matters: This defeats the feature’s primary cross-leg correlation guarantee.
Fix: Enforce and test request_id before trace_context, or make both filters consume the shared context/pending ID.

[MINOR] W3C future-version and reserved-flag handling is incomplete
File: apis/src/correlation/mod.rs:298
What: Version 00 values such as flags ff are re-emitted unchanged, despite unsupported bits requiring zeroing. Higher-version headers containing extension fields are rejected because the parser requires exactly four fields.
Why it matters: Reserved semantics can leak downstream, and traces restart during future-version rollouts despite otherwise valid base fields.
Fix: Mask flags to the supported sampled bit; parse the fixed base fields of higher versions, ignore extensions, then emit version 00. See the W3C versioning and flag requirements.

[NIT] New modules violate mandatory source ordering
Files: apis/src/correlation/mod.rs:47, filters/src/trace_context/mod.rs:40
What: Test modules precede imports and implementation rather than appearing last; required section separators are absent; several fields/methods are not alphabetized.
Fix: Apply the layout mandated by CONTRIBUTING.md.

[aslakknutsen]

My first instinct here is that most of this should be part of core and just "work out of the box" when enabled. The call site shouldn't have to be involved for the regular tracing headers./context

SubRequestClient already has FrameworkHeaders that can be used, but we're missing a request scoped context the SubRequestClient can fetch the trace context from.

Maybe HttpFilterContext::execute_subrequest(), or ctx.subrequests().execute() or similar... ?

@leseb

[praxis-bot]

Re-review (7 new commits)

All three prior findings have been addressed:

1. callout_headers resolved independently -- fixed: resolved once in resolve_and_rewrite and threaded through all phases.
2. sanitize could produce all-zero trace-id -- fixed: FALLBACK_TRACE_ID/FALLBACK_SPAN_ID constants with is_all_zero guards.
3. from_filter_context doc misleading -- fixed: doc now states the precondition and a debug! emits on fallback.

One new finding below.

Severity	Count
Critical	0
Large	0
Medium	1

[praxis-bot]

[Medium] tracestate is silently dropped when continuing a client trace.

resolve_trace extracts traceparent and discards the companion tracestate header. W3C Trace Context section 3.3.4 requires that when a tracing system continues a trace, it MUST propagate tracestate in the outgoing request. A downstream observability system that carries vendor-specific correlation data in tracestate (e.g. a sampling priority, tenant tag, or upstream span link) loses it at this hop.

Since the proxy does not maintain its own tracestate entry, passive forwarding is sufficient: capture the inbound tracestate alongside the traceparent in InboundTrace, store it in TraceContext, and emit it from Correlation::apply and TraceContext::headers_for_hop. When the proxy starts a new trace (no valid inbound traceparent), no tracestate is needed.

struct InboundTrace {
    flags: String,
    trace_id: String,
    tracestate: Option<String>,  // new
}

Then in TraceContext:

pub struct TraceContext {
    flags: String,
    request_id: String,
    trace_id: String,
    tracestate: Option<String>,  // forwarded from inbound
}

And in headers_for_hop / Correlation::apply, include tracestate when Some.

[VaishnaviHire]

My first instinct here is that most of this should be part of core and just "work out of the box" when enabled. The call site shouldn't have to be involved for the regular tracing headers./context

SubRequestClient already has FrameworkHeaders that can be used, but we're missing a request scoped context the SubRequestClient can fetch the trace context from.

Maybe HttpFilterContext::execute_subrequest(), or ctx.subrequests().execute() or similar... ?

@leseb

Here is the enhancement and draft PR

[praxis-bot]

We haven't heard back from you on this in some time, so we're going to mark it closed for now. However, if you're still interested in pushing this forward please don't hesitate to re-open and update 🖖