feat(observability): emit structured intelligent routing decisions

[nerdalert]

Summary

Add bounded, structured routing-decision logs to the Praxis AI intelligent_route filter so operators can understand how Grid-produced routing state was applied by each Praxis gateway.

This issue covers routing decisions only. It does not add general inference request logging, access logging, prompt logging, or response logging.

The implementation belongs in Praxis AI, while Grid owns the producer-side routing contract and the cross-repository demonstration and evidence.

Scope Boundary And Related Gap

Capability	Primary owner	Observability path	Status
Routing-decision logging	Praxis AI intelligent_route, with Grid contract and demonstration coverage	Structured gateway telemetry	Confirmed gap covered by this issue
Inference request logging through the gateway	Inference Gateway / Praxis	OTel Collector and Tempo	Separate confirmed gap tracked by RHAIRFE-2776 (New)

Grid MVP observability adds routing-decision logs. It does not add inference
request logging. This issue must not be used to claim that the separate
RHAIRFE-2776 request-logging gap is implemented or closed.

Current State

The current stack is split across:

• praxis-proxy/ai#339, which introduces intelligent_route;
• praxis-proxy/ai#540, which adds the versioned routing envelope and hot reload; and
• praxis-proxy/ai#386, which adds the authenticated provider-routing boundary.

Today, intelligent_route already records bounded in-process decision metadata for a successful selection, including:

• capability kind and model or tool selector;
• local and selected site;
• selected upstream cluster and stable candidate ID;
• admission state;
• selection tier and rank, when present; and
• session binding, reuse, and failover outcome.

For authenticated provider hops, it also records a generated hop request ID and carries the serving routing revision in trusted provider context.

Existing logs cover parts of the filter lifecycle and failure paths:

• debug logs for an already-selected cluster, missing or invalid selectors, unsupported MCP methods, and no matching eligible candidate;
• info logs for initial overlay acceptance, watcher startup and shutdown, and successful overlay reload;
• warnings for a full session-binding table and watcher problems;
• errors for watcher startup, overlay read, validation, and reload failures, including the retained last-known-good revision; and
• a trace log when file-watcher events are coalesced.

The original static implementation in #339 logs a successful selection at debug level. The stacked hot-reload request path in #540 and #386 no longer emits an equivalent structured event when a request is successfully routed. The decision metadata is present, but it is not itself an operator-visible log record.

There are also no focused tests that capture and validate the routing log event names, levels, fields, or sensitive-data exclusions.

Required Work

Define and implement a small, stable logging contract within intelligent_route for these routing outcomes:

• a new candidate is selected;
• an existing session binding is reused;
• a stale or ineligible session binding fails over to another candidate;
• no eligible candidate exists;
• an invalid routing selector is rejected; and
• a previously selected cluster is preserved.

Successful decision events should use the same immutable routing snapshot that selected the candidate. In overlay mode, the event must report the serving revision from that snapshot rather than reading mutable state again.

Use structured fields rather than formatting the decision into an unstructured message. Where applicable, include:

• a stable event or outcome name;
• capability kind and bounded model or tool selector;
• local site and selected site;
• selected cluster and stable candidate ID;
• admission state;
• selection tier and rank;
• session outcome: unbound, newly bound, reused, or failed over;
• serving overlay revision; and
• the existing correlation or provider-hop request ID when one is available.

Document and apply a deliberate level policy:

• info for the bounded operator-facing decision evidence required by the Grid demonstration, if normal production verbosity is intended to include each decision;
• warn only for actionable degraded behavior, not ordinary misses or caller-generated invalid input that could create warning floods;
• debug for routing outcomes and selection context that are useful during troubleshooting but too frequent for normal operation; and
• trace for candidate evaluation details and exclusion reasons when deeper diagnosis is enabled.

The implementation should avoid duplicating the same decision at several levels. The final level policy and stable fields must be documented so the Grid demonstration and operational guidance do not depend on matching prose.

Safety And Boundedness

• Do not log prompt or response bodies, message content, MCP arguments, authorization headers, credentials, credential values, cookies, session keys, or arbitrary request headers.
• Do not turn this work into general inference request or access logging.
• Reuse fields that have already passed intelligent_route validation and length bounds.
• Do not add request IDs, revisions, candidate IDs, clusters, sites, models, or tools as Prometheus metric labels.
• Avoid logging the full candidate list on every request.
• Preserve fail-closed routing behavior; logging failure must never change a routing result.

Acceptance Criteria

• A successful new selection emits one structured routing-decision event with the selected candidate, routing outcome, and serving revision.
• Session reuse and session failover are distinguishable without logging the session key.
• No-eligible-candidate and invalid-selector outcomes have documented, bounded diagnostics at appropriate levels.
• Static and hot-reloaded routing use the same event names and field vocabulary.
• The logged serving revision comes from the snapshot used for that request.
• Tests capture the emitted events and verify levels and required fields for selection, reuse, failover, and rejection paths.
• Negative tests verify that request bodies, credentials, authorization, cookies, and session keys are absent.
• Filter documentation explains the routing log contract and explicitly distinguishes it from inference request logging.
• The Grid global ingress demonstration captures gateway logs and proves at least one routing decision matches the observed selected provider and serving overlay revision.
• Existing routing, hot-reload, last-known-good, session-affinity, and provider-boundary behavior remains unchanged.

Non-Goals

• Logging complete inference requests or responses.
• Logging prompts, generated content, MCP arguments, or token contents.
• Adding a new log storage or aggregation system.
• Adding distributed tracing propagation outside the existing Praxis tracing architecture.
• Changing Grid scoring, candidate ordering, admission, or routing behavior.
• Adding high-cardinality Prometheus labels for individual routing decisions.

Open-Source Requirements

The events must use standard structured tracing and remain usable with common open-source log and OpenTelemetry collectors. The contract must not depend on a proprietary logging backend or distribution-specific control plane.