feat: Enrich auth pipeline trace spans with decision attributes and per-evaluator child spans

[asanzgom]

Summary

Authorino's current tracing emits a Check span with only authorino.request_id and auto-generated gRPC attributes. All useful auth metadata — denial reason, policy name, identity source, per-evaluator timing — is only available in logs or HTTP headers, not in trace spans.

This proposal adds domain-specific span attributes and per-evaluator child spans to make Authorino traces actionable for debugging auth flows in production.

Proposed Changes

1. Check Span Attribute Enrichment (smallest PR, highest value)

Add attributes to the existing Check span in pkg/service/auth.go:

Attribute	Type	Description
authorino.auth.result	string	ALLOW / DENY
authorino.auth.response_code	string	gRPC status code name
authorino.auth.denial_reason	string	result.Message (denied only)
authorino.auth.denial_code	int	result.Code (denied only)
authorino.auth_config.name	string	Matched AuthConfig CR name
authorino.auth_config.namespace	string	AuthConfig CR namespace
authorino.identity.source	string	Resolved identity evaluator name
authorino.identity.type	string	Identity evaluator type

2. Per-Phase Child Spans

Add child spans in pkg/service/auth_pipeline.go for each pipeline phase:

Check (enriched)
+-- identity
+-- metadata
+-- authorization
+-- response
+-- callbacks

Each phase span carries authorino.phase, authorino.phase.evaluator_count, and authorino.phase.result.

3. Per-Evaluator Child Spans

Add child spans inside evaluateAuthConfig() for each evaluator invocation:

Check
+-- identity
|   +-- evaluator: keycloak-jwt [oidc]
|   +-- evaluator: api-key-auth [apiKey]
+-- authorization
|   +-- evaluator: admin-policy [opa]
|   +-- evaluator: rbac-check [kubernetes-subjectaccessreview]
...

Each evaluator span carries authorino.evaluator.name, authorino.evaluator.type, authorino.evaluator.priority, and authorino.evaluator.result.

4. Config Flag

A verbosity toggle (e.g., tracing.detailedSpans: true/false) to gate per-evaluator spans at high request rates.

Attribute Naming

Using authorino.* namespace, consistent with the existing authorino.request_id. OTel has no general-purpose auth semantic conventions — the only auth-decision attributes are aspnetcore.* (framework-specific), and semantic-conventions#183 (API Gateway conventions) has been open since 2023 with no progress.

Implementation Plan

This would be delivered as 4 incremental PRs:

1. Check span enrichment — adds result attributes to the existing span (~200 LOC, low risk)
2. Per-phase child spans — 5 child spans + context propagation (~300-500 LOC, medium risk)
3. Per-evaluator child spans — spans inside evaluateAuthConfig() (~200-300 LOC, low risk)
4. Config flag — CRD schema change to gate span verbosity (~300-400 LOC, medium risk)

Context

This work is part of RHOAI Observability Standards and Telemetry Contract. We are looking at enriching traces across the RHOAI inference stack (Envoy, Authorino, model server), and Authorino auth spans are the current gap.

PR #574 (control-plane tracing) is a direct precedent for this work.

Happy to discuss naming conventions, span hierarchy, or config approach before submitting PRs.