Duplicate traces on terminal deny action combined with auth action

[silvi-t]

When a PipelinePolicy with an add_deny action (terminal: true) is used alongside an AuthPolicy (with or without RateLimitPolicy) on the same route, and the deny predicate matches:

1. Duplicate traces — a single HTTP request produces two separate traces
2. Duplicate Authorino call — the auth gRPC call is made twice for one request, adding unnecessary latency and load
3. Missing grpc_response span — the deny action interrupts the pipeline before the auth gRPC response is processed

When only RateLimitPolicy + PipelinePolicy (no AuthPolicy)

• No duplicate traces — only one trace per request
• Missing grpc_response span still present — the deny action still interrupts the pending rate limit gRPC call before its response arrives

Root cause

The terminal deny action calls Envoy's send_local_response(403) before pending gRPC responses arrive. This causes the missing grpc_response span in all cases. The duplicate trace specifically requires the auth action — the auth service's response-phase callback chain causes Envoy to re-invoke the wasm filter, triggering a full pipeline re-execution with a second Authorino call.

Reproduction matrix

Policy combination	Deny matches	Duplicate trace	Missing grpc_response
PipelinePolicy alone	yes	no	yes
RateLimitPolicy + PipelinePolicy	yes	no	yes
AuthPolicy + PipelinePolicy	yes	yes	yes
AuthPolicy + RateLimitPolicy + PipelinePolicy	yes	yes	yes
Any combination	no (200)	no	no

Does NOT happen when

• add_fail is used instead of add_deny
• The deny predicate does not match (request returns 200)
• No AuthPolicy is involved (duplication specifically)

Evidence

Two Jaeger trace JSONs from a single curl request:

• Different trace IDs and request IDs
• Same action set hash
• ~17ms apart
• Both contain auth gRPC call to Authorino
• Neither contains a grpc_response span

[adam-cattermole]

If it's easy to reproduce it'd be great to have the EnvoyFilter with the wasm config for this - that should quite easily tell us if this is a bug in the wasm-shim or it's just doing what it was told to do

[silvi-t]

If it's easy to reproduce it'd be great to have the EnvoyFilter with the wasm config for this - that should quite easily tell us if this is a bug in the wasm-shim or it's just doing what it was told to do

Here is the wasm confign from EnvoyFilter:

{
    "actionSets": [
      {
        "actions": [
          {
            "scope": "12c3d8aa631c3fde8ba9b813904af34490abb6b2450f67e60d9992a2ea167105",
            "service": "auth-service",
            "sources": ["authpolicy.kuadrant.io:kuadrant/authz-starabov--qqc0"]
          },
          {
            "messageBuilder": "threat.v1.ThreatRequest{uri: request.path, source_ip: source.address}",
            "predicate": "true",
            "service": "ext-a80174be134bff00",
            "sources": ["PipelinePolicy/kuadrant/pipeline-starabov--snsf"],
            "terminal": false,
            "type": "grpc",
            "var": "threatResponse"
          },
          {
            "denyWith": "DenyResponse{status: 403u}",
            "predicate": "request.url_path == \"/blocked\"",
            "sources": ["PipelinePolicy/kuadrant/pipeline-starabov--snsf"],
            "terminal": true,
            "type": "deny"
          },
          {
            "headers": "[['x-threat-assessed', 'true']]",
            "predicate": "true",
            "sources": ["PipelinePolicy/kuadrant/pipeline-starabov--snsf"],
            "target": "response",
            "terminal": false,
            "type": "headers"
          }
        ],
        "name": "c051d26069475c02a9b2ad61451f7a84770eea9d8be96d73a02a2e2ab160c858",
        "routeRuleConditions": {
          "hostnames": ["hostname-starabov--sl2n.test.com"],
          "predicates": ["request.url_path.startsWith('/')"]
        }
      }
    ],
    "descriptorService": "kuadrant-operator-grpc",
    "observability": {
      "defaultLevel": "DEBUG",
      "httpHeaderIdentifier": "x-request-id",
      "tracing": {"service": "tracing-service"}
    },
    "services": {
      "auth-service": {"endpoint": "kuadrant-auth-service", "failureMode": "deny", "timeout": "1000ms", "type": "auth"},
      "ext-a80174be134bff00": {"endpoint": "ext-threat-starabov--edl2-kuadrant-svc-cluster-local-8080", "failureMode": "deny", "grpcMethod": "AssessRequest", "grpcService": "threat.v1.ThreatAssessmentService",
  "timeout": "100ms", "type": "dynamic"},
      "tracing-service": {"endpoint": "kuadrant-tracing-service", "failureMode": "allow", "timeout": "1000ms", "type": "tracing"},
      "ratelimit-service": {"endpoint": "kuadrant-ratelimit-service", "failureMode": "allow", "timeout": "1000ms", "type": "ratelimit"},
      "ratelimit-check-service": {"endpoint": "kuadrant-ratelimit-service", "failureMode": "allow", "timeout": "1000ms", "type": "ratelimit-check"},
      "ratelimit-report-service": {"endpoint": "kuadrant-ratelimit-service", "failureMode": "allow", "timeout": "1000ms", "type": "ratelimit-report"}
    }
  }