test: add a validation for single-cluster multi-gateway routing

docs/README.md

@@ -52,6 +52,9 @@
   proves shared identity-scoped quota enforcement across gateway replicas,
   regional provider selection, expiry, restart persistence, fail-closed state
   storage, and storage-network isolation.
+- [Single-cluster Multi-gateway Qualification](../tests/e2e/topologies/grid-single-cluster-multi-gateway/README.md) -
+  proves shared overlay delivery and independent consumer/provider gateway
+  behavior within one Kind cluster and one GridSite.
 
 These integration tests create their environments through Forge and execute
 through first-class Rust `xtask` commands. Their topology READMEs document

docs/architecture/ci-kind-e2e.md

@@ -11,6 +11,7 @@ validation suite.
 | Unit | Operator, scoring, xtask, and parser tests | Validates controller logic, overlay rendering, scoring, metrics handling, and harness helpers without Kind. |
 | Smoke Kind | Single-topology operator routing validation | Proves the operator can reconcile resources, render an overlay, and drive a consumer gateway in Kind. |
 | Multi-cluster Kind | SWIM, CRDT, stale GC, metrics routing, and credential validation | Proves the distributed control-plane paths across multiple Kind clusters. |
+| Single-cluster multi-gateway Kind | Shared-site overlays, independent consumer processes, provider attribution, and gateway failure behavior | Proves that multiple gateways can share one Kubernetes control plane without implying cross-cluster behavior. |
 
 ## Gate implementation
 
@@ -26,6 +27,22 @@ Sequence:
    validation.
 4. Full two-provider suite for nightly or release validation.
 
+## Single-cluster multi-gateway coverage
+
+The `run-grid-single-cluster-multi-gateway-qualification` command uses one Kind
+cluster, one Grid operator, one GridNetwork, and one GridSite named `single`, with
+two consumer gateways and three provider gateways. It is distinct from both a
+single gateway smoke test and the multi-cluster provider-traffic qualification:
+the single-cluster test shares one site, Kubernetes control plane, and overlay
+state, while the multi-cluster test exercises multiple sites connected through
+SWIM. Provider-selection cursors remain local to each Praxis process. The
+qualification must prove accepted and serving overlay revisions, per-consumer
+attributed traffic, provider withdrawal and restoration, consumer failure, and
+positive and negative security probes before it can report success.
+
+Topology and execution details are maintained in the
+[single-cluster topology README](../../tests/e2e/topologies/grid-single-cluster-multi-gateway/README.md).
+
 ## Multi-cluster coverage set
 
 | Validation | Behavior proven |

tests/e2e/topologies/grid-single-cluster-multi-gateway/.gitignore

@@ -0,0 +1 @@
+evidence/

tests/e2e/topologies/grid-single-cluster-multi-gateway/README.md

@@ -0,0 +1,179 @@
+# Single-cluster multi-gateway qualification
+
+This qualification keeps one Kubernetes control plane and one Grid site while
+running two independent consumer gateways, three independent provider gateways,
+and one attributed simulator per provider. It complements the multi-cluster
+provider-traffic qualification: this topology exercises shared Kubernetes and
+overlay state, but it does not claim WAN SWIM or cross-cluster network behavior.
+
+More precisely, this topology contains one Kubernetes cluster, one Grid
+operator, one GridNetwork, and one GridSite named `single`. Multiple consumer
+and provider gateway processes share that site and its generated routing state.
+
+```mermaid
+flowchart LR
+  client[restricted client pod]
+  subgraph kind[one Kind cluster]
+    op[Grid operator]
+    network[GridNetwork: single]
+    site[GridSite: single]
+    ca[consumer-a]
+    cb[consumer-b]
+    pa[provider-a]
+    pb[provider-b]
+    pc[provider-c]
+    ba[simulator-a]
+    bb[simulator-b]
+    bc[simulator-c]
+    op -->|two accepted overlays| ca
+    op -->|two accepted overlays| cb
+    op --> network
+    network --> site
+    pa --> ba
+    pb --> bb
+    pc --> bc
+  end
+  client --> ca
+  client --> cb
+  ca --> pa
+  ca --> pb
+  ca --> pc
+  cb --> pa
+  cb --> pb
+  cb --> pc
+```
+
+```mermaid
+flowchart TD
+  request[request] --> snapshot[accepted local overlay]
+  snapshot --> group[best eligible group]
+  group --> cursor[consumer-local round-robin cursor]
+  cursor --> provider[provider gateway]
+  provider --> backend[attributed simulator]
+```
+
+Round-robin state is process-local. The qualification therefore checks a valid
+balanced rotation independently through each consumer and does not require one
+global interleaved sequence. Grid publishes eligibility, groups, and policy;
+Praxis selects from its already-loaded local snapshot on the request path.
+
+## Health convergence
+
+This is a single-site qualification, so the Grid operator relies on its direct
+provider health checks rather than multi-site SWIM failure detection. The
+topology sets each provider's `healthCheck.interval` to `10s`, instead of the
+`30s` production default, to make local provider withdrawal and recovery
+convergence observable within the qualification run. This is a test-topology
+tuning example; it does not change the operator's production default.
+
+## Configuration
+
+The topology sets `selectionPolicy.mode: roundRobin` and publishes the three
+providers in the same eligible selection group. Both consumers receive their
+own generated overlay and maintain an independent request-selection cursor.
+
+| Path | Purpose |
+|---|---|
+| [`forge.yaml`](./forge.yaml) | One-cluster Forge environment and ordered stack definitions |
+| [`configs/consumer/`](./configs/consumer/) | Consumer filter chains and provider-hop clusters |
+| [`configs/provider/`](./configs/provider/) | Provider routes and trusted response attribution |
+| [`resources/common/`](./resources/common/) | VCR simulators, restricted client, namespace, and NetworkPolicy |
+
+The Forge environment retains `crossCluster: true` because Forge uses that
+network mode to allocate MetalLB addresses reachable by the host-side test
+orchestrator. It still creates exactly one Kubernetes cluster and does not test
+cross-cluster discovery.
+
+## Run the qualification
+
+Prerequisites are Docker, Kind, `kubectl`, Helm, OpenSSL, Rust, and an AI source
+checkout next to or otherwise accessible from this Grid checkout. The runner
+uses `imagePullPolicy: Never` and requires these exact local image references:
+
+- `grid-operator:single-cluster-qualification`
+- `grid-overlay-sync:single-cluster-qualification`
+- `praxis-ai:single-cluster-qualification`
+- `ghcr.io/neuralmagic/vllm-vcr:vllm0.23`
+
+Build Forge and the Grid images from this checkout:
+
+```console
+cargo build -p forge
+
+docker build -f deploy/operator/Containerfile \
+  -t grid-operator:single-cluster-qualification .
+
+docker build -f overlay-sync/Containerfile \
+  -t grid-overlay-sync:single-cluster-qualification .
+```
+
+Build the gateway from a clean Praxis AI checkout. This qualification uses the
+standard provider-selection path and does not require the optional distributed
+quota filters:
+
+```console
+docker build -f Containerfile \
+  -t praxis-ai:single-cluster-qualification .
+```
+
+Pull the pinned simulator image, validate the topology, and run focused static
+tests before creating the cluster:
+
+```console
+docker pull ghcr.io/neuralmagic/vllm-vcr:vllm0.23
+
+target/debug/praxis-forge \
+  --config tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml \
+  config validate
+
+cargo test -p xtask single_cluster_multi_gateway --locked
+```
+
+Run the qualification from the Grid repository root:
+
+```console
+cargo xtask env run-grid-single-cluster-multi-gateway-qualification \
+  --forge-config tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml
+```
+
+Use `--keep` only for bounded diagnosis; it intentionally leaves the created
+cluster running. Use `--evidence-dir PATH` to place evidence somewhere other
+than the ignored topology-local `evidence/` directory.
+
+## Expected result
+
+A passing run proves that:
+
+- every required image is present in the Kind node before stack application;
+- all stacks and Deployments reach their observed generations;
+- both consumers receive the same three-candidate Grid overlay;
+- each consumer's accepted and serving revisions match the Grid revision;
+- each consumer independently follows the attributed A/B/C rotation;
+- removing provider B's backend withdraws B from both accepted overlays and
+  new traffic continues through A and C;
+- restoring provider B returns it to both overlays;
+- consumer B continues serving while consumer A is unavailable, and consumer A
+  serves again after recovery;
+- concurrent requests retain trusted attribution and collectively reach all
+  three providers; and
+- the restricted client can use the consumer path but cannot connect directly
+  to a protected inference backend.
+
+The runner writes timestamped `results.json` and `SUMMARY.md` files. Generated
+evidence is ignored by Git and must not be committed. It uses bounded
+subprocesses, JSON resource reads, observed-generation readiness, and automatic
+Forge teardown. It never edits an accepted overlay directly.
+
+## Scope
+
+This qualification proves multiple independent gateway processes inside one
+Kubernetes cluster and one Grid site. It does not prove WAN connectivity, SWIM
+membership between sites, a single globally coordinated round-robin cursor, or
+load balancing among replicas hidden behind one provider gateway. Use the
+multi-cluster provider-traffic qualification for cross-site discovery and
+routing.
+
+The checked-in qualification must remain honest about the distinction between
+bootstrap evidence and request-path evidence. A run is not successful unless
+overlay/serving revision barriers, provider attribution, withdrawal/restoration,
+consumer failure, and positive/negative security probes all pass.

tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/consumer/praxis-a.yaml

@@ -0,0 +1,78 @@
+insecure_options:
+  allow_private_endpoints: true
+
+listeners:
+  - name: proxy
+    address: "0.0.0.0:8080"
+    filter_chains:
+      - main
+
+filter_chains:
+  - name: main
+    filters:
+      - filter: json_body_field
+        field: model
+        header: X-Model
+      - filter: headers
+        response_set:
+          - name: X-Grid-Provider-Traffic-Consumer-Gateway
+            value: "{{ cluster.name }}"
+      - filter: intelligent_route
+        overlay_file: /etc/praxis/routing/routing-overlay.json
+        model_header: X-Model
+        provider_hop_clusters:
+          - vcr-provider-a-provider
+          - vcr-provider-b-provider
+          - vcr-provider-c-provider
+        expected_overlay_scope:
+          network: grid-single-cluster-multi-gateway
+          gateway: consumer-gateway-a
+          namespace: grid-system
+          local_site: "single"
+        reload:
+          enabled: true
+          debounce_ms: 500
+        session_affinity:
+          enabled: true
+          header: X-Session-Id
+          ttl_secs: 3600
+      - filter: load_balancer
+        clusters:
+          - name: vcr-provider-a-provider
+            tls:
+              ca:
+                ca_path: /etc/praxis/tls/ca.crt
+              client_cert:
+                cert_path: /etc/praxis/tls/tls.crt
+                key_path: /etc/praxis/tls/tls.key
+              sni: provider-a.grid.internal
+              verify: true
+            endpoints:
+              - "provider-gateway-a.grid-system.svc.cluster.local:8443"
+          - name: vcr-provider-b-provider
+            tls:
+              ca:
+                ca_path: /etc/praxis/tls/ca.crt
+              client_cert:
+                cert_path: /etc/praxis/tls/tls.crt
+                key_path: /etc/praxis/tls/tls.key
+              sni: provider-b.grid.internal
+              verify: true
+            endpoints:
+              - "provider-gateway-b.grid-system.svc.cluster.local:8443"
+          - name: vcr-provider-c-provider
+            tls:
+              ca:
+                ca_path: /etc/praxis/tls/ca.crt
+              client_cert:
+                cert_path: /etc/praxis/tls/tls.crt
+                key_path: /etc/praxis/tls/tls.key
+              sni: provider-c.grid.internal
+              verify: true
+            endpoints:
+              - "provider-gateway-c.grid-system.svc.cluster.local:8443"
+
+admin:
+  address: "127.0.0.1:9901"
+
+shutdown_timeout_secs: 5

tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/consumer/praxis-b.yaml

@@ -0,0 +1,78 @@
+insecure_options:
+  allow_private_endpoints: true
+
+listeners:
+  - name: proxy
+    address: "0.0.0.0:8080"
+    filter_chains:
+      - main
+
+filter_chains:
+  - name: main
+    filters:
+      - filter: json_body_field
+        field: model
+        header: X-Model
+      - filter: headers
+        response_set:
+          - name: X-Grid-Provider-Traffic-Consumer-Gateway
+            value: "{{ cluster.name }}"
+      - filter: intelligent_route
+        overlay_file: /etc/praxis/routing/routing-overlay.json
+        model_header: X-Model
+        provider_hop_clusters:
+          - vcr-provider-a-provider
+          - vcr-provider-b-provider
+          - vcr-provider-c-provider
+        expected_overlay_scope:
+          network: grid-single-cluster-multi-gateway
+          gateway: consumer-gateway-b
+          namespace: grid-system
+          local_site: "single"
+        reload:
+          enabled: true
+          debounce_ms: 500
+        session_affinity:
+          enabled: true
+          header: X-Session-Id
+          ttl_secs: 3600
+      - filter: load_balancer
+        clusters:
+          - name: vcr-provider-a-provider
+            tls:
+              ca:
+                ca_path: /etc/praxis/tls/ca.crt
+              client_cert:
+                cert_path: /etc/praxis/tls/tls.crt
+                key_path: /etc/praxis/tls/tls.key
+              sni: provider-b.grid.internal
+              verify: true
+            endpoints:
+              - "provider-gateway-a.grid-system.svc.cluster.local:8443"
+          - name: vcr-provider-b-provider
+            tls:
+              ca:
+                ca_path: /etc/praxis/tls/ca.crt
+              client_cert:
+                cert_path: /etc/praxis/tls/tls.crt
+                key_path: /etc/praxis/tls/tls.key
+              sni: provider-b.grid.internal
+              verify: true
+            endpoints:
+              - "provider-gateway-b.grid-system.svc.cluster.local:8443"
+          - name: vcr-provider-c-provider
+            tls:
+              ca:
+                ca_path: /etc/praxis/tls/ca.crt
+              client_cert:
+                cert_path: /etc/praxis/tls/tls.crt
+                key_path: /etc/praxis/tls/tls.key
+              sni: provider-c.grid.internal
+              verify: true
+            endpoints:
+              - "provider-gateway-c.grid-system.svc.cluster.local:8443"
+
+admin:
+  address: "127.0.0.1:9901"
+
+shutdown_timeout_secs: 5