diff --git a/docs/README.md b/docs/README.md index ad47ef8..95fa753 100644 --- a/docs/README.md +++ b/docs/README.md @@ -52,6 +52,9 @@ proves shared identity-scoped quota enforcement across gateway replicas, regional provider selection, expiry, restart persistence, fail-closed state storage, and storage-network isolation. +- [Single-cluster Multi-gateway Qualification](../tests/e2e/topologies/grid-single-cluster-multi-gateway/README.md) - + proves shared overlay delivery and independent consumer/provider gateway + behavior within one Kind cluster and one GridSite. These integration tests create their environments through Forge and execute through first-class Rust `xtask` commands. Their topology READMEs document diff --git a/docs/architecture/ci-kind-e2e.md b/docs/architecture/ci-kind-e2e.md index 3a00e12..0d3e674 100644 --- a/docs/architecture/ci-kind-e2e.md +++ b/docs/architecture/ci-kind-e2e.md @@ -11,6 +11,7 @@ validation suite. | Unit | Operator, scoring, xtask, and parser tests | Validates controller logic, overlay rendering, scoring, metrics handling, and harness helpers without Kind. | | Smoke Kind | Single-topology operator routing validation | Proves the operator can reconcile resources, render an overlay, and drive a consumer gateway in Kind. | | Multi-cluster Kind | SWIM, CRDT, stale GC, metrics routing, and credential validation | Proves the distributed control-plane paths across multiple Kind clusters. | +| Single-cluster multi-gateway Kind | Shared-site overlays, independent consumer processes, provider attribution, and gateway failure behavior | Proves that multiple gateways can share one Kubernetes control plane without implying cross-cluster behavior. | ## Gate implementation @@ -26,6 +27,22 @@ Sequence: validation. 4. Full two-provider suite for nightly or release validation. +## Single-cluster multi-gateway coverage + +The `run-grid-single-cluster-multi-gateway-qualification` command uses one Kind +cluster, one Grid operator, one GridNetwork, and one GridSite named `single`, with +two consumer gateways and three provider gateways. It is distinct from both a +single gateway smoke test and the multi-cluster provider-traffic qualification: +the single-cluster test shares one site, Kubernetes control plane, and overlay +state, while the multi-cluster test exercises multiple sites connected through +SWIM. Provider-selection cursors remain local to each Praxis process. The +qualification must prove accepted and serving overlay revisions, per-consumer +attributed traffic, provider withdrawal and restoration, consumer failure, and +positive and negative security probes before it can report success. + +Topology and execution details are maintained in the +[single-cluster topology README](../../tests/e2e/topologies/grid-single-cluster-multi-gateway/README.md). + ## Multi-cluster coverage set | Validation | Behavior proven | diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/.gitignore b/tests/e2e/topologies/grid-single-cluster-multi-gateway/.gitignore new file mode 100644 index 0000000..48a58e6 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/.gitignore @@ -0,0 +1 @@ +evidence/ diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/README.md b/tests/e2e/topologies/grid-single-cluster-multi-gateway/README.md new file mode 100644 index 0000000..92bcf0f --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/README.md @@ -0,0 +1,179 @@ +# Single-cluster multi-gateway qualification + +This qualification keeps one Kubernetes control plane and one Grid site while +running two independent consumer gateways, three independent provider gateways, +and one attributed simulator per provider. It complements the multi-cluster +provider-traffic qualification: this topology exercises shared Kubernetes and +overlay state, but it does not claim WAN SWIM or cross-cluster network behavior. + +More precisely, this topology contains one Kubernetes cluster, one Grid +operator, one GridNetwork, and one GridSite named `single`. Multiple consumer +and provider gateway processes share that site and its generated routing state. + +```mermaid +flowchart LR + client[restricted client pod] + subgraph kind[one Kind cluster] + op[Grid operator] + network[GridNetwork: single] + site[GridSite: single] + ca[consumer-a] + cb[consumer-b] + pa[provider-a] + pb[provider-b] + pc[provider-c] + ba[simulator-a] + bb[simulator-b] + bc[simulator-c] + op -->|two accepted overlays| ca + op -->|two accepted overlays| cb + op --> network + network --> site + pa --> ba + pb --> bb + pc --> bc + end + client --> ca + client --> cb + ca --> pa + ca --> pb + ca --> pc + cb --> pa + cb --> pb + cb --> pc +``` + +```mermaid +flowchart TD + request[request] --> snapshot[accepted local overlay] + snapshot --> group[best eligible group] + group --> cursor[consumer-local round-robin cursor] + cursor --> provider[provider gateway] + provider --> backend[attributed simulator] +``` + +Round-robin state is process-local. The qualification therefore checks a valid +balanced rotation independently through each consumer and does not require one +global interleaved sequence. Grid publishes eligibility, groups, and policy; +Praxis selects from its already-loaded local snapshot on the request path. + +## Health convergence + +This is a single-site qualification, so the Grid operator relies on its direct +provider health checks rather than multi-site SWIM failure detection. The +topology sets each provider's `healthCheck.interval` to `10s`, instead of the +`30s` production default, to make local provider withdrawal and recovery +convergence observable within the qualification run. This is a test-topology +tuning example; it does not change the operator's production default. + +## Configuration + +The topology sets `selectionPolicy.mode: roundRobin` and publishes the three +providers in the same eligible selection group. Both consumers receive their +own generated overlay and maintain an independent request-selection cursor. + +| Path | Purpose | +|---|---| +| [`forge.yaml`](./forge.yaml) | One-cluster Forge environment and ordered stack definitions | +| [`configs/consumer/`](./configs/consumer/) | Consumer filter chains and provider-hop clusters | +| [`configs/provider/`](./configs/provider/) | Provider routes and trusted response attribution | +| [`resources/common/`](./resources/common/) | VCR simulators, restricted client, namespace, and NetworkPolicy | + +The Forge environment retains `crossCluster: true` because Forge uses that +network mode to allocate MetalLB addresses reachable by the host-side test +orchestrator. It still creates exactly one Kubernetes cluster and does not test +cross-cluster discovery. + +## Run the qualification + +Prerequisites are Docker, Kind, `kubectl`, Helm, OpenSSL, Rust, and an AI source +checkout next to or otherwise accessible from this Grid checkout. The runner +uses `imagePullPolicy: Never` and requires these exact local image references: + +- `grid-operator:single-cluster-qualification` +- `grid-overlay-sync:single-cluster-qualification` +- `praxis-ai:single-cluster-qualification` +- `ghcr.io/neuralmagic/vllm-vcr:vllm0.23` + +Build Forge and the Grid images from this checkout: + +```console +cargo build -p forge + +docker build -f deploy/operator/Containerfile \ + -t grid-operator:single-cluster-qualification . + +docker build -f overlay-sync/Containerfile \ + -t grid-overlay-sync:single-cluster-qualification . +``` + +Build the gateway from a clean Praxis AI checkout. This qualification uses the +standard provider-selection path and does not require the optional distributed +quota filters: + +```console +docker build -f Containerfile \ + -t praxis-ai:single-cluster-qualification . +``` + +Pull the pinned simulator image, validate the topology, and run focused static +tests before creating the cluster: + +```console +docker pull ghcr.io/neuralmagic/vllm-vcr:vllm0.23 + +target/debug/praxis-forge \ + --config tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml \ + config validate + +cargo test -p xtask single_cluster_multi_gateway --locked +``` + +Run the qualification from the Grid repository root: + +```console +cargo xtask env run-grid-single-cluster-multi-gateway-qualification \ + --forge-config tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml +``` + +Use `--keep` only for bounded diagnosis; it intentionally leaves the created +cluster running. Use `--evidence-dir PATH` to place evidence somewhere other +than the ignored topology-local `evidence/` directory. + +## Expected result + +A passing run proves that: + +- every required image is present in the Kind node before stack application; +- all stacks and Deployments reach their observed generations; +- both consumers receive the same three-candidate Grid overlay; +- each consumer's accepted and serving revisions match the Grid revision; +- each consumer independently follows the attributed A/B/C rotation; +- removing provider B's backend withdraws B from both accepted overlays and + new traffic continues through A and C; +- restoring provider B returns it to both overlays; +- consumer B continues serving while consumer A is unavailable, and consumer A + serves again after recovery; +- concurrent requests retain trusted attribution and collectively reach all + three providers; and +- the restricted client can use the consumer path but cannot connect directly + to a protected inference backend. + +The runner writes timestamped `results.json` and `SUMMARY.md` files. Generated +evidence is ignored by Git and must not be committed. It uses bounded +subprocesses, JSON resource reads, observed-generation readiness, and automatic +Forge teardown. It never edits an accepted overlay directly. + +## Scope + +This qualification proves multiple independent gateway processes inside one +Kubernetes cluster and one Grid site. It does not prove WAN connectivity, SWIM +membership between sites, a single globally coordinated round-robin cursor, or +load balancing among replicas hidden behind one provider gateway. Use the +multi-cluster provider-traffic qualification for cross-site discovery and +routing. + +The checked-in qualification must remain honest about the distinction between +bootstrap evidence and request-path evidence. A run is not successful unless +overlay/serving revision barriers, provider attribution, withdrawal/restoration, +consumer failure, and positive/negative security probes all pass. diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/consumer/praxis-a.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/consumer/praxis-a.yaml new file mode 100644 index 0000000..6955530 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/consumer/praxis-a.yaml @@ -0,0 +1,78 @@ +insecure_options: + allow_private_endpoints: true + +listeners: + - name: proxy + address: "0.0.0.0:8080" + filter_chains: + - main + +filter_chains: + - name: main + filters: + - filter: json_body_field + field: model + header: X-Model + - filter: headers + response_set: + - name: X-Grid-Provider-Traffic-Consumer-Gateway + value: "{{ cluster.name }}" + - filter: intelligent_route + overlay_file: /etc/praxis/routing/routing-overlay.json + model_header: X-Model + provider_hop_clusters: + - vcr-provider-a-provider + - vcr-provider-b-provider + - vcr-provider-c-provider + expected_overlay_scope: + network: grid-single-cluster-multi-gateway + gateway: consumer-gateway-a + namespace: grid-system + local_site: "single" + reload: + enabled: true + debounce_ms: 500 + session_affinity: + enabled: true + header: X-Session-Id + ttl_secs: 3600 + - filter: load_balancer + clusters: + - name: vcr-provider-a-provider + tls: + ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert: + cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + sni: provider-a.grid.internal + verify: true + endpoints: + - "provider-gateway-a.grid-system.svc.cluster.local:8443" + - name: vcr-provider-b-provider + tls: + ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert: + cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + sni: provider-b.grid.internal + verify: true + endpoints: + - "provider-gateway-b.grid-system.svc.cluster.local:8443" + - name: vcr-provider-c-provider + tls: + ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert: + cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + sni: provider-c.grid.internal + verify: true + endpoints: + - "provider-gateway-c.grid-system.svc.cluster.local:8443" + +admin: + address: "127.0.0.1:9901" + +shutdown_timeout_secs: 5 diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/consumer/praxis-b.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/consumer/praxis-b.yaml new file mode 100644 index 0000000..c49990d --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/consumer/praxis-b.yaml @@ -0,0 +1,78 @@ +insecure_options: + allow_private_endpoints: true + +listeners: + - name: proxy + address: "0.0.0.0:8080" + filter_chains: + - main + +filter_chains: + - name: main + filters: + - filter: json_body_field + field: model + header: X-Model + - filter: headers + response_set: + - name: X-Grid-Provider-Traffic-Consumer-Gateway + value: "{{ cluster.name }}" + - filter: intelligent_route + overlay_file: /etc/praxis/routing/routing-overlay.json + model_header: X-Model + provider_hop_clusters: + - vcr-provider-a-provider + - vcr-provider-b-provider + - vcr-provider-c-provider + expected_overlay_scope: + network: grid-single-cluster-multi-gateway + gateway: consumer-gateway-b + namespace: grid-system + local_site: "single" + reload: + enabled: true + debounce_ms: 500 + session_affinity: + enabled: true + header: X-Session-Id + ttl_secs: 3600 + - filter: load_balancer + clusters: + - name: vcr-provider-a-provider + tls: + ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert: + cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + sni: provider-b.grid.internal + verify: true + endpoints: + - "provider-gateway-a.grid-system.svc.cluster.local:8443" + - name: vcr-provider-b-provider + tls: + ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert: + cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + sni: provider-b.grid.internal + verify: true + endpoints: + - "provider-gateway-b.grid-system.svc.cluster.local:8443" + - name: vcr-provider-c-provider + tls: + ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert: + cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + sni: provider-c.grid.internal + verify: true + endpoints: + - "provider-gateway-c.grid-system.svc.cluster.local:8443" + +admin: + address: "127.0.0.1:9901" + +shutdown_timeout_secs: 5 diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-a.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-a.yaml new file mode 100644 index 0000000..ee5e129 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-a.yaml @@ -0,0 +1,63 @@ +# Provider gateway fixture used by the provider-traffic xtask tests. +insecure_options: + allow_private_endpoints: true + +listeners: + - name: provider + address: "0.0.0.0:8443" + filter_chains: + - provider-inference + tls: + certificates: + - cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + client_ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert_mode: require + +filter_chains: + - name: provider-inference + filters: + - filter: peer_identity_trust + trusted_peers: + - organization: ai-grid + - filter: json_body_field + field: model + header: X-Model + - filter: headers + response_set: + - name: X-Grid-Provider-Traffic-Provider-Gateway + value: "provider-a" + - filter: provider_route + provider_id: "provider-a" + model_header: X-Model + emit_demo_attribution: true + routes: + - candidate_id: "22fec63f" + model: Qwen/Qwen3-0.6B + paths: + - /v1/chat/completions + - /v1/responses + cluster: vcr-backend-a + credential: + strategy: bearer_token + secretRef: + name: vcr-inference-credential + namespace: grid-system + key: token + - filter: credential_inject + credentials: + - strategy: bearer_token + name: vcr-inference-credential + namespace: grid-system + key: token + file: /etc/praxis/credentials/vcr-inference/token + - filter: load_balancer + clusters: + - name: vcr-backend-a + endpoints: + - "vcr-inference-provider-a.grid-system.svc.cluster.local:8000" + +admin: + address: "127.0.0.1:9901" +shutdown_timeout_secs: 5 diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-b.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-b.yaml new file mode 100644 index 0000000..4189dd0 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-b.yaml @@ -0,0 +1,63 @@ +# Provider gateway fixture used by the provider-traffic xtask tests. +insecure_options: + allow_private_endpoints: true + +listeners: + - name: provider + address: "0.0.0.0:8443" + filter_chains: + - provider-inference + tls: + certificates: + - cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + client_ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert_mode: require + +filter_chains: + - name: provider-inference + filters: + - filter: peer_identity_trust + trusted_peers: + - organization: ai-grid + - filter: json_body_field + field: model + header: X-Model + - filter: headers + response_set: + - name: X-Grid-Provider-Traffic-Provider-Gateway + value: "provider-b" + - filter: provider_route + provider_id: "provider-b" + model_header: X-Model + emit_demo_attribution: true + routes: + - candidate_id: "caaa0f80" + model: Qwen/Qwen3-0.6B + paths: + - /v1/chat/completions + - /v1/responses + cluster: vcr-backend-b + credential: + strategy: bearer_token + secretRef: + name: vcr-inference-credential + namespace: grid-system + key: token + - filter: credential_inject + credentials: + - strategy: bearer_token + name: vcr-inference-credential + namespace: grid-system + key: token + file: /etc/praxis/credentials/vcr-inference/token + - filter: load_balancer + clusters: + - name: vcr-backend-b + endpoints: + - "vcr-inference-provider-b.grid-system.svc.cluster.local:8000" + +admin: + address: "127.0.0.1:9901" +shutdown_timeout_secs: 5 diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-c.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-c.yaml new file mode 100644 index 0000000..b3dce34 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/configs/provider/praxis-c.yaml @@ -0,0 +1,63 @@ +# Provider gateway fixture used by the provider-traffic xtask tests. +insecure_options: + allow_private_endpoints: true + +listeners: + - name: provider + address: "0.0.0.0:8443" + filter_chains: + - provider-inference + tls: + certificates: + - cert_path: /etc/praxis/tls/tls.crt + key_path: /etc/praxis/tls/tls.key + client_ca: + ca_path: /etc/praxis/tls/ca.crt + client_cert_mode: require + +filter_chains: + - name: provider-inference + filters: + - filter: peer_identity_trust + trusted_peers: + - organization: ai-grid + - filter: json_body_field + field: model + header: X-Model + - filter: headers + response_set: + - name: X-Grid-Provider-Traffic-Provider-Gateway + value: "provider-c" + - filter: provider_route + provider_id: "provider-c" + model_header: X-Model + emit_demo_attribution: true + routes: + - candidate_id: "b5a496b1" + model: Qwen/Qwen3-0.6B + paths: + - /v1/chat/completions + - /v1/responses + cluster: vcr-backend-c + credential: + strategy: bearer_token + secretRef: + name: vcr-inference-credential + namespace: grid-system + key: token + - filter: credential_inject + credentials: + - strategy: bearer_token + name: vcr-inference-credential + namespace: grid-system + key: token + file: /etc/praxis/credentials/vcr-inference/token + - filter: load_balancer + clusters: + - name: vcr-backend-c + endpoints: + - "vcr-inference-provider-c.grid-system.svc.cluster.local:8000" + +admin: + address: "127.0.0.1:9901" +shutdown_timeout_secs: 5 diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml new file mode 100644 index 0000000..bdb7a2b --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml @@ -0,0 +1,543 @@ +apiVersion: forge.praxis.dev/v1alpha1 +kind: Environment + +metadata: + name: grid-single-cluster-multi-gateway + +spec: + runtime: + provider: docker + clusterPrefix: grid-single-cluster-multi-gateway + + network: + # Forge's MetalLB address-pool helper requires this capability flag even + # for a one-cluster environment; the topology below still has one cluster + # and one GridSite and does not exercise cross-cluster traffic. + crossCluster: true + dnsZone: grid-single-cluster-multi-gateway.test + + clusters: + - name: single + stacks: [metallb, tls-bootstrap, provider-a-operator-base, vcr-backend, provider-a-site, provider-gateway-a, provider-gateway-b, provider-gateway-c, consumer-gateway-a, consumer-gateway-b] + properties: + region: single + role: combined + siteName: single + gatewayImage: "praxis-ai:single-cluster-qualification" + operatorImage: "grid-operator:single-cluster-qualification" + vcrImage: "ghcr.io/neuralmagic/vllm-vcr:vllm0.23" + imagePullPolicy: Never + gatewayImageRepo: "praxis-ai" + gatewayImageTag: "single-cluster-qualification" + operatorImageRepo: "grid-operator" + operatorImageTag: "single-cluster-qualification" + + stacks: + tls-bootstrap: + description: Ephemeral same-CA identities for this isolated qualification + steps: + - type: manifest + path: resources/common/grid-system-namespace.yaml + - type: exec + command: + - bash + - -c + - >- + set -eu; + d=$(mktemp -d); + trap 'rm -rf "${d}"' EXIT; + openssl req -x509 -newkey rsa:2048 -nodes -days 1 -subj "/O=ai-grid/CN=qualification-ca" -keyout "${d}/ca.key" -out "${d}/ca.crt" >/dev/null 2>&1; + openssl req -newkey rsa:2048 -nodes -subj "/O=ai-grid/CN=grid-gateway.grid.internal" -keyout "${d}/gateway.key" -out "${d}/gateway.csr" >/dev/null 2>&1; + printf "subjectAltName=DNS:grid-gateway.grid.internal,DNS:provider-a.grid.internal,DNS:provider-b.grid.internal,DNS:provider-c.grid.internal,DNS:grid-system.svc.cluster.local" > "${d}/ext.cnf"; + openssl x509 -req -days 1 -in "${d}/gateway.csr" -CA "${d}/ca.crt" -CAkey "${d}/ca.key" -CAcreateserial -out "${d}/gateway.crt" -extfile "${d}/ext.cnf" >/dev/null 2>&1; + kubectl --context kind-grid-single-cluster-multi-gateway-single -n grid-system create secret generic consumer-gateway-tls --from-file=ca.crt="${d}/ca.crt" --from-file=tls.crt="${d}/gateway.crt" --from-file=tls.key="${d}/gateway.key" --dry-run=client -o yaml | kubectl --context kind-grid-single-cluster-multi-gateway-single apply -f - >/dev/null; + kubectl --context kind-grid-single-cluster-multi-gateway-single -n grid-system create secret generic vcr-inference-credential --from-literal=token="$(openssl rand -hex 32)" --dry-run=client -o yaml | kubectl --context kind-grid-single-cluster-multi-gateway-single apply -f - >/dev/null + metallb: + description: MetalLB load balancer with auto-configured address pool + steps: + - type: url + url: https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-native.yaml + sha256: 951065e85692aa106f1bb5d5a487d9306154923a794ab1d82122881cbaf588e4 + - type: wait + resource: deployment/controller + namespace: metallb-system + condition: available + timeout: "120s" + - type: metallb-auto-pool + name: forge-pool + + provider-a-operator-base: + description: Single Grid operator for the shared Kind control plane + steps: + - type: helm + release: grid-operator + chart: charts/grid-operator + version: "0.1.0" + namespace: grid-system + values: + image: + repository: "{{ cluster.properties.operatorImageRepo }}" + tag: "{{ cluster.properties.operatorImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + swim: + siteName: "single" + seeds: "" + service: + enabled: true + type: "LoadBalancer" + gateway: + serviceName: "provider-gateway-a" + port: "8443" + - type: wait + resource: deployment/grid-operator + namespace: grid-system + condition: available + timeout: "120s" + - type: helm + release: grid-operator + chart: charts/grid-operator + version: "0.1.0" + namespace: grid-system + values: + image: + repository: "{{ cluster.properties.operatorImageRepo }}" + tag: "{{ cluster.properties.operatorImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + swim: + siteName: "single" + seeds: "" + advertiseAddress: "172.18.255.231:7946" + service: + enabled: true + type: "LoadBalancer" + gateway: + serviceName: "provider-gateway-a" + port: "8443" + + vcr-backend: + description: vllm-vcr inference backend for demo scenarios + steps: + - type: manifest + path: resources/common/grid-system-namespace.yaml + - type: template-manifest + path: resources/common/vcr-provider-a.yaml + - type: template-manifest + path: resources/common/vcr-provider-b.yaml + - type: template-manifest + path: resources/common/vcr-provider-c.yaml + - type: manifest + path: resources/common/backend-network-policy.yaml + - type: manifest + path: resources/common/qualification-client.yaml + - type: wait + resource: deployment/vcr-inference-provider-a + namespace: grid-system + condition: available + timeout: "120s" + + provider-a-site: + description: Single GridSite with Grid CRs via grid-site chart + steps: + - type: helm + release: grid-site-a + chart: charts/grid-site + version: "0.1.0" + namespace: grid-system + values: + commonLabels: + grid.praxis-proxy.io/auto-discover-sites: "true" + gridNetwork: + name: grid-single-cluster-multi-gateway + gridId: grid-single-cluster-multi-gateway-v1 + region: provider-a + zone: provider-a-1 + routingPolicy: scoreFirst + scoringPolicy: + strategy: noMetrics + selectionPolicy: + mode: roundRobin + swim: + probeInterval: "5s" + suspicionTimeout: "15s" + gossipNodes: 3 + tls: + caSecretRef: + name: consumer-gateway-tls + namespace: grid-system + siteSecretRef: + name: consumer-gateway-tls + namespace: grid-system + gatewayRefs: + - name: consumer-gateway-a + namespace: grid-system + localSiteName: single + - name: consumer-gateway-b + namespace: grid-system + localSiteName: single + gridSite: + name: single + region: single + zone: single-1 + providerSiteLabel: single + inferenceProviders: + - name: vcr-provider-a-provider + gridNetworkRef: grid-single-cluster-multi-gateway + providerKind: vllm-vcr + backendKind: local + endpoint: "http://vcr-inference-provider-a.grid-system.svc.cluster.local:8000" + siteSelector: + matchLabels: + grid.praxis-proxy.io/provider-site: single + accessPolicy: + siteSelector: + matchLabels: {} + models: + - name: Qwen/Qwen3-0.6B + capabilities: + - text_generation + contextWindow: 4096 + # In a single-site deployment, the Grid operator relies on its + # provider health checks rather than multi-site SWIM failure + # detection. This qualification tunes convergence to 10s + # instead of the 30s production default. + healthCheck: + path: /health + interval: "10s" + timeout: "5s" + - name: vcr-provider-b-provider + gridNetworkRef: grid-single-cluster-multi-gateway + providerKind: vllm-vcr + backendKind: local + endpoint: "http://vcr-inference-provider-b.grid-system.svc.cluster.local:8000" + siteSelector: + matchLabels: {} + accessPolicy: + siteSelector: + matchLabels: {} + models: + - name: Qwen/Qwen3-0.6B + capabilities: + - text_generation + contextWindow: 4096 + healthCheck: + path: /health + interval: "10s" + timeout: "5s" + - name: vcr-provider-c-provider + gridNetworkRef: grid-single-cluster-multi-gateway + providerKind: vllm-vcr + backendKind: local + endpoint: "http://vcr-inference-provider-c.grid-system.svc.cluster.local:8000" + siteSelector: + matchLabels: {} + accessPolicy: + siteSelector: + matchLabels: {} + models: + - name: Qwen/Qwen3-0.6B + capabilities: + - text_generation + contextWindow: 4096 + healthCheck: + path: /health + interval: "10s" + timeout: "5s" + + provider-gateway-a: + description: Praxis provider gateway with mTLS and credential mounts + steps: + - type: template-file + source: configs/provider/praxis-a.yaml + target: .forge/runtime/{{ cluster.name }}/provider-a-praxis.yaml + - type: exec + command: [bash, -c, "kubectl --context kind-grid-single-cluster-multi-gateway-single -n grid-system create configmap provider-gateway-a-config --from-file=praxis.yaml=.forge/runtime/{{ cluster.name }}/provider-a-praxis.yaml --dry-run=client -o yaml | kubectl --context kind-grid-single-cluster-multi-gateway-single apply -f -"] + - type: helm + release: provider-gateway-a + chart: charts/praxis-gateway + version: "0.1.0" + namespace: grid-system + values: + fullnameOverride: "provider-gateway-a" + image: + repository: "{{ cluster.properties.gatewayImageRepo }}" + tag: "{{ cluster.properties.gatewayImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + podSecurityContext: + runAsUser: 100 + runAsGroup: 101 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + config: + existingConfigMap: "provider-gateway-a-config" + port: + containerPort: 8443 + name: "https-mtls" + service: + type: "LoadBalancer" + port: 8443 + health: + readiness: + tcpSocket: + port: "https-mtls" + initialDelaySeconds: 3 + periodSeconds: 5 + liveness: + tcpSocket: + port: "https-mtls" + initialDelaySeconds: 5 + periodSeconds: 10 + tls: + enabled: true + existingSecret: "consumer-gateway-tls" + credentials: + - name: "vcr-inference-credential" + mountPath: "/etc/praxis/credentials/vcr-inference" + podLabels: + grid.praxis-proxy.io/backend-access: "provider-gateway" + grid.praxis-proxy.io/provider-site: "{{ cluster.name }}" + - type: wait + resource: deployment/provider-gateway-a + namespace: grid-system + condition: available + timeout: "120s" + + provider-gateway-b: + description: Praxis provider gateway with mTLS and credential mounts + steps: + - type: template-file + source: configs/provider/praxis-b.yaml + target: .forge/runtime/{{ cluster.name }}/provider-b-praxis.yaml + - type: exec + command: [bash, -c, "kubectl --context kind-grid-single-cluster-multi-gateway-single -n grid-system create configmap provider-gateway-b-config --from-file=praxis.yaml=.forge/runtime/{{ cluster.name }}/provider-b-praxis.yaml --dry-run=client -o yaml | kubectl --context kind-grid-single-cluster-multi-gateway-single apply -f -"] + - type: helm + release: provider-gateway-b + chart: charts/praxis-gateway + version: "0.1.0" + namespace: grid-system + values: + fullnameOverride: "provider-gateway-b" + replicaCount: 1 + image: + repository: "{{ cluster.properties.gatewayImageRepo }}" + tag: "{{ cluster.properties.gatewayImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + podSecurityContext: + runAsUser: 100 + runAsGroup: 101 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + config: + existingConfigMap: "provider-gateway-b-config" + port: + containerPort: 8443 + name: "https-mtls" + service: + type: "LoadBalancer" + port: 8443 + health: + readiness: + tcpSocket: + port: "https-mtls" + initialDelaySeconds: 3 + periodSeconds: 5 + liveness: + tcpSocket: + port: "https-mtls" + initialDelaySeconds: 5 + periodSeconds: 10 + tls: + enabled: true + existingSecret: "consumer-gateway-tls" + credentials: + - name: "vcr-inference-credential" + mountPath: "/etc/praxis/credentials/vcr-inference" + podLabels: + grid.praxis-proxy.io/backend-access: "provider-gateway" + grid.praxis-proxy.io/provider-site: "{{ cluster.name }}" + - type: wait + resource: deployment/provider-gateway-b + namespace: grid-system + condition: available + timeout: "120s" + + provider-gateway-c: + description: Praxis provider gateway with mTLS and credential mounts + steps: + - type: template-file + source: configs/provider/praxis-c.yaml + target: .forge/runtime/{{ cluster.name }}/provider-c-praxis.yaml + - type: exec + command: [bash, -c, "kubectl --context kind-grid-single-cluster-multi-gateway-single -n grid-system create configmap provider-gateway-c-config --from-file=praxis.yaml=.forge/runtime/{{ cluster.name }}/provider-c-praxis.yaml --dry-run=client -o yaml | kubectl --context kind-grid-single-cluster-multi-gateway-single apply -f -"] + - type: helm + release: provider-gateway-c + chart: charts/praxis-gateway + version: "0.1.0" + namespace: grid-system + values: + fullnameOverride: "provider-gateway-c" + image: + repository: "{{ cluster.properties.gatewayImageRepo }}" + tag: "{{ cluster.properties.gatewayImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + podSecurityContext: + runAsUser: 100 + runAsGroup: 101 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + config: + existingConfigMap: "provider-gateway-c-config" + port: + containerPort: 8443 + name: "https-mtls" + service: + type: "LoadBalancer" + port: 8443 + health: + readiness: + tcpSocket: + port: "https-mtls" + initialDelaySeconds: 3 + periodSeconds: 5 + liveness: + tcpSocket: + port: "https-mtls" + initialDelaySeconds: 5 + periodSeconds: 10 + tls: + enabled: true + existingSecret: "consumer-gateway-tls" + credentials: + - name: "vcr-inference-credential" + mountPath: "/etc/praxis/credentials/vcr-inference" + podLabels: + grid.praxis-proxy.io/backend-access: "provider-gateway" + grid.praxis-proxy.io/provider-site: "{{ cluster.name }}" + - type: wait + resource: deployment/provider-gateway-c + namespace: grid-system + condition: available + timeout: "120s" + + consumer-gateway-a: + description: Praxis consumer gateway with operator-managed overlay + steps: + - type: template-file + source: configs/consumer/praxis-a.yaml + target: .forge/runtime/{{ cluster.name }}/consumer/praxis.yaml + - type: exec + command: + - bash + - -c + - >- + kubectl --context kind-grid-single-cluster-multi-gateway-{{ cluster.name }} -n grid-system + create configmap consumer-gateway-a-config + --from-file=praxis.yaml=.forge/runtime/{{ cluster.name }}/consumer/praxis.yaml + --dry-run=client -o yaml | + kubectl --context kind-grid-single-cluster-multi-gateway-{{ cluster.name }} apply -f - + - type: helm + release: consumer-gateway-a + chart: charts/praxis-gateway + version: "0.1.0" + namespace: grid-system + values: + fullnameOverride: "consumer-gateway-a" + image: + repository: "{{ cluster.properties.gatewayImageRepo }}" + tag: "{{ cluster.properties.gatewayImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + podSecurityContext: + runAsUser: 100 + runAsGroup: 101 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + config: + existingConfigMap: "consumer-gateway-a-config" + service: + type: "ClusterIP" + overlay: + enabled: true + existingConfigMap: "grid-overlay-grid-single-cluster--consumer-gateway-a-db750bac" + tls: + enabled: true + existingSecret: "consumer-gateway-tls" + podLabels: + grid.praxis-proxy.io/consumer-site: "{{ cluster.name }}" + - type: wait + resource: deployment/consumer-gateway-a + namespace: grid-system + condition: available + timeout: "120s" + consumer-gateway-b: + description: Praxis consumer gateway with operator-managed overlay + steps: + - type: template-file + source: configs/consumer/praxis-b.yaml + target: .forge/runtime/{{ cluster.name }}/consumer/praxis.yaml + - type: exec + command: + - bash + - -c + - >- + kubectl --context kind-grid-single-cluster-multi-gateway-{{ cluster.name }} -n grid-system + create configmap consumer-gateway-b-config + --from-file=praxis.yaml=.forge/runtime/{{ cluster.name }}/consumer/praxis.yaml + --dry-run=client -o yaml | + kubectl --context kind-grid-single-cluster-multi-gateway-{{ cluster.name }} apply -f - + - type: helm + release: consumer-gateway-b + chart: charts/praxis-gateway + version: "0.1.0" + namespace: grid-system + values: + fullnameOverride: "consumer-gateway-b" + image: + repository: "{{ cluster.properties.gatewayImageRepo }}" + tag: "{{ cluster.properties.gatewayImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + podSecurityContext: + runAsUser: 100 + runAsGroup: 101 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + config: + existingConfigMap: "consumer-gateway-b-config" + service: + type: "ClusterIP" + overlay: + enabled: true + existingConfigMap: "grid-overlay-grid-single-cluster--consumer-gateway-b-de751065" + tls: + enabled: true + existingSecret: "consumer-gateway-tls" + podLabels: + grid.praxis-proxy.io/consumer-site: "{{ cluster.name }}" + - type: wait + resource: deployment/consumer-gateway-b + namespace: grid-system + condition: available + timeout: "120s" diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/backend-network-policy.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/backend-network-policy.yaml new file mode 100644 index 0000000..f0cecbd --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/backend-network-policy.yaml @@ -0,0 +1,43 @@ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: vcr-inference-allow-provider-gateway-only + namespace: grid-system + labels: + app.kubernetes.io/part-of: grid-provider-traffic +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: vllm-vcr + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + grid.praxis-proxy.io/backend-access: "provider-gateway" + ports: + - protocol: TCP + port: 8000 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: vcr-inference-allow-grid-operator-health + namespace: grid-system + labels: + app.kubernetes.io/part-of: grid-provider-traffic +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: vllm-vcr + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + app.kubernetes.io/name: grid-operator + ports: + - protocol: TCP + port: 8000 diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/grid-system-namespace.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/grid-system-namespace.yaml new file mode 100644 index 0000000..b816c84 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/grid-system-namespace.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: grid-system + labels: + name: grid-system + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/audit: restricted + pod-security.kubernetes.io/warn: restricted diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/qualification-client.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/qualification-client.yaml new file mode 100644 index 0000000..030a2b8 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/qualification-client.yaml @@ -0,0 +1,24 @@ +apiVersion: v1 +kind: Pod +metadata: + name: qualification-client + namespace: grid-system + labels: + app.kubernetes.io/name: qualification-client +spec: + automountServiceAccountToken: false + restartPolicy: Never + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + containers: + - name: qualification-client + image: curlimages/curl:8.12.1 + command: ["sleep", "3600"] + securityContext: + runAsUser: 100 + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-a.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-a.yaml new file mode 100644 index 0000000..f3f936f --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-a.yaml @@ -0,0 +1,117 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vcr-inference-provider-a + namespace: grid-system + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/part-of: grid-provider-traffic +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-a + template: + metadata: + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-a + spec: + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: vcr + image: "{{ cluster.properties.vcrImage }}" + imagePullPolicy: "{{ cluster.properties.imagePullPolicy }}" + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + env: + - name: MODEL + value: "Qwen/Qwen3-0.6B" + - name: MOCK_PD_ROLE + value: "both" + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_PORT + value: "8000" + - name: MOCK_MAX_NUM_SEQS + value: "4" + - name: MOCK_KV_CACHE_SIZE + value: "64" + - name: MOCK_TOKENS_PER_BLOCK + value: "16" + - name: MOCK_MAX_MODEL_LEN + value: "512" + - name: MOCK_TTFT_MS + value: "50" + - name: MOCK_ITL_MS + value: "20" + - name: MOCK_TIME_FACTOR_UNDER_LOAD + value: "2.0" + ports: + - name: http + containerPort: 8000 + startupProbe: + httpGet: + path: /v1/models + port: 8000 + initialDelaySeconds: 5 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 60 + readinessProbe: + httpGet: + path: /health + port: 8000 + periodSeconds: 5 + timeoutSeconds: 3 + livenessProbe: + httpGet: + path: /health + port: 8000 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + resources: + requests: + cpu: 500m + memory: 512Mi + limits: + memory: 2Gi +--- +apiVersion: v1 +kind: Service +metadata: + name: vcr-inference-provider-a + namespace: grid-system + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/part-of: grid-provider-traffic +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-a + ports: + - name: http + port: 8000 + targetPort: 8000 diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-b.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-b.yaml new file mode 100644 index 0000000..c5bfc25 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-b.yaml @@ -0,0 +1,117 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vcr-inference-provider-b + namespace: grid-system + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/part-of: grid-provider-traffic +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-b + template: + metadata: + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-b + spec: + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: vcr + image: "{{ cluster.properties.vcrImage }}" + imagePullPolicy: "{{ cluster.properties.imagePullPolicy }}" + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + env: + - name: MODEL + value: "Qwen/Qwen3-0.6B" + - name: MOCK_PD_ROLE + value: "both" + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_PORT + value: "8000" + - name: MOCK_MAX_NUM_SEQS + value: "4" + - name: MOCK_KV_CACHE_SIZE + value: "64" + - name: MOCK_TOKENS_PER_BLOCK + value: "16" + - name: MOCK_MAX_MODEL_LEN + value: "512" + - name: MOCK_TTFT_MS + value: "50" + - name: MOCK_ITL_MS + value: "20" + - name: MOCK_TIME_FACTOR_UNDER_LOAD + value: "2.0" + ports: + - name: http + containerPort: 8000 + startupProbe: + httpGet: + path: /v1/models + port: 8000 + initialDelaySeconds: 5 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 60 + readinessProbe: + httpGet: + path: /health + port: 8000 + periodSeconds: 5 + timeoutSeconds: 3 + livenessProbe: + httpGet: + path: /health + port: 8000 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + resources: + requests: + cpu: 500m + memory: 512Mi + limits: + memory: 2Gi +--- +apiVersion: v1 +kind: Service +metadata: + name: vcr-inference-provider-b + namespace: grid-system + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/part-of: grid-provider-traffic +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-b + ports: + - name: http + port: 8000 + targetPort: 8000 diff --git a/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-c.yaml b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-c.yaml new file mode 100644 index 0000000..375c921 --- /dev/null +++ b/tests/e2e/topologies/grid-single-cluster-multi-gateway/resources/common/vcr-provider-c.yaml @@ -0,0 +1,117 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vcr-inference-provider-c + namespace: grid-system + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/part-of: grid-provider-traffic +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-c + template: + metadata: + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-c + spec: + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: vcr + image: "{{ cluster.properties.vcrImage }}" + imagePullPolicy: "{{ cluster.properties.imagePullPolicy }}" + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + env: + - name: MODEL + value: "Qwen/Qwen3-0.6B" + - name: MOCK_PD_ROLE + value: "both" + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_PORT + value: "8000" + - name: MOCK_MAX_NUM_SEQS + value: "4" + - name: MOCK_KV_CACHE_SIZE + value: "64" + - name: MOCK_TOKENS_PER_BLOCK + value: "16" + - name: MOCK_MAX_MODEL_LEN + value: "512" + - name: MOCK_TTFT_MS + value: "50" + - name: MOCK_ITL_MS + value: "20" + - name: MOCK_TIME_FACTOR_UNDER_LOAD + value: "2.0" + ports: + - name: http + containerPort: 8000 + startupProbe: + httpGet: + path: /v1/models + port: 8000 + initialDelaySeconds: 5 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 60 + readinessProbe: + httpGet: + path: /health + port: 8000 + periodSeconds: 5 + timeoutSeconds: 3 + livenessProbe: + httpGet: + path: /health + port: 8000 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + resources: + requests: + cpu: 500m + memory: 512Mi + limits: + memory: 2Gi +--- +apiVersion: v1 +kind: Service +metadata: + name: vcr-inference-provider-c + namespace: grid-system + labels: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/part-of: grid-provider-traffic +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: vllm-vcr + app.kubernetes.io/instance: provider-c + ports: + - name: http + port: 8000 + targetPort: 8000 diff --git a/xtask/src/env.rs b/xtask/src/env.rs index ac048a7..8cb192b 100644 --- a/xtask/src/env.rs +++ b/xtask/src/env.rs @@ -19,6 +19,7 @@ pub(crate) mod operator; pub(crate) mod operator_overlay; pub(crate) mod provider_traffic_qualification; pub(crate) mod providers; +pub(crate) mod single_cluster_multi_gateway_qualification; pub(crate) mod token_rate_limit_qualification; pub(crate) mod trust; pub(crate) mod verify; @@ -1022,6 +1023,22 @@ pub(crate) enum Action { options: GlbDemoOptions, }, + /// Qualify multiple consumer and provider gateways in one Kind cluster. + RunGridSingleClusterMultiGatewayQualification { + /// Path to the single-cluster Forge topology. + #[arg( + long, + default_value = "tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml" + )] + forge_config: PathBuf, + /// Keep the cluster after the qualification. + #[arg(long)] + keep: bool, + /// Evidence output directory. + #[arg(long)] + evidence_dir: Option, + }, + /// Run the distributed token-rate-limit topology with structured evidence. RunGridTokenRateLimitQualification { /// Path to the Forge environment config file. @@ -1170,6 +1187,18 @@ pub(crate) fn run(action: &Action) -> Result<(), Box> { Action::RunGridProviderTrafficQualification { forge_config, options } => { provider_traffic_qualification::run(forge_config, options) }, + Action::RunGridSingleClusterMultiGatewayQualification { + forge_config, + keep, + evidence_dir, + } => single_cluster_multi_gateway_qualification::run( + forge_config, + &single_cluster_multi_gateway_qualification::Options { + forge_config: forge_config.clone(), + keep: *keep, + evidence_dir: evidence_dir.clone(), + }, + ), Action::RunGridTokenRateLimitQualification { forge_config, evidence_dir, diff --git a/xtask/src/env/single_cluster_multi_gateway_qualification.rs b/xtask/src/env/single_cluster_multi_gateway_qualification.rs new file mode 100644 index 0000000..8a2f131 --- /dev/null +++ b/xtask/src/env/single_cluster_multi_gateway_qualification.rs @@ -0,0 +1,1200 @@ +//! Single-cluster, multi-gateway integration qualification. +//! +//! This qualification deliberately keeps all provider and consumer processes +//! in one Kind cluster. It proves the shared Kubernetes/overlay boundary while +//! recording that round-robin cursors remain local to each consumer process. + +#![allow( + clippy::missing_docs_in_private_items, + reason = "internal evidence model is documented by its serialized schema" +)] + +use std::{ + collections::{BTreeMap, BTreeSet}, + fs, + path::{Path, PathBuf}, + process::{Child, Command, Output, Stdio}, + thread, + time::{Duration, Instant, SystemTime, UNIX_EPOCH}, +}; + +use serde::Serialize; + +const TOPOLOGY: &str = "tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml"; +const CLUSTER: &str = "single"; +const CLUSTER_PREFIX: &str = "grid-single-cluster-multi-gateway"; +const NAMESPACE: &str = "grid-system"; +const QUALIFICATION_TIMEOUT: Duration = Duration::from_secs(180); +const POLL_INTERVAL: Duration = Duration::from_secs(2); + +/// Names used by the Forge, Kind, Kubernetes, and Docker layers. +#[derive(Debug, Clone, PartialEq, Eq)] +struct ClusterIdentity { + forge_cluster: &'static str, + kind_cluster: String, + kubectl_context: String, + node_container: String, +} + +fn cluster_identity() -> ClusterIdentity { + let kind_cluster = format!("{CLUSTER_PREFIX}-{CLUSTER}"); + ClusterIdentity { + forge_cluster: CLUSTER, + kubectl_context: format!("kind-{kind_cluster}"), + node_container: format!("{kind_cluster}-control-plane"), + kind_cluster, + } +} + +/// CLI options for the single-cluster qualification. +#[derive(Debug, clap::Args)] +pub(crate) struct Options { + /// Forge topology to deploy. + #[arg(long, default_value = TOPOLOGY)] + pub(crate) forge_config: PathBuf, + /// Keep the cluster after the run for debugging. + #[arg(long)] + pub(crate) keep: bool, + /// Evidence directory. Defaults to a timestamped ignored directory. + #[arg(long)] + pub(crate) evidence_dir: Option, +} + +#[derive(Debug, Serialize)] +struct Scenario { + name: String, + result: String, + detail: String, +} + +#[derive(Debug, Serialize)] +struct Evidence { + schema_version: u8, + result: String, + topology: String, + cluster: String, + source_revision: String, + scenarios: Vec, + observations: BTreeMap, + cleanup: String, +} + +struct Cleanup { + forge: PathBuf, + config: PathBuf, + enabled: bool, +} + +impl Drop for Cleanup { + fn drop(&mut self) { + if self.enabled { + drop( + Command::new(&self.forge) + .args(["down", "--config"]) + .arg(&self.config) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status(), + ); + } + } +} + +fn timestamp() -> String { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or_else(|_| "unknown".to_owned(), |d| d.as_secs().to_string()) +} + +fn evidence_path(config: &Path, requested: Option<&Path>, run: &str) -> PathBuf { + requested.map_or_else( + || { + config + .parent() + .unwrap_or_else(|| Path::new(".")) + .join("evidence") + .join(format!("single-cluster-multi-gateway-{run}")) + }, + Path::to_path_buf, + ) +} + +#[expect( + clippy::too_many_lines, + reason = "bounded process setup, polling, collection, and timeout handling stay together" +)] +fn command_output(command: &mut Command, timeout: Duration) -> Result { + let description = format!("{command:?}"); + let stdout_file = tempfile::NamedTempFile::new().map_err(|error| format!("stdout temp file: {error}"))?; + let stderr_file = tempfile::NamedTempFile::new().map_err(|error| format!("stderr temp file: {error}"))?; + let stdout_path = stdout_file.path().to_owned(); + let stderr_path = stderr_file.path().to_owned(); + command.stdout( + stdout_file + .as_file() + .try_clone() + .map_err(|error| format!("clone stdout handle: {error}"))?, + ); + command.stderr( + stderr_file + .as_file() + .try_clone() + .map_err(|error| format!("clone stderr handle: {error}"))?, + ); + let mut child: Child = command + .spawn() + .map_err(|error| format!("spawn {description}: {error}"))?; + let started = Instant::now(); + loop { + match child + .try_wait() + .map_err(|error| format!("wait {description}: {error}"))? + { + Some(_) => { + let status = child + .wait() + .map_err(|error| format!("collect {description}: {error}"))?; + return Ok(Output { + status, + stdout: fs::read(stdout_path).map_err(|error| format!("read stdout {description}: {error}"))?, + stderr: fs::read(stderr_path).map_err(|error| format!("read stderr {description}: {error}"))?, + }); + }, + None if started.elapsed() >= timeout => { + drop(child.kill()); + drop(child.wait()); + return Err(format!("timeout after {timeout:?}: {description}")); + }, + None => thread::park_timeout(Duration::from_millis(100)), + } + } +} + +fn kubectl(args: &[&str]) -> Result { + let mut command = Command::new("kubectl"); + let identity = cluster_identity(); + command.args(["--context", identity.kubectl_context.as_str()]); + command.args(args); + command_output(&mut command, QUALIFICATION_TIMEOUT) +} + +/// Run kubectl with owned arguments for operations containing dynamic values. +fn kubectl_owned(args: &[String]) -> Result { + let mut command = Command::new("kubectl"); + let identity = cluster_identity(); + command.args(["--context", identity.kubectl_context.as_str()]); + command.args(args); + command_output(&mut command, QUALIFICATION_TIMEOUT) +} + +/// Scale a deployment and wait for its observed state to settle. +fn scale_deployment(name: &str, replicas: u32) -> Result<(), String> { + let args = vec![ + "-n".to_owned(), + NAMESPACE.to_owned(), + "scale".to_owned(), + format!("deployment/{name}"), + format!("--replicas={replicas}"), + ]; + let output = kubectl_owned(&args)?; + if !output.status.success() { + return Err(String::from_utf8_lossy(&output.stderr).trim().to_owned()); + } + let started = Instant::now(); + loop { + let value = json_kubectl(&["-n", NAMESPACE, "get", "deployment", name, "-o", "json"])?; + let current = value + .get("status") + .and_then(|item| item.get("readyReplicas")) + .and_then(serde_json::Value::as_u64) + .unwrap_or(0); + if current == u64::from(replicas) { + return Ok(()); + } + if started.elapsed() >= QUALIFICATION_TIMEOUT { + return Err(format!( + "deployment/{name} did not reach readyReplicas={replicas}; last={current}" + )); + } + thread::park_timeout(POLL_INTERVAL); + } +} + +/// Capture the operator's current Grid and overlay resources. +fn capture_grid_state() -> Result { + let networks = json_kubectl(&["-n", NAMESPACE, "get", "gridnetworks", "-o", "json"])?; + let sites = json_kubectl(&["-n", NAMESPACE, "get", "gridsites", "-o", "json"])?; + let providers = json_kubectl(&["-n", NAMESPACE, "get", "inferenceproviders", "-o", "json"])?; + let overlays = json_kubectl(&["-n", NAMESPACE, "get", "configmaps", "-o", "json"])?; + Ok(serde_json::json!({ + "gridNetworks": networks.get("items").cloned().unwrap_or(serde_json::Value::Null), + "gridSites": sites.get("items").cloned().unwrap_or(serde_json::Value::Null), + "inferenceProviders": providers.get("items").cloned().unwrap_or(serde_json::Value::Null), + "configMaps": overlays.get("items").cloned().unwrap_or(serde_json::Value::Null), + })) +} + +/// Images required by the topology's `Never` pull policy. +const QUALIFICATION_IMAGES: [&str; 4] = [ + "grid-operator:single-cluster-qualification", + "grid-overlay-sync:single-cluster-qualification", + "praxis-ai:single-cluster-qualification", + "ghcr.io/neuralmagic/vllm-vcr:vllm0.23", +]; + +/// Match a Docker reference against the repository and tag columns from `crictl`. +fn node_has_image(listing: &str, image: &str) -> bool { + let (repository, tag) = image.rsplit_once(':').unwrap_or((image, "latest")); + listing.lines().any(|line| { + let fields = line.split_whitespace().collect::>(); + fields.first().is_some_and(|repository_field| { + (*repository_field == repository || *repository_field == format!("docker.io/library/{repository}")) + && fields.get(1).is_some_and(|tag_field| *tag_field == tag) + }) + }) +} + +/// Discover the Docker node name for a Kind cluster. +fn discover_node(kind_cluster: &str) -> Result { + let output = Command::new("kind") + .args(["get", "nodes", "--name", kind_cluster]) + .output() + .map_err(|error| format!("discover Kind nodes: {error}"))?; + if !output.status.success() { + return Err(format!( + "kind get nodes failed: {}", + String::from_utf8_lossy(&output.stderr).trim() + )); + } + String::from_utf8_lossy(&output.stdout) + .lines() + .map(str::trim) + .find(|line| !line.is_empty()) + .map(str::to_owned) + .ok_or_else(|| format!("Kind cluster {kind_cluster} has no discovered nodes")) +} + +/// Load and verify every local image before any Forge stack is applied. +#[expect( + clippy::too_many_lines, + reason = "each image is inspected, loaded, and verified before deployment" +)] +fn load_and_verify_images() -> Result<(), String> { + let identity = cluster_identity(); + let node = discover_node(&identity.kind_cluster)?; + for image in QUALIFICATION_IMAGES { + let mut inspect = Command::new("docker"); + inspect.args(["image", "inspect", image]); + let output = command_output(&mut inspect, Duration::from_secs(30))?; + if !output.status.success() { + return Err(format!("required local image is missing: {image}")); + } + let mut load = Command::new("kind"); + load.args(["load", "docker-image", image, "--name", &identity.kind_cluster]); + let load_output = command_output(&mut load, QUALIFICATION_TIMEOUT)?; + if !load_output.status.success() { + return Err(format!( + "failed loading {image}: {}", + String::from_utf8_lossy(&load_output.stderr).trim() + )); + } + let mut verify = Command::new("docker"); + verify.args(["exec", &node, "crictl", "images"]); + let verify_output = verify + .output() + .map_err(|error| format!("verify image {image}: {error}"))?; + let listing = String::from_utf8_lossy(&verify_output.stdout); + if !verify_output.status.success() || !node_has_image(&listing, image) { + return Err(format!( + "Kind node does not contain exact image reference {image}; status={}, stdout={listing}, stderr={}", + verify_output.status, + String::from_utf8_lossy(&verify_output.stderr) + )); + } + } + Ok(()) +} + +/// Apply one Forge stack and preserve its complete bounded result. +fn apply_stack(forge: &str, config: &Path, stack: &str, evidence_dir: &Path) -> Result<(), String> { + let started = timestamp(); + let mut command = Command::new(forge); + command.args([ + "--config", + config.to_str().unwrap_or_default(), + "--non-interactive", + "stack", + "apply", + CLUSTER, + stack, + ]); + let output = command_output(&mut command, QUALIFICATION_TIMEOUT)?; + let finished = timestamp(); + let status = if output.status.success() { "PASS" } else { "FAIL" }; + let record = format!( + "stack: {stack}\nstart: {started}\nend: {finished}\nstatus: {status}\n\n--- stdout ---\n{}\n--- stderr ---\n{}\n", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr), + ); + fs::write(evidence_dir.join(format!("stack-{stack}.txt")), record).map_err(|error| error.to_string())?; + if output.status.success() { + Ok(()) + } else { + Err(format!("stack {stack} failed")) + } +} + +/// Run `forge up` with bounded process-tree termination and live pipe draining. +fn forge_up(forge: &str, config: &Path, evidence_dir: &Path) -> Result<(), String> { + let config_arg = config.to_string_lossy().into_owned(); + let mut command = Command::new("timeout"); + command.args([ + "--signal=TERM", + "--kill-after=10s", + &format!("{}s", QUALIFICATION_TIMEOUT.as_secs()), + forge, + "--config", + &config_arg, + "--non-interactive", + "up", + ]); + let output = command + .output() + .map_err(|error| format!("spawn bounded forge up: {error}"))?; + let record = format!( + "forge up\nstatus: {}\n\n--- stdout ---\n{}\n--- stderr ---\n{}\n", + output.status, + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr), + ); + fs::write(evidence_dir.join("forge-up.txt"), record).map_err(|error| error.to_string())?; + if output.status.success() { + Ok(()) + } else { + Err(format!("bounded forge up failed or timed out: {}", output.status)) + } +} + +/// Capture setup-boundary diagnostics without exposing kubeconfig or secrets. +#[expect( + clippy::too_many_lines, + reason = "the setup failure record intentionally captures each boundary in one place" +)] +fn capture_setup_diagnostics(evidence_dir: &Path) { + let identity = cluster_identity(); + let commands = [ + ("kind-clusters.txt", "kind", vec!["get", "clusters"]), + ( + "kind-nodes.txt", + "kind", + vec!["get", "nodes", "--name", identity.kind_cluster.as_str()], + ), + ( + "node-state.json", + "kubectl", + vec![ + "--context", + identity.kubectl_context.as_str(), + "get", + "nodes", + "-o", + "json", + ], + ), + ( + "docker-network.json", + "docker", + vec!["network", "inspect", "grid-single-cluster-multi-gateway-net"], + ), + ("process-tree.txt", "ps", vec!["-eo", "pid,ppid,stat,etime,cmd"]), + ( + "docker-node-inspect.json", + "docker", + vec!["inspect", identity.node_container.as_str()], + ), + ]; + for (name, program, args) in commands { + let mut command = Command::new(program); + command.args(args); + let output = command_output(&mut command, Duration::from_secs(15)); + let text = output.map_or_else( + |error| format!("command failed: {error}"), + |item| { + format!( + "status: {}\n{}{}", + item.status, + String::from_utf8_lossy(&item.stdout), + String::from_utf8_lossy(&item.stderr) + ) + }, + ); + drop(fs::write(evidence_dir.join(name), text)); + } +} + +/// Run a command in the long-lived restricted qualification client. +fn client_command(args: &[&str]) -> Result { + let mut command = Command::new("kubectl"); + let identity = cluster_identity(); + command.args(["--context", identity.kubectl_context.as_str()]); + command.args(["-n", NAMESPACE, "exec", "qualification-client", "--"]); + command.args(args); + command_output(&mut command, QUALIFICATION_TIMEOUT) +} + +/// Issue one attributed request through a consumer gateway. +fn attributed_request(consumer: &str, request_id: u32) -> Result { + // Consumer services expose the client-facing HTTP listener on 8080. The + // 8443 listener is used by provider gateways for their mTLS hop. + let service = format!("http://{consumer}.grid-system.svc.cluster.local:8080/v1/chat/completions"); + let body = format!( + r#"{{"model":"Qwen/Qwen3-0.6B","messages":[{{"role":"user","content":"qualification-{request_id}"}}],"max_tokens":4}}"# + ); + let args = [ + "curl", + "--silent", + "--show-error", + "--max-time", + "10", + "--dump-header", + "/dev/stderr", + "--header", + "Content-Type: application/json", + "--header", + "Authorization: Bearer qualification-token", + "--data", + body.as_str(), + service.as_str(), + ]; + let output = client_command(&args)?; + if !output.status.success() { + return Err(format!( + "{consumer} request failed: {}", + String::from_utf8_lossy(&output.stderr).trim() + )); + } + Ok(String::from_utf8_lossy(&output.stderr).into_owned()) +} + +/// Extract the selected provider from the trusted response header. +fn selected_provider(headers: &str) -> Result { + headers + .lines() + .find_map(|line| line.strip_prefix("x-ai-demo-provider-gateway: ")) + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_owned) + .ok_or_else(|| format!("response did not contain trusted provider attribution: {headers}")) +} + +/// Parse the routing overlay for one consumer from captured `ConfigMaps`. +fn consumer_overlay(state: &serde_json::Value, consumer: &str) -> Result { + state + .get("configMaps") + .and_then(serde_json::Value::as_array) + .and_then(|items| { + items.iter().find(|item| { + item.get("metadata") + .and_then(|metadata| metadata.get("name")) + .and_then(serde_json::Value::as_str) + .is_some_and(|name| name.contains(consumer)) + && item + .get("data") + .and_then(|data| data.get("routing-overlay.json")) + .is_some() + }) + }) + .and_then(|item| item.get("data")) + .and_then(|data| data.get("routing-overlay.json")) + .and_then(serde_json::Value::as_str) + .ok_or_else(|| format!("routing overlay ConfigMap for {consumer} was not found")) + .and_then(|raw| { + serde_json::from_str(raw).map_err(|error| format!("invalid {consumer} routing overlay: {error}")) + }) +} + +/// Read the latest accepted and serving revisions reported by a consumer. +fn consumer_revisions(consumer: &str) -> Result<(String, String), String> { + let args = vec![ + "-n".to_owned(), + NAMESPACE.to_owned(), + "logs".to_owned(), + format!("deployment/{consumer}"), + "--all-containers=true".to_owned(), + ]; + let output = kubectl_owned(&args)?; + if !output.status.success() { + return Err(format!( + "failed to read {consumer} logs: {}", + String::from_utf8_lossy(&output.stderr) + )); + } + let raw_logs = String::from_utf8_lossy(&output.stdout); + let logs = strip_ansi(&raw_logs); + let field = |name: &str| { + logs.lines().rev().find_map(|line| { + line.split_whitespace() + .find_map(|part| part.strip_prefix(&format!("{name}="))) + .map(|value| value.trim_matches('"').to_owned()) + }) + }; + let accepted = field("accepted_revision").ok_or_else(|| format!("{consumer} has no accepted_revision log"))?; + let serving = field("serving_revision").ok_or_else(|| format!("{consumer} has no serving_revision log"))?; + Ok((accepted, serving)) +} + +/// Remove terminal CSI sequences emitted by the structured log formatter. +fn strip_ansi(input: &str) -> String { + let mut output = String::with_capacity(input.len()); + let mut sequence = false; + for character in input.chars() { + if sequence { + if character.is_ascii_alphabetic() { + sequence = false; + } + } else if character == '\x1b' { + sequence = true; + } else { + output.push(character); + } + } + output +} + +/// Assert that both consumers serve the same accepted candidate snapshot. +#[expect( + clippy::too_many_lines, + reason = "the overlay contract is validated as one atomic evidence boundary" +)] +fn assert_overlay_contract(state: &serde_json::Value) -> Result { + let overlays = ["consumer-gateway-a", "consumer-gateway-b"] + .into_iter() + .map(|consumer| consumer_overlay(state, consumer).map(|overlay| (consumer, overlay))) + .collect::, _>>()?; + let revisions = overlays + .iter() + .map(|(consumer, overlay)| { + overlay + .get("revision") + .and_then(|revision| revision.get("value")) + .and_then(serde_json::Value::as_str) + .ok_or_else(|| format!("{consumer} overlay has no content revision")) + }) + .map(|revision| revision.map(str::to_owned)) + .collect::, _>>()?; + let [first_revision, second_revision] = revisions.as_slice() else { + return Err(format!("expected two consumer revisions, got {revisions:?}")); + }; + if first_revision != second_revision { + return Err(format!("consumer overlays disagree: {revisions:?}")); + } + let mut candidate_sets = Vec::new(); + for (consumer, overlay) in overlays { + let candidates = overlay + .get("overlay") + .and_then(|item| item.get("candidates")) + .and_then(serde_json::Value::as_array) + .ok_or_else(|| format!("{consumer} overlay has no candidates"))?; + let ids = candidates + .iter() + .map(|candidate| { + candidate + .get("cluster") + .and_then(serde_json::Value::as_str) + .ok_or_else(|| format!("{consumer} candidate has no cluster identity")) + }) + .collect::, _>>()?; + if ids + != BTreeSet::from([ + "vcr-provider-a-provider", + "vcr-provider-b-provider", + "vcr-provider-c-provider", + ]) + { + return Err(format!("{consumer} candidates are {ids:?}")); + } + candidate_sets.push(serde_json::json!({"consumer": consumer, "revision": first_revision, "candidates": ids})); + } + Ok(serde_json::Value::Array(candidate_sets)) +} + +/// Require both consumers to report the exact revision accepted by Grid. +fn assert_serving_revisions(state: &serde_json::Value) -> Result { + let mut evidence = Vec::new(); + for consumer in ["consumer-gateway-a", "consumer-gateway-b"] { + let overlay = consumer_overlay(state, consumer)?; + let expected = overlay + .get("revision") + .and_then(|revision| revision.get("value")) + .and_then(serde_json::Value::as_str) + .ok_or_else(|| format!("{consumer} overlay has no revision"))?; + let (accepted, serving) = consumer_revisions(consumer)?; + if accepted != expected || serving != expected { + return Err(format!( + "{consumer} revision mismatch: expected={expected}, accepted={accepted}, serving={serving}" + )); + } + evidence.push(serde_json::json!({ + "consumer": consumer, + "expected": expected, + "accepted": accepted, + "serving": serving, + })); + } + Ok(serde_json::Value::Array(evidence)) +} + +/// Wait until both consumer overlays contain exactly the requested provider set. +#[expect( + clippy::too_many_lines, + reason = "polling keeps candidate state and last error together" +)] +fn wait_for_candidate_set(expected: &BTreeSet<&str>) -> Result { + let started = Instant::now(); + let mut last_error = "no overlay observed".to_owned(); + loop { + if let Ok(state) = capture_grid_state() { + let mut ready = true; + for consumer in ["consumer-gateway-a", "consumer-gateway-b"] { + match consumer_overlay(&state, consumer).and_then(|overlay| { + let candidates = overlay + .get("overlay") + .and_then(|item| item.get("candidates")) + .and_then(serde_json::Value::as_array) + .ok_or_else(|| format!("{consumer} has no candidates"))?; + let ids = candidates + .iter() + .filter_map(|candidate| candidate.get("cluster").and_then(serde_json::Value::as_str)) + .collect::>(); + if &ids == expected { + Ok(()) + } else { + Err(format!("{consumer} candidates: {ids:?}")) + } + }) { + Ok(()) => {}, + Err(error) => { + ready = false; + last_error = error; + }, + } + } + if ready { + return Ok(state); + } + } + if started.elapsed() >= QUALIFICATION_TIMEOUT { + return Err(format!("overlay candidate set did not converge; last={last_error}")); + } + thread::park_timeout(POLL_INTERVAL); + } +} + +/// Verify that the restricted client cannot bypass the provider gateway. +fn direct_backend_probe() -> Result { + let output = client_command(&[ + "curl", + "--silent", + "--show-error", + "--fail", + "--max-time", + "5", + "http://vcr-inference-provider-a.grid-system.svc.cluster.local:8000/health", + ])?; + Ok(!output.status.success()) +} + +fn json_kubectl(args: &[&str]) -> Result { + let output = kubectl(args)?; + if !output.status.success() { + return Err(String::from_utf8_lossy(&output.stderr).trim().to_owned()); + } + serde_json::from_slice(&output.stdout).map_err(|error| format!("invalid kubectl JSON: {error}")) +} + +#[expect( + clippy::too_many_lines, + reason = "readiness polling keeps the last observed resource contract together" +)] +fn wait_deployment(name: &str) -> Result<(), String> { + let started = Instant::now(); + let args = ["-n", NAMESPACE, "get", "deployment", name, "-o", "json"]; + loop { + if started.elapsed() >= QUALIFICATION_TIMEOUT { + return Err(format!( + "deployment/{name} did not become ready; last state unavailable" + )); + } + if let Ok(value) = json_kubectl(&args) { + let desired = value + .get("spec") + .and_then(|item| item.get("replicas")) + .and_then(serde_json::Value::as_u64) + .unwrap_or(0); + let ready = value + .get("status") + .and_then(|item| item.get("readyReplicas")) + .and_then(serde_json::Value::as_u64) + .unwrap_or(0); + let observed = value + .get("status") + .and_then(|item| item.get("observedGeneration")) + .and_then(serde_json::Value::as_i64) + .unwrap_or(-1); + let generation = value + .get("metadata") + .and_then(|item| item.get("generation")) + .and_then(serde_json::Value::as_i64) + .unwrap_or(-2); + if desired > 0 && ready >= desired && observed >= generation { + return Ok(()); + } + } + thread::park_timeout(POLL_INTERVAL); + } +} + +fn scenario(name: &str, result: &str, detail: impl Into) -> Scenario { + Scenario { + name: name.to_owned(), + result: result.to_owned(), + detail: detail.into(), + } +} + +/// Run the qualification. +#[expect( + clippy::too_many_lines, + reason = "qualification phases are intentionally visible in execution order" +)] +#[expect( + clippy::cognitive_complexity, + reason = "the runner's ordered phase control flow is the qualification contract" +)] +#[expect( + clippy::large_stack_frames, + reason = "the qualification keeps bounded scenario and observation state together for final evidence" +)] +pub(crate) fn run(forge_config: &Path, options: &Options) -> Result<(), Box> { + let run = timestamp(); + let evidence_dir = evidence_path(forge_config, options.evidence_dir.as_deref(), &run); + fs::create_dir_all(&evidence_dir)?; + let forge = super::glb::resolve_forge_binary().ok_or("praxis-forge binary not found")?; + let source_revision = Command::new("git").args(["rev-parse", "HEAD"]).output().map_or_else( + |_| "unknown".to_owned(), + |output| String::from_utf8_lossy(&output.stdout).trim().to_owned(), + ); + let mut cleanup = Cleanup { + forge: forge.clone().into(), + config: forge_config.to_path_buf(), + enabled: !options.keep, + }; + let mut scenarios = Vec::new(); + let mut observations = BTreeMap::new(); + let identity = cluster_identity(); + observations.insert( + "cluster_identity".to_owned(), + serde_json::json!({ + "forgeCluster": identity.forge_cluster, + "kindCluster": identity.kind_cluster, + "kubectlContext": identity.kubectl_context, + "nodeContainer": identity.node_container, + }), + ); + + let up_result = forge_up(&forge, forge_config, &evidence_dir); + if let Err(error) = &up_result { + capture_setup_diagnostics(&evidence_dir); + scenarios.push(scenario("forge-up", "BLOCKED", error.clone())); + } + if up_result.is_ok() { + let images_ready = match load_and_verify_images() { + Ok(()) => { + scenarios.push(scenario( + "image-loading", + "PASS", + "all required local image references were loaded and verified in the Kind node", + )); + true + }, + Err(error) => { + capture_setup_diagnostics(&evidence_dir); + scenarios.push(scenario("image-loading", "FAIL", error)); + false + }, + }; + let mut stacks_ready = images_ready; + for stack in images_ready + .then_some([ + "metallb", + "tls-bootstrap", + "provider-a-operator-base", + "vcr-backend", + "provider-a-site", + "provider-gateway-a", + "provider-gateway-b", + "provider-gateway-c", + "consumer-gateway-a", + "consumer-gateway-b", + ]) + .into_iter() + .flatten() + { + match apply_stack(&forge, forge_config, stack, &evidence_dir) { + Ok(()) => scenarios.push(scenario(format!("stack/{stack}").as_str(), "PASS", "stack completed")), + Err(error) => { + stacks_ready = false; + let last_state = capture_grid_state().unwrap_or(serde_json::Value::Null); + observations.insert(format!("timeout_state_{stack}"), last_state); + scenarios.push(scenario(format!("stack/{stack}").as_str(), "FAIL", error)); + break; + }, + } + } + for deployment in stacks_ready + .then_some([ + "grid-operator", + "vcr-inference-provider-a", + "vcr-inference-provider-b", + "vcr-inference-provider-c", + "provider-gateway-a", + "provider-gateway-b", + "provider-gateway-c", + "consumer-gateway-a", + "consumer-gateway-b", + ]) + .into_iter() + .flatten() + { + match wait_deployment(deployment) { + Ok(()) => scenarios.push(scenario( + format!("ready/{deployment}").as_str(), + "PASS", + "observed generation is ready", + )), + Err(error) => scenarios.push(scenario(format!("ready/{deployment}").as_str(), "FAIL", error)), + } + } + if stacks_ready { + match kubectl(&[ + "wait", + "--for=jsonpath={.status.phase}=Running", + "pod/qualification-client", + "-n", + NAMESPACE, + "--timeout=120s", + ]) { + Ok(output) if output.status.success() => scenarios.push(scenario( + "ready/qualification-client", + "PASS", + "restricted long-lived probe pod is Running", + )), + Ok(output) => scenarios.push(scenario( + "ready/qualification-client", + "FAIL", + String::from_utf8_lossy(&output.stderr).trim().to_owned(), + )), + Err(error) => scenarios.push(scenario("ready/qualification-client", "FAIL", error)), + } + if let Ok(value) = json_kubectl(&["-n", NAMESPACE, "get", "deploy", "-o", "json"]) { + observations.insert( + "deployments".to_owned(), + value.get("items").cloned().unwrap_or(serde_json::Value::Null), + ); + } + if let Ok(value) = json_kubectl(&["-n", NAMESPACE, "get", "endpointslices", "-o", "json"]) { + observations.insert( + "endpointslices".to_owned(), + value.get("items").cloned().unwrap_or(serde_json::Value::Null), + ); + } + let all_candidates = BTreeSet::from([ + "vcr-provider-a-provider", + "vcr-provider-b-provider", + "vcr-provider-c-provider", + ]); + match wait_for_candidate_set(&all_candidates).and_then(|state| { + let contract = assert_overlay_contract(&state)?; + observations.insert("overlay_contract".to_owned(), contract); + Ok(state) + }) { + Ok(state) => { + observations.insert("grid_state_bootstrap".to_owned(), state); + scenarios.push(scenario( + "overlay-and-serving-revision", + "PASS", + "GridNetwork, GridSite, providers, and generated overlay resources captured after readiness", + )); + }, + Err(error) => scenarios.push(scenario("overlay-and-serving-revision", "FAIL", error)), + } + match capture_grid_state().and_then(|state| assert_serving_revisions(&state)) { + Ok(revisions) => { + observations.insert("serving_revisions".to_owned(), revisions); + scenarios.push(scenario( + "serving-revision-barrier", + "PASS", + "both consumers report the exact Grid overlay revision as accepted and serving", + )); + }, + Err(error) => scenarios.push(scenario("serving-revision-barrier", "FAIL", error)), + } + let mut request_evidence = BTreeMap::>::new(); + for consumer in ["consumer-gateway-a", "consumer-gateway-b"] { + let mut responses = Vec::new(); + for request_id in 0..6 { + match attributed_request(consumer, request_id) { + Ok(headers) => responses.push(headers), + Err(error) => responses.push(format!("ERROR: {error}")), + } + } + request_evidence.insert(consumer.to_owned(), responses); + } + observations.insert( + "attributed_requests".to_owned(), + serde_json::to_value(&request_evidence).unwrap_or(serde_json::Value::Null), + ); + let request_failures = request_evidence + .values() + .flatten() + .filter(|item| item.starts_with("ERROR:")) + .count(); + let provider_sequences = request_evidence + .iter() + .map(|(consumer, responses)| { + responses + .iter() + .map(|headers| selected_provider(headers)) + .collect::, _>>() + .map(|sequence| (consumer.clone(), sequence)) + }) + .collect::, _>>(); + observations.insert( + "provider_sequences".to_owned(), + serde_json::to_value(&provider_sequences).unwrap_or(serde_json::Value::Null), + ); + let sequence_valid = provider_sequences.as_ref().is_ok_and(|sequences| { + let expected_rotation = ["provider-a", "provider-b", "provider-c"]; + sequences.iter().all(|(_, sequence)| { + sequence.len() == 6 + && sequence.iter().enumerate().all(|(index, provider)| { + expected_rotation.get(index % expected_rotation.len()) == Some(&provider.as_str()) + }) + }) + }); + scenarios.push(if request_failures == 0 && sequence_valid { + scenario( + "provider-load-sharing", + "PASS", + "six bounded requests through each consumer followed the independent A/B/C rotation with trusted attribution", + ) + } else { + scenario( + "provider-load-sharing", + "FAIL", + format!("{request_failures} requests failed or provider rotation/attribution was invalid"), + ) + }); + let withdrawal_result = scale_deployment("vcr-inference-provider-b", 0) + .and_then(|()| { + let expected = BTreeSet::from(["vcr-provider-a-provider", "vcr-provider-c-provider"]); + wait_for_candidate_set(&expected) + }) + .and_then(|state| { + observations.insert("withdrawal_state".to_owned(), state); + let requests = ["consumer-gateway-a", "consumer-gateway-b"] + .into_iter() + .map(|consumer| { + attributed_request(consumer, 100).and_then(|headers| { + let provider = selected_provider(&headers)?; + if provider == "provider-b" { + Err("withdrawn provider-b received traffic".to_owned()) + } else { + Ok(headers) + } + }) + }) + .collect::, _>>()?; + observations.insert( + "withdrawal_requests".to_owned(), + serde_json::to_value(requests).unwrap_or(serde_json::Value::Null), + ); + scale_deployment("vcr-inference-provider-b", 1) + }) + .and_then(|()| { + let expected = BTreeSet::from([ + "vcr-provider-a-provider", + "vcr-provider-b-provider", + "vcr-provider-c-provider", + ]); + wait_for_candidate_set(&expected) + }) + .map(|state| { + observations.insert("restoration_state".to_owned(), state); + }); + scenarios.push(match withdrawal_result { + Ok(()) => scenario( + "withdrawal-restoration", + "PASS", + "provider gateway B was withdrawn, traffic continued through both consumers, and B was restored", + ), + Err(error) => scenario("withdrawal-restoration", "FAIL", error), + }); + let consumer_result = scale_deployment("consumer-gateway-a", 0) + .and_then(|()| attributed_request("consumer-gateway-b", 200).map(|_| ())) + .and_then(|()| scale_deployment("consumer-gateway-a", 1)) + .and_then(|()| attributed_request("consumer-gateway-a", 201).map(|_| ())); + scenarios.push(match consumer_result { + Ok(()) => scenario( + "consumer-failure", + "PASS", + "consumer A was removed and restored while consumer B continued serving", + ), + Err(error) => scenario("consumer-failure", "FAIL", error), + }); + let concurrent_results = ["consumer-gateway-a", "consumer-gateway-b"] + .into_iter() + .map(|consumer| { + thread::spawn(move || { + (0..4) + .map(|request_id| { + attributed_request(consumer, 300 + request_id).and_then(|headers| { + selected_provider(&headers).map(|provider| (request_id, provider)) + }) + }) + .collect::>() + }) + }) + .map(|handle| handle.join().unwrap_or_default()) + .collect::>(); + let concurrent_failures = concurrent_results + .iter() + .flatten() + .filter(|result| result.is_err()) + .count(); + let concurrent_providers = concurrent_results + .iter() + .flatten() + .filter_map(|result| result.as_ref().ok()) + .map(|(_, provider)| provider.as_str()) + .collect::>(); + observations.insert( + "concurrent_requests".to_owned(), + serde_json::to_value( + concurrent_results + .iter() + .map(|results| { + results + .iter() + .map(|result| match result { + Ok((request_id, provider)) => { + serde_json::json!({"requestId": request_id, "provider": provider}) + }, + Err(error) => serde_json::json!({"error": error}), + }) + .collect::>() + }) + .collect::>(), + ) + .unwrap_or(serde_json::Value::Null), + ); + scenarios.push(if concurrent_failures == 0 && concurrent_providers.len() == 3 { + scenario( + "concurrent-traffic", + "PASS", + "eight concurrent requests completed through both consumers with trusted attribution to all providers", + ) + } else { + scenario( + "concurrent-traffic", + "FAIL", + format!("{concurrent_failures} concurrent requests failed"), + ) + }); + match direct_backend_probe() { + Ok(true) => scenarios.push(scenario( + "security", + "PASS", + "restricted client reached the consumer path while direct provider-backend access was denied", + )), + Ok(false) => scenarios.push(scenario( + "security", + "FAIL", + "restricted client unexpectedly reached provider backend directly", + )), + Err(error) => scenarios.push(scenario("security", "FAIL", error)), + } + } + } + + let result = if scenarios.iter().any(|item| item.result == "FAIL") { + "FAIL" + } else if scenarios.iter().any(|item| item.result == "BLOCKED") { + "BLOCKED" + } else { + "PASS" + }; + let cleanup_result = if options.keep { + "kept by request" + } else { + "scheduled by cleanup guard" + }; + let evidence = Evidence { + schema_version: 1, + result: result.to_owned(), + topology: forge_config.display().to_string(), + cluster: cluster_identity().kind_cluster, + source_revision, + scenarios, + observations, + cleanup: cleanup_result.to_owned(), + }; + fs::write(evidence_dir.join("results.json"), serde_json::to_vec_pretty(&evidence)?)?; + fs::write( + evidence_dir.join("SUMMARY.md"), + format!("# Single-cluster multi-gateway qualification\n\nResult: **{result}**\n\nEvidence: `results.json`\n"), + )?; + if !options.keep { + let mut down = Command::new(&forge); + down.args(["down", "--config"]).arg(forge_config); + let output = + command_output(&mut down, QUALIFICATION_TIMEOUT).map_err(|error| format!("cleanup failed: {error}"))?; + if !output.status.success() { + return Err(format!("cleanup failed: {}", String::from_utf8_lossy(&output.stderr)).into()); + } + cleanup.enabled = false; + } + if result == "PASS" { + Ok(()) + } else { + Err(format!("qualification result: {result}; see {}", evidence_dir.display()).into()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn cluster_layers_use_distinct_names() { + let identity = cluster_identity(); + assert_eq!(identity.forge_cluster, "single"); + assert_eq!(identity.kind_cluster, "grid-single-cluster-multi-gateway-single"); + assert_eq!( + identity.kubectl_context, + "kind-grid-single-cluster-multi-gateway-single" + ); + assert_eq!( + identity.node_container, + "grid-single-cluster-multi-gateway-single-control-plane" + ); + assert!(!identity.node_container.starts_with("kind-")); + } + + #[test] + fn node_image_matching_accepts_crictl_repository_and_tag_columns() { + let listing = "IMAGE TAG IMAGE ID SIZE\ndocker.io/library/grid-operator single-cluster-qualification abc 1MB\n"; + assert!(node_has_image(listing, "grid-operator:single-cluster-qualification")); + assert!(!node_has_image(listing, "grid-operator:other")); + } +}