docs(demos): expand cloud burst resilience example

[nerdalert]

Summary

This expands the Grid cloud-burst demo into a complete regional resilience and multi-cloud overflow example.

The demo now follows inference requests across two consumer gateways, two provider sites, four local llm-d simulators, and an external overflow group containing Azure OpenAI and OpenAI. It demonstrates
healthy regional balancing, progressive backend failure, cross-site fallback, pressure-driven admission, cloud overflow, recovery, shared sliding-window soft quotas, and exact request attribution.

A new recording is included at the top of the README.

User stories

• As an inference-platform operator, I want requests to remain on healthy site-local capacity, fail over to another Grid site when necessary, and use external providers only when the local fleet cannot accept
  new work.

• As a reliability engineer, I want backend failure and queue pressure to produce distinct, explainable admission states.

• As an application owner, I want one shared token allocation to follow my workload across gateways and provider changes.

• As an operator investigating a request, I want to see the consumer gateway, provider boundary, selected inference backend or cloud route, quota decision, and upstream HTTP result.

Demonstrated behavior

Scenario	Expected behavior
Healthy regional traffic	East and west requests balance across their respective local backend pairs.
One backend unavailable	Traffic remains site-local and uses the surviving backend.
One provider site unavailable	Traffic falls back to healthy Grid capacity in the other site.
All local providers unavailable	Traffic uses the Azure OpenAI/OpenAI overflow group.
Local queue pressure	Local admission changes to existing_only; new traffic uses overflow.
Recovery	Restored local providers become eligible after a replacement overlay is accepted.
Shared soft quota	Application/model sliding windows remain continuous across consumers and provider changes.
Exact attribution	Request history distinguishes the provider gateway from the selected backend or external route.

Resilience model

The demo makes the fallback order explicit:

1. Healthy providers in the preferred site.
2. Healthy Grid providers in the other site.
3. Azure OpenAI and OpenAI as the final overflow group.

Backend failure and queue pressure remain separate conditions. Failed endpoints are withdrawn through health reconciliation, while healthy but pressured providers remain available for existing work and reject
only new admission.

Grid computes and publishes these transitions asynchronously. Praxis continues serving from its last accepted routing snapshot and switches only after a newer revision has been validated and loaded. There is
no synchronous Grid or Kubernetes lookup on the request path.

Multi-cloud overflow

Azure OpenAI and OpenAI participate in the same external provider group, with round-robin selection inside that group.

The documentation includes:

• Azure provider resources.
• Dedicated Azure provider gateways.
• Entra client-credential authentication.
• Secret-backed credential injection.
• HTTP authority and TLS SNI configuration.
• Consumer provider-hop mappings.
• A bounded Azure provider validation script.
• Configuration safeguards that preserve the existing local and OpenAI routes.

The recording includes an Azure-attributed HTTP 429. This is identified as an upstream Azure limit, not a Praxis quota denial. A Praxis denial occurs before provider routing and has no provider hop.

Quota behavior

The consumer gateways share application/model sliding-window state through Valkey.

Each demo application has an independent rolling token allocation. Tokens are reserved before provider selection and reconciled against actual response usage afterward. Because the quota key is independent of
the selected provider, regional failover and cloud overflow do not reset or duplicate the application allowance.

The recorded policy uses soft enforcement, so over-allocation remains visible without turning every overage into a pre-provider rejection.

Documentation and resources

The README now includes:

• Detailed architecture and ownership summaries.
• Improved Mermaid topology, request-flow, failover, and recovery diagrams.
• Representative Grid, provider, external-cluster, and token-quota configuration.
• Deployment prerequisites and ordered setup instructions.
• OpenAI and Azure configuration guidance.
• Jaeger and tracing UI configuration.
• Security boundaries and known limitations.
• A complete failure, pressure, recovery, and attribution validation matrix.
• Links to the development branches and related upstream work.

New declarative resources cover:

• Four regional local simulator backends.
• Regional InferenceProvider resources.
• OpenAI and Azure overflow providers.
• Azure provider gateways.
• East and west consumer configurations.
• Optional Jaeger deployment.

Diagnostic and qualification scripts cover Azure validation, queue pressure, backend withdrawal, request stalls, single-provider latency, and the full cloud-burst matrix.

Scope and limitations

This remains an early work-in-progress demonstration assembled from development branches while the underlying contracts are finalized upstream.

It demonstrates group fallback rather than gradual percentage-based bursting or cost-aware placement. Example spend values are illustrative and are not billing records. External providers retain their own
independent quotas, rate limits, and availability.

Reproducing the environment may require manual integration work and familiarity with Kubernetes, Grid, Praxis, external-provider credentials, and the referenced development branches.

Related work

• AI provider-selection foundation: feat(routing): add provider load balancing ai#731
• Grid provider-selection groups: feat(routing): publish provider selection modes grid#65
• AI provider-backend attribution: feat(observability): trace provider backend routes ai#834

[github-actions]

PR too large: 1913 lines added (limit: 750, excludes Cargo files, tests, docs, examples, and benchmarks). Please split into smaller PRs. Add skip/pr-conventions label to override.