diff --git a/.gitignore b/.gitignore index 6da57ec..59ffe15 100644 --- a/.gitignore +++ b/.gitignore @@ -24,3 +24,14 @@ target # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. #.idea/ + +# Serena MCP trial (project config/cache/logs; not yet a team decision) +.serena/ + +# Cursor MCP config (machine-local; mirrors kubernaut-family's .gitignore convention) +.cursor/* +!.cursor/rules/ +!.cursor/skills/ + +# macOS +.DS_Store diff --git a/demos/README.md b/demos/README.md index da1cf92..43b50cc 100644 --- a/demos/README.md +++ b/demos/README.md @@ -6,3 +6,4 @@ Demonstrations of experimental Praxis features live in this directory. - [Praxis Grid - Distributed token rate limiting with Grid routing](grid-distributed-token-rate-limit/README.md) - [Praxis Grid - Intelligent Overflow](grid-cloud-burst/README.md) +- [Per-app token budgets with token_rate_limit](token-rate-limit-per-app-budgets/README.md) diff --git a/demos/token-rate-limit-per-app-budgets/README.md b/demos/token-rate-limit-per-app-budgets/README.md new file mode 100644 index 0000000..27bdef8 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/README.md @@ -0,0 +1,604 @@ +# Per-app token budgets with token_rate_limit, mixed algorithms per rule + +> [!IMPORTANT] +> This is early exploratory work intended to validate the sliding-window +> ledger and bucket-key architecture and inform the design questions +> still open on [ai#658](https://github.com/praxis-proxy/ai/pull/658), +> the canonical Token Rate Limiting proposal. It is not the final +> upstream design — see "Current scope" and "Open design questions" +> below. Configuration and behavior are expected to change, possibly +> substantially, as the proposal is reviewed and implemented upstream. +> +> **The `token_rate_limit` code this demo exercises has not landed in +> `praxis-ai` (or any upstream repo) yet.** It only exists on a personal +> fork branch. "Build and run" below is not optional boilerplate — you +> cannot build this demo's gateway image without pointing at that +> branch, because `main` does not have this filter's sliding-window/ +> token-bucket/Valkey support at all. +> +> **As of this writing, the mixed-algorithm commits described below are +> committed locally on that branch but not yet pushed to the fork.** +> `PRAXIS_AI_SRC` pointed at a local checkout of the branch will pick +> them up; cloning the fork remote fresh (as "Build and run" shows) +> will not, until the push happens. This note should be removed once +> the branch is pushed. + +This demo runs **two independent Praxis AI gateway instances** sharing +one Valkey-backed `token_rate_limit` budget per application, evaluated +against **two rules, each independently choosing its own admission +algorithm** — the current scenario for per-app budgets: an app's +traffic can land on either gateway replica and still draw down the +same budget, and different apps can be on entirely different +algorithms in the same deployment. It validates, end-to-end and across +a real process boundary: + +- **Per-rule algorithm choice**, per + [ai#789](https://github.com/praxis-proxy/ai/issues/789) (originally + filed as `praxis#551`, same issue — see "Open design questions"): + a `gold-tier` rule (`x-tier: gold`) enforces an exact sliding-window + budget; a `silver-tier` rule (`x-tier: silver`) enforces a + continuously-refilling token-bucket budget, in the same filter + instance. See "Current scope" for exactly what's implemented where. +- [ai#129](https://github.com/praxis-proxy/ai/issues/129)'s + `bucket_key_header` proposal: within whichever rule matched, one + independent token budget per unique value of a configured request + header (`x-app-id` here). +- Reservation-based admission, reconciled against actual + provider-reported usage, shared atomically across gateway replicas + via Valkey — for **both** algorithms, not just one. + +## What this demonstrates + +- A platform operator configures an ordered list of `token_rate_limit` + rules, each with its own optional `match` condition and its own + admission algorithm (`sliding_window` or `token_bucket`); the first + rule whose `match` is satisfied applies. Within a matched rule, a + `bucket_key_header` (e.g. `x-app-id`) gives every unique header value + its own independent budget, shared by every gateway instance pointed + at the same Valkey namespace. +- **An app's budget is enforced consistently regardless of which + gateway instance its traffic lands on, for either algorithm.** A + request admitted on gateway A that exhausts app-a's (sliding-window) + or app-b's (token-bucket) budget is visible immediately as an + exhausted budget on gateway B — there is no per-replica budget + multiplication, and this guarantee doesn't depend on which algorithm + the matched rule picked. +- Each app's traffic draws down only its own budget: one app exhausting + its budget and receiving `429`s has zero effect on any other app, + even sharing the same Valkey namespace, even when they're on + different rules/algorithms entirely. +- Requests missing the configured `bucket_key_header` fall back to one + shared budget per rule. +- Praxis reserves an estimated token cost at admission and reconciles + it against the provider's actual reported usage once the response + completes — unused capacity is returned to the budget either way, + but the two algorithms recover it differently: sliding_window returns + it as the trailing window slides past the reservation; token_bucket + returns it immediately on reconciliation and additionally refills + continuously over time. +- Standard `429` responses carry token-denominated + `X-RateLimit-*-Tokens` headers and a `Retry-After` value, computed + correctly for either algorithm. + +## Architecture + +```mermaid +flowchart LR + A1[app-a traffic] -->|x-tier: gold
x-app-id: app-a| GA[Gateway A :8080] + A2[app-a traffic] -->|x-tier: gold
x-app-id: app-a| GB[Gateway B :8081] + C1[app-c traffic] -->|x-tier: gold
x-app-id: app-c| GA + B1[app-b traffic] -->|x-tier: silver
x-app-id: app-b| GA + B1b[app-b traffic] -->|x-tier: silver
x-app-id: app-b| GB + + GA --> TRLA[token_rate_limit] + GB --> TRLB[token_rate_limit] + + TRLA -->|x-tier: gold| RWA[gold-tier rule
sliding_window] + TRLA -->|x-tier: silver| RSA[silver-tier rule
token_bucket] + TRLB -->|x-tier: gold| RWB[gold-tier rule
sliding_window] + TRLB -->|x-tier: silver| RSB[silver-tier rule
token_bucket] + + RWA <--> V[(Shared Valkey
one ledger per rule per app)] + RSA <--> V + RWB <--> V + RSB <--> V + + RWA -->|admitted| BE[Backend] + RSA -->|admitted| BE + RWB -->|admitted| BE + RSB -->|admitted| BE +``` + +Unlike the +[distributed token rate limiting with Grid routing](../grid-distributed-token-rate-limit/README.md) +demo (which layers a shared quota under Grid's multi-cluster provider +routing), this demo is deliberately narrow: two gateway replicas, one +Valkey instance, no Grid, no multi-cluster routing. It isolates the +per-app bucket-key architecture and the sliding-window/Valkey backend +questions from the routing-layer questions the other demo explores. + +## Recorded walkthrough + + +https://github.com/user-attachments/assets/f8ee89ed-ab77-45c4-a31f-bb7c67a78ca0 + +`recording/output/k8s-real-pods-token-rate-limit.mp4` (1920x1080, h264/aac, +~143s) is a narrated recording of the mixed-algorithm scenario below, +driven through `dashboard/` against a real Kubernetes (`kind`) deployment +of the two-gateway + Valkey stack (`k8s/`, `deploy.sh`) -- every pod name, +gateway log line, and per-request HTTP call shown is real, not simulated. +Requests are paced against the narration throughout the clip; each app's +budget renders as a live gauge plus a rolling chart so the `sliding_window` +"flat until the window slides" recovery and the `token_bucket` "continuous +ramp" recovery are visually distinct, not just narrated; and a live +namespace/pod-status strip shows every pod in the deployment, polled from +the real Kubernetes API. See `recording/RECORDING.md` for what it proves +and how it was produced. + +`dashboard/` is a browser-facing convenience used for recording; it wraps +the same HTTP contract as the curl walkthrough below and is not part of +the Praxis AI filter chain. + +## Prerequisites + +- Docker or Podman with Compose (`docker compose` / `podman compose`) +- Git and `curl` + +## Build and run + +Clone this repo and the source branch as siblings, then point +`PRAXIS_AI_SRC` at the source checkout and bring the stack up: + +```bash +git clone https://github.com/praxis-proxy/experimental.git +git clone --branch jordigilh/token-rate-limit-per-app-budgets \ + https://github.com/jordigilh/praxis-ai.git praxis-ai-trl-demo + +cd experimental/demos/token-rate-limit-per-app-budgets +export PRAXIS_AI_SRC=../../../praxis-ai-trl-demo +docker compose up --build -d # or: podman compose up --build -d +``` + +This builds the gateway image from the source branch's own +`Containerfile` (a first build compiles the whole Rust workspace, so +expect it to take several minutes) and starts four containers: + +| Service | Role | +| --- | --- | +| `valkey` | Shared ledger backend for both the sliding-window (`gold-tier`) and token-bucket (`silver-tier`) rules | +| `backend` | Minimal stub upstream, returns an OpenAI-shaped response with a tier-aware `usage.total_tokens` (15 for `x-tier: gold`, 7 for `x-tier: silver`, matching `config.yaml`'s `estimate_tokens` exactly -- see "Validate the request flow" for why that match matters) | +| `gateway-a` | Praxis AI gateway, `127.0.0.1:8080` | +| `gateway-b` | Praxis AI gateway, same image/config/Valkey namespace, `127.0.0.1:8081` | + +Both gateways load the exact same `config.yaml` (two rules, one per +algorithm); the only thing that makes their budgets *shared* rather +than *independent* is pointing both at the same Valkey `namespace`. + +## Run on Kubernetes (kind) + +The recorded walkthrough above ran this same stack on a real `kind` +cluster instead of `docker compose` -- every pod name, gateway log +line, and per-request HTTP call in the video is real. `k8s/deploy.sh` +reproduces that deployment. You'll additionally need `kind` and +`kubectl` installed. + +```bash +# 1. Build the gateway image from the source branch checked out above. +cd ../../../praxis-ai-trl-demo +podman build -t docker.io/library/praxis-ai-trl-demo:local -f Containerfile . + +# 2. Create a kind cluster and load the image into it -- k8s/03-gateways.yaml +# sets imagePullPolicy: Never, so the image must already be on the node +# rather than pulled from a registry. +KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --name trl-demo +podman save docker.io/library/praxis-ai-trl-demo:local -o /tmp/praxis-ai-trl-demo.tar +KIND_EXPERIMENTAL_PROVIDER=podman kind load image-archive \ + /tmp/praxis-ai-trl-demo.tar --name trl-demo + +# 3. Deploy the namespace, Valkey, backend, both gateways, three apps, and +# the dashboard, and wait for every pod to be ready. +cd ../experimental/demos/token-rate-limit-per-app-budgets +./k8s/deploy.sh + +# 4. Port-forward the same ports "Validate the request flow" and the +# dashboard use below, so those curl commands work unmodified. +kubectl -n trl-demo port-forward svc/gateway-a 8080:8080 & +kubectl -n trl-demo port-forward svc/gateway-b 8081:8080 & +kubectl -n trl-demo port-forward svc/dashboard 3000:3000 & +``` + +Docker users can drop `KIND_EXPERIMENTAL_PROVIDER=podman` and replace the +`podman save` + `kind load image-archive` steps with `kind load +docker-image docker.io/library/praxis-ai-trl-demo:local --name trl-demo` +(loads directly from Docker's image store, no tar file needed). Tear down +with `KIND_EXPERIMENTAL_PROVIDER=podman kind delete cluster --name +trl-demo` when done. + +## Validate the request flow + +`gold-tier` (`sliding_window`) uses `capacity: 40` tokens, +`estimate_tokens: 15` per request — deliberately *lower* than capacity, so +a single request never exhausts the budget outright: two requests admit +(15 + 15 = 30 reserved), a third is denied (needs 15, only 10 remain). +`silver-tier` (`token_bucket`) uses `capacity: 10`, `estimate_tokens: 7` — +one request admits (7 reserved, 3 remaining), a second immediately after +is denied (needs 7, only 3 remain), but waiting just ~2 more seconds +refills 4 tokens (`refill_rate: 2`/sec) — 3 + 4 = 7, exactly enough for a +retry to succeed. Both tiers' `estimate_tokens` are set to **exactly** +match what the stub backend reports as `usage.total_tokens` for that tier +(read from the `x-tier` header — see `docker-compose.yml`'s `backend` +service) and *must* stay matched: reconciliation debits `actual - estimate` +as an *extra* charge on top of the estimate already reserved whenever +actual usage exceeds the estimate — symmetrically for both algorithms, per +the source branch's `Ledger::reconcile`/`TokenBucketLedger::reconcile` and +their unit tests — so a lower estimate than the real backend usage would +silently over-drain either budget past this walkthrough's math. Keeping +`estimate_tokens` matched to actual usage for both tiers keeps the numbers +below exact and avoids depending on reconciliation's refund/overage timing +at all. Both `gold-tier`'s +`window: 10s` and `silver-tier`'s `refill_rate: 2` tokens/sec are +similarly shortened from realistic production values purely so recovery +is watchable in seconds instead of the hour+ a production deployment +would take. Every request must carry `x-tier` (selects the rule/algorithm) +alongside `x-app-id` (selects the per-app budget within that rule) — a +request missing `x-tier` matches no rule and isn't rate-limited by this +filter instance at all (see `config.yaml`). + +Send app-a's (`x-tier: gold`, sliding_window) first two requests as a +rapid burst — one to gateway A, one to gateway B, the *other* process — +then a third, back on gateway A: + +```bash +echo "== app-a (gold/sliding_window) on gateway A (expect 200 -- 15/40 reserved, 25 remaining) ==" +curl -si http://127.0.0.1:8080/v1/chat/completions \ + -H "x-tier: gold" -H "x-app-id: app-a" -H 'Content-Type: application/json' \ + -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}' \ + | grep -Ei '^(HTTP|x-ratelimit|retry-after)' + +echo "== app-a on gateway B, right after (expect 200 -- shared ledger keeps accumulating: 30/40 reserved, 10 left) ==" +curl -si http://127.0.0.1:8081/v1/chat/completions \ + -H "x-tier: gold" -H "x-app-id: app-a" -H 'Content-Type: application/json' \ + -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}' \ + | grep -Ei '^(HTTP|x-ratelimit|retry-after)' + +echo "== app-a's third request, back on gateway A (expect 429 -- needs 15, only 10 remain, on either gateway) ==" +curl -si http://127.0.0.1:8080/v1/chat/completions \ + -H "x-tier: gold" -H "x-app-id: app-a" -H 'Content-Type: application/json' \ + -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}' \ + | grep -Ei '^(HTTP|x-ratelimit|retry-after)' + +echo "== app-c (gold/sliding_window) on gateway A (expect 200 -- unaffected by app-a) ==" +curl -si http://127.0.0.1:8080/v1/chat/completions \ + -H "x-tier: gold" -H "x-app-id: app-c" -H 'Content-Type: application/json' \ + -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}' \ + | grep -Ei '^(HTTP|x-ratelimit|retry-after)' + +echo "== app-b (silver/token_bucket) on gateway A (expect 200 -- 7/10 reserved, 3 remaining) ==" +curl -si http://127.0.0.1:8080/v1/chat/completions \ + -H "x-tier: silver" -H "x-app-id: app-b" -H 'Content-Type: application/json' \ + -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}' \ + | grep -Ei '^(HTTP|x-ratelimit|retry-after)' + +echo "== app-b on gateway B, right after (expect 429 -- needs 7, 3 remain -- Valkey enforces token_bucket too) ==" +curl -si http://127.0.0.1:8081/v1/chat/completions \ + -H "x-tier: silver" -H "x-app-id: app-b" -H 'Content-Type: application/json' \ + -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}' \ + | grep -Ei '^(HTTP|x-ratelimit|retry-after)' +``` + +Expect (verified against a live run of this exact scenario, driven +directly against the built gateway binaries against a real Valkey +instance while authoring this demo): + +- app-a's request on gateway A: `200`. +- app-a's request on gateway B, immediately after: `200` — the *other* + gateway process sees and adds to the very same accumulated reservation, + because both consult the same Valkey ledger rather than keeping + independent in-process state. +- app-a's third request, back on gateway A: `429` with + `X-RateLimit-Remaining-Tokens: 0`, `Retry-After: 10` — gold-tier is now + exhausted (30 reserved + 15 needed > 40 capacity), consistently, no + matter which gateway is asked. +- app-c's request on gateway A: `200` — app-c's budget is untouched by + app-a's exhaustion, even on the same rule/algorithm and Valkey + namespace. +- app-b's request on gateway A: `200` — a completely different rule + (`silver-tier`, `token_bucket`), matched purely on `x-tier`, with its + own budget. +- app-b's request on gateway B, immediately after: `429` with + `X-RateLimit-Remaining-Tokens: 0`, `Retry-After: 2` — a single burst + already leaves too little (3 remaining) for a second 7-token call, + proving the shared-Valkey, cross-instance guarantee holds for + `token_bucket` too, not just `sliding_window`. + +Wait for both algorithms to recover, then retry both app-a on gateway A +and app-b on gateway B — no restart, no manual reset, nothing but time +passing: + +```bash +sleep 11 +echo "== app-a on gateway A again (expect 200 -- sliding window recovered) ==" +curl -si http://127.0.0.1:8080/v1/chat/completions \ + -H "x-tier: gold" -H "x-app-id: app-a" -H 'Content-Type: application/json' \ + -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}' \ + | grep -Ei '^(HTTP|x-ratelimit|retry-after)' + +echo "== app-b on gateway B again (expect 200 -- token bucket refilled) ==" +curl -si http://127.0.0.1:8081/v1/chat/completions \ + -H "x-tier: silver" -H "x-app-id: app-b" -H 'Content-Type: application/json' \ + -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}' \ + | grep -Ei '^(HTTP|x-ratelimit|retry-after)' +``` + +Expect `200` on both. What "recovered" means differs sharply by algorithm: + +- **app-a (sliding_window)**: the trailing window has moved forward far + enough that app-a's earlier reservations are no longer counted against + its budget. There is no fixed reset boundary (like a calendar month + rolling over); the budget recovers continuously and independently as + its own past usage ages out — but it needs the *full* `window: 10s` to + do so. +- **app-b (token_bucket)**: the bucket has been continuously refilling at + `refill_rate: 2` tokens/sec since app-b's last request (reconciliation + itself credits back ~0 here, by design — see "Validate the request + flow" above, since actual usage always equals the estimate). It becomes + admissible again in just **2 seconds** from the denied attempt (3 + remaining + 2s × 2 tokens/sec = 7, exactly the estimate needed) — a + fraction of gold-tier's full 10s window, since refill is continuous + rather than gated on a fixed boundary. The `sleep 11` above is sized for + gold-tier's slower path, not because silver-tier needs anywhere near + that long. + +This was verified against a live run of the built gateway binaries +(`praxis-ai` from the source branch) directly against a real Valkey +instance while authoring this demo -- two gateway processes on +different ports, both pointed at the same Valkey, driven with the +`curl` commands above, not just checked against source. It was **not** +separately re-verified through the `docker compose` stack itself in +this environment (a local container-runtime issue blocked bringing the +compose stack up while authoring this update); the compose stack uses +the identical `config.yaml` and gateway image, so the same behavior is +expected, but that specific path is unconfirmed as of this writing. + +## Current scope + +This needs to be read alongside +[ai#658](https://github.com/praxis-proxy/ai/pull/658)'s own review +thread, not as a substitute for it: + +- **Per-rule algorithm choice is now implemented on the source + branch**: `rules:` is an ordered list, each with its own optional + `match`, its own algorithm (`sliding_window` or `token_bucket`), and + its own budget -- exactly the capability the "Open design questions" + section below used to flag as confirmed-but-not-built. **This is + still only on the personal fork branch this demo builds from, not + merged into `praxis-ai` upstream.** +- **sliding_window**: an exact trailing-window admission ledger, + matching ai#658's current design doc ("windows are sliding: a + `window: 1h` budget tracks usage in the most recent 60 minutes from + the current instant"). [ai#789](https://github.com/praxis-proxy/ai/issues/789) + ("Sliding window rate limiting" — note: this issue was originally + filed as `praxis#551` and later transferred to the `ai` repo as + `ai#789`; both numbers refer to the *same* single issue, not two + separate tracks) is still open; this filter carries its own + sliding-window ledger rather than depending on it, so it doesn't + block this MVP. +- **token_bucket**: continuous refill up to `capacity` at `refill_rate` + tokens/sec, reusing Praxis's own lock-free + `traffic_management::token_bucket` refill formula, extended with the + reserve/reconcile split this filter needs. This demo's *first* + version implemented token bucket unconditionally (before + sliding_window existed in the source branch); see "Alternative + implementations considered" below for that history. Per-rule choice + means both now coexist rather than one superseding the other. +- **The per-key bucket architecture** (resolve a key from a + configurable header, look it up in a keyed map, fall back to a + shared budget when the header's absent, evict idle keys) is meant to + be reusable regardless of which rate-limiting algorithm ai#658 + ultimately specifies. +- **State is pluggable**: in-process (default, one gateway, no shared + state) or Valkey (`backend: {kind: valkey}`, this demo). Both share + the same reservation/reconciliation semantics; only where the ledger + lives differs. +- Composite/multi-dimension keys, per-model keys, CEL-expression keys, + configurable token estimation, and token-type-aware accounting are + all out of scope here — see the module doc on the source branch for + the full list. + +## Alternative implementations considered + +- **Envoy-style external rate-limit gRPC service.** The canonical + Envoy pattern runs quota state behind a separate gRPC sidecar/service + that every proxy instance calls out to. Rejected here in favor of an + in-filter backend trait (`reserve`/`reconcile` calls made directly + from the filter, Valkey accessed via a plain client rather than a + bespoke RPC surface): it avoids standing up and operating an + additional service just for this MVP, at the cost of coupling the + quota logic to Praxis AI's own filter lifecycle rather than making it + reusable outside Praxis. Worth revisiting if quota enforcement needs + to be shared with non-Praxis callers. +- **Token bucket only, no algorithm choice (this demo's first + version).** The very first version of this demo implemented a + token-bucket algorithm (`rate`/`burst` refill) unconditionally, not + sliding-window, because the sliding-window ledger didn't exist yet in + the source branch and ai#789 (filed as `praxis#551`) was — and still + is — open/unresolved. That version was superseded once the source + branch grew its own exact sliding-window ledger (adapted from + [nerdalert's spike branch](https://github.com/nerdalert/ai/tree/poc/distributed-token-rate-limit-demo)), + closing the gap with ai#658's design doc without waiting on ai#789 -- + and *that* version, in turn, is what this demo now extends with + per-rule algorithm choice rather than picking one algorithm to keep + and one to drop. +- **[Distributed token rate limiting with Grid routing](../grid-distributed-token-rate-limit/README.md) + demo's approach.** That demo validates the same Valkey-backed + sliding-window ledger concept, but keyed by authenticated + principal+model and layered under Grid's multi-cluster provider + routing on a full Kind/Helm/Grid stack. This demo deliberately keeps + the same core idea (shared Valkey ledger, reservation/reconciliation) + but strips out Grid, multi-cluster routing, and authentication + entirely, keyed by a plain request header instead, so the per-app + bucket-key question can be evaluated in isolation on a two-container + `docker compose` stack instead of a multi-cluster deployment. +- **Local-only in-memory state for the "shared across replicas" + scenario.** Rejected as insufficient for what this demo specifically + needs to show: in-process state is already covered by this filter's + default (no `backend:` block) and is fine for a single gateway + instance, but says nothing about the multi-replica case, which is + the actual point of this demo. Valkey is the minimum needed to prove + budgets survive a real process boundary. + +## Open design questions + +These are unresolved as of this writing. Expect this demo's behavior, +config shape, or scope to change once they're settled — treat it as a +snapshot of one point in an ongoing design discussion, not a preview of +the final feature. + +- **Per-rule algorithm choice is implemented here, but only on a + personal fork branch, and upstream hasn't weighed in on the shape.** + Neither the epic ([ai#121](https://github.com/praxis-proxy/ai/issues/121)) + nor the proposal ([ai#658](https://github.com/praxis-proxy/ai/pull/658)) + originally said whether `token_rate_limit` should support only sliding + window, or let an operator pick an algorithm per rule; the "Sliding + window rate limiting" issue itself + ([ai#789](https://github.com/praxis-proxy/ai/issues/789), also + reachable via its original number `praxis#551` — same issue, not a + separate one) states per-rule algorithm choice as an explicit goal + ("independent of token bucket, operators choose per rule"), and this + demo/source branch now implement exactly that (`rules:` with a + `match` + `algorithm` per rule -- see "Current scope"). What's still + open: whether `rules:`/`match: {headers: ...}` is the config shape + upstream actually wants (vs. e.g. a different matcher syntax, CEL + expressions per praxis#189/#232, or a different name), and whether a + third algorithm (fixed window) should exist alongside these two. + Don't read this demo's specific config shape as upstream-approved; + it's one concrete proposal for maintainer review, not a decision. +- **Window duration and refill rate are config knobs, not yet + customer-tunable requirements anywhere in the proposal.** This demo + uses `window: 10s` and `refill_rate: 2` purely to make each + algorithm's recovery visible in a short recording; nothing in ai#658 + pins either value, and calendar-aligned windows (e.g. reset at UTC + midnight rather than "most recent N seconds") are a distinct semantic + that neither algorithm here provides and hasn't been requested yet. +- ~~The two algorithms reconcile differently~~ **Correction**: an earlier + version of this section claimed `token_bucket` and `sliding_window` + reconcile estimate/actual gaps differently, framed as an open upstream + question. That claim was wrong — re-reading the source branch's + `Ledger::reconcile` and `TokenBucketLedger::reconcile` (and their unit + tests, e.g. `reconcile_releases_unused_tokens_on_overestimate`) shows both + apply the identical estimate-vs-actual delta; `sliding_window` does + retroactively shrink what's counted against its window on an + overestimate, it just does so by recording the *settled* usage at the + actual token count rather than by adjusting a live balance the way a + bucket does. There is no known open design question here upstream; see + `recording/RECORDING.md`'s "Correction" section for the full + explanation. +- **The Valkey backend is a spike, not yet aligned with + [grid#83](https://github.com/praxis-proxy/grid/issues/83)**, the + authoritative spec for Valkey-backed distributed quota state (published + after this demo's backend was first written). Concretely, against + grid#83's requirements: + - *Met:* atomic reserve/reconcile via Lua (`EVAL`), idempotent + reconciliation (a reservation can only be settled once), reservation + TTL + cleanup for abandoned requests, and **fail-closed on backend + error** — a Valkey timeout or error returns `503`, not silent + admission (see `on_request` in the source branch's `mod.rs`). + - *Not met, by deliberate demo simplification:* this compose stack's + Valkey runs with **no authentication** and **`--save ""` + (persistence disabled)** — see `docker-compose.yml`. grid#83 requires + explicit auth, private network access, and documented durable + storage/backup behavior; this demo proves none of that, only the + reservation/reconciliation logic on top of an ephemeral, unauthenticated + instance. + - *Not exercised by this demo's walkthrough or test suite:* grid#83's + validation checklist also asks for proof that usage survives a + consumer restart, that concurrent reservations can't oversubscribe + capacity under load, and that a Valkey outage-then-recovery cycle + fails closed and then resumes without resetting existing usage. None + of those are demonstrated here — only the steady-state admit/deny/ + recover-on-window-slide path is. + - *Config gap:* the backend/Lua layer already supports multiple atomic + budgets per key (`Vec`), but `token_rate_limit`'s own config + schema only exposes a single `window`/`capacity` pair per rule today — + multi-window enforcement exists underneath but isn't wired up to + configuration yet. + +## References + +This demo's config surface and admission logic are traceable to specific +upstream discussions and, where nothing upstream covers it yet, to +established prior art outside Praxis. Neither list below is a claim of +correctness or of upstream endorsement — see "Current scope" and "Open +design questions" above for what's actually settled vs. still open. + +### Praxis issues this demo derives from + +- [ai#121: Epic — Token Rate + Limiting](https://github.com/praxis-proxy/ai/issues/121) — the + umbrella epic this whole filter falls under. +- [ai#658: Canonical Token Rate Limiting + proposal](https://github.com/praxis-proxy/ai/pull/658) — this demo's + baseline design doc; see "Current scope" for exactly which parts of it + are implemented here vs. still under review. +- [ai#789: Sliding window rate + limiting](https://github.com/praxis-proxy/ai/issues/789) (originally + filed as + [praxis#551](https://github.com/praxis-proxy/praxis/issues/551), same + issue) — the per-rule-algorithm-choice requirement this demo/source + branch implement; still open upstream. +- [ai#129: Per-header rate limit bucket + keys](https://github.com/praxis-proxy/ai/issues/129) — the + `bucket_key_header` proposal this demo's per-app (`x-app-id`) + budgeting is a concrete implementation of. +- [grid#83: Support Valkey-backed distributed quota + state](https://github.com/praxis-proxy/grid/issues/83) — the + authoritative spec this demo's Valkey backend is a deliberately + simplified spike against; see "Open design questions" for the specific + gaps (auth, persistence, load/failover testing). + +### External prior art (for logic not yet specified in Praxis) + +Where a design choice isn't covered by any of the issues above, it +follows established external precedent instead of being invented for +this demo: + +- [Token bucket (Wikipedia)](https://en.wikipedia.org/wiki/Token_bucket) — + the classical algorithm `silver-tier`'s `token_bucket` admission + follows: continuous refill up to a fixed capacity, drained per + admitted request, independent of any window boundary. +- [Cloudflare — "How we built rate limiting capable of scaling to + millions of + domains"](https://blog.cloudflare.com/counting-things-a-lot-of-different-things/) + — the canonical reference for sliding-window-style rate limiting at + scale. Note `gold-tier`'s `sliding_window` here is an **exact** + trailing-window ledger (sums real reservations still inside the + window, per ai#658's design doc), not Cloudflare's O(1)-memory + *approximate* two-counter estimate described in that post — related + algorithm family, different technique, cited here for the general + sliding-window concept rather than as a claim this filter + reimplements Cloudflare's specific approximation. +- [envoyproxy/ratelimit](https://github.com/envoyproxy/ratelimit) — the + canonical external-gRPC-rate-limit-service pattern, considered and + rejected for this filter in favor of an in-filter backend trait; see + "Alternative implementations considered". +- [Redis — rate limiter + patterns](https://redis.io/docs/latest/develop/use-cases/rate-limiter/) — + the atomic reserve/consume-via-Lua-`EVAL` pattern this filter's + Valkey backend follows to keep the reserve-then-decide step + race-free across concurrent requests from both gateway replicas. +- [nerdalert's `distributed-token-rate-limit-demo` spike + branch](https://github.com/nerdalert/ai/tree/poc/distributed-token-rate-limit-demo) + — the source this demo's exact sliding-window ledger was originally + adapted from, before per-rule algorithm choice was added on top; see + "Alternative implementations considered" for that history. + +### Related demo and source + +- [Source + branch](https://github.com/jordigilh/praxis-ai/tree/jordigilh/token-rate-limit-per-app-budgets) + — the personal fork branch this demo's gateway image builds from (see + the warning banner at the top of this README). +- [Distributed token rate limiting with Grid + routing](../grid-distributed-token-rate-limit/README.md) — a + complementary demo exploring distributed counters, authentication, + and multi-gateway quota sharing under Grid routing. diff --git a/demos/token-rate-limit-per-app-budgets/config.yaml b/demos/token-rate-limit-per-app-budgets/config.yaml new file mode 100644 index 0000000..9f9f0a0 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/config.yaml @@ -0,0 +1,92 @@ +# Demo fixture for the token-rate-limit-per-app-budgets demo (see the +# sibling README.md). Both gateway instances in docker-compose.yml load +# this same config, pointed at the same Valkey backend: that's what +# makes their per-app-per-tier budgets shared rather than independent. +# Deliberately small window/capacity/refill values so the curl +# walkthrough exhausts and recovers a budget within seconds. +# +# Source: examples/configs/token-rate-limit-mixed-algorithms.yaml on +# https://github.com/jordigilh/praxis-ai/tree/jordigilh/token-rate-limit-per-app-budgets +# +# Two rules, two algorithms, matched by an `x-tier` header -- the +# per-rule algorithm choice from ai#789/praxis#551 (see README.md's +# "Current scope"). `x-app-id` still keys an independent budget per app +# *within* whichever rule matched, per ai#129. +# +# `window: 10s` / `refill_rate: 2` are also deliberately short (a real +# deployment would use much larger values): long enough to observe each +# algorithm's own recovery mechanics, short enough to watch happen +# within a single curl walkthrough or recording. +# +# estimate_tokens is deliberately LOWER than each rule's capacity for both +# tiers (15/40 gold, 7/10 silver) so a single request never exhausts a +# budget outright -- multiple real, independently admitted requests are +# visible before the eventual denial, instead of a one-shot cliff-edge. +# Both values MUST exactly match what the backend's `usage.total_tokens` +# reports for that tier (see k8s/02-backend.yaml / docker-compose.yml's +# `backend` service, which reads the `x-tier` header for this reason) -- +# NOT a smaller "optimistic" estimate. This isn't cosmetic: reconciliation +# debits `actual - estimate` as an *extra* charge beyond the estimate +# already reserved whenever actual > estimate, symmetrically for both +# algorithms (verified against the source branch's `Ledger::reconcile` / +# `TokenBucketLedger::reconcile` and their unit tests -- see +# recording/RECORDING.md's "Correction" section, which supersedes an +# earlier, incorrect claim that the two algorithms reconciled +# differently), so an estimate lower than the real backend usage would +# silently over-drain either budget past what the naive "estimate only" +# arithmetic below assumes. Keeping estimate and actual matched for both +# tiers avoids depending on that reconciliation math at all. + +listeners: + - name: default + address: "0.0.0.0:8080" + filter_chains: + - main + +filter_chains: + - name: main + filters: + - filter: router + routes: + - path_prefix: "/" + cluster: backend + + - filter: token_rate_limit + rules: + - name: gold-tier + match: + headers: + x-tier: gold + algorithm: sliding_window + window: 10s # sliding window duration, per app + capacity: 40 # each app's budget within the window, in tokens + estimate_tokens: 15 # cost reserved per request -- 2 requests admit (30/40), a 3rd is denied + bucket_key_header: x-app-id # one independent budget per app, within this rule + backend: + kind: valkey # shared across every gateway instance/replica + url: "${TOKEN_RATE_LIMIT_VALKEY_URL}" + namespace: praxis-demo:token-rate-limit-per-app + - name: silver-tier + match: + headers: + x-tier: silver + algorithm: token_bucket + capacity: 10 # each app's bucket ceiling, in tokens + refill_rate: 2 # tokens refilled per second, up to `capacity` + estimate_tokens: 7 # cost reserved per request -- 1 request admits (3 left), a back-to-back 2nd is denied; a 2s wait refills 4 (3+4=7) so a retry then admits + bucket_key_header: x-app-id # one independent budget per app, within this rule + backend: + kind: valkey # shared across every gateway instance/replica + url: "${TOKEN_RATE_LIMIT_VALKEY_URL}" + namespace: praxis-demo:token-rate-limit-per-app + + - filter: token_count + provider: openai # openai | anthropic | google | bedrock | azure + + - filter: access_log + + - filter: load_balancer + clusters: + - name: backend + endpoints: + - "backend:3000" diff --git a/demos/token-rate-limit-per-app-budgets/dashboard/index.html b/demos/token-rate-limit-per-app-budgets/dashboard/index.html new file mode 100644 index 0000000..a7f955b --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/dashboard/index.html @@ -0,0 +1,635 @@ + + + + +Per-app token budgets — live demo + + + +

Per-app token budgets — mixed algorithms, shared across gateway replicas

+

Two token_rate_limit rules matched by x-tier, each keyed by x-app-id: gold-tier (sliding_window) for app-a/app-c, silver-tier (token_bucket) for app-b — both backed by the same shared Valkey namespace.

+ +
+ Namespace: trl-demo + Live pods: resolving via Kubernetes API… +
+ +
+
+
+
Gold tier · sliding_windowcapacity 40, window 10s — usage counted against the window doesn't free up until the window itself slides forward
+
+
+
+
Silver tier · token_bucketcapacity 10, refill 2/s — drains on use, then refills continuously at a fixed rate, independent of any window
+
+
+ +
+
+
app-a · gold-tier (sliding_window)
+
pod: resolving…
+
idle
+
no requests yet
+
remaining40 / 40
+
+ +
+
+
app-b · silver-tier (token_bucket)
+
pod: resolving…
+
idle
+
no requests yet
+
remaining10 / 10
+
+ +
+
+
app-c · gold-tier (sliding_window)
+
pod: resolving…
+
idle
+
no requests yet
+
remaining40 / 40
+
+ +
+
+ +
+ +
+
+ +
+
+

Live request telemetry — real fields from each app pod's actual HTTP call, not simulated

+
+
+
+

Live gateway pod logs — real stdout, streamed via the Kubernetes API (kubectl logs -f equivalent)

+
+
+
Gateway A resolving…
+
+
+
+
Gateway B resolving…
+
+
+
+
+
+

What this proves

+
Per-rule algorithm choice: gold-tier (sliding_window) vs silver-tier (token_bucket), matched by x-tier
+
Cross-instance shared budget: exhausted on Gateway A means denied on Gateway B too — for both algorithms
+
Per-app isolation: app-c's budget is untouched by app-a's exhaustion, same rule, same namespace
+
Reservation-based admission: cost reserved atomically before the response is known
+
Automatic window recovery: app-a's sliding window ages its reservation out on its own
+
Automatic bucket recovery: app-b's token bucket refills continuously, on its own terms
+
+
+ + + + diff --git a/demos/token-rate-limit-per-app-budgets/dashboard/nginx.conf b/demos/token-rate-limit-per-app-budgets/dashboard/nginx.conf new file mode 100644 index 0000000..8474ee8 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/dashboard/nginx.conf @@ -0,0 +1,44 @@ +# Same-origin dashboard + reverse proxy for the token-rate-limit-per-app-budgets +# demo. Proxies to the app-a/app-b/app-c pods (K8s), which each make their own +# real, server-side call to a gateway -- the dashboard never talks to a +# gateway directly, so "which app sent this" is a real separate pod, not a +# header value picked in the browser. Proxying under one origin also avoids +# CORS preflight for the dashboard's fetch() calls. +# +# `proxy_buffering off` + a long `proxy_read_timeout` are required for +# /apps/*/gw-logs -- a long-lived Server-Sent-Events stream of each gateway +# pod's real stdout -- or nginx would buffer/chunk it into large delayed +# blocks (or time it out) instead of forwarding lines as they arrive. +server { + listen 3000; + server_name _; + + location / { + root /usr/share/nginx/html; + index index.html; + } + + location /apps/app-a/ { + proxy_pass http://app-a:4000/; + proxy_set_header Host $host; + proxy_http_version 1.1; + proxy_buffering off; + proxy_read_timeout 3600s; + } + + location /apps/app-b/ { + proxy_pass http://app-b:4000/; + proxy_set_header Host $host; + proxy_http_version 1.1; + proxy_buffering off; + proxy_read_timeout 3600s; + } + + location /apps/app-c/ { + proxy_pass http://app-c:4000/; + proxy_set_header Host $host; + proxy_http_version 1.1; + proxy_buffering off; + proxy_read_timeout 3600s; + } +} diff --git a/demos/token-rate-limit-per-app-budgets/docker-compose.yml b/demos/token-rate-limit-per-app-budgets/docker-compose.yml new file mode 100644 index 0000000..93ab30e --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/docker-compose.yml @@ -0,0 +1,127 @@ +# Two independent Praxis AI gateway instances, sharing one Valkey-backed +# sliding-window ledger for the token_rate_limit filter -- the "final +# scenario" for per-app budgets (ai#129 keying + the ai#658 sliding-window +# design), demonstrating that an app's budget is enforced consistently no +# matter which gateway replica its traffic happens to land on. +# +# `PRAXIS_AI_SRC` must point at a checkout of the source branch (see +# README.md's "Build and run" section); it's only needed to build the +# gateway image, not at runtime. + +services: + valkey: + image: docker.io/valkey/valkey:8-alpine + command: ["valkey-server", "--save", ""] + healthcheck: + test: ["CMD", "valkey-cli", "ping"] + interval: 2s + timeout: 2s + retries: 15 + networks: [demo] + + # Minimal OpenAI-shaped backend so admitted requests get a real 200 + # with token usage data, exercising reservation reconciliation as well + # as admission/denial. `usage.total_tokens` is tier-aware (read from the + # `x-tier` header) and must exactly equal config.yaml's `estimate_tokens` + # for that tier (15 gold, 7 silver) -- see config.yaml's comment for why + # a mismatch silently breaks the reconciliation math. + backend: + image: docker.io/library/python:3.13-alpine + working_dir: /srv + command: + - python3 + - -c + - | + import json + from http.server import BaseHTTPRequestHandler, HTTPServer + + TOTAL_TOKENS_BY_TIER = {"gold": 15, "silver": 7} + + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + tier = self.headers.get("x-tier", "") + total_tokens = TOTAL_TOKENS_BY_TIER.get(tier, 10) + prompt_tokens = total_tokens // 2 + body = json.dumps({ + "id": "demo-completion", + "object": "chat.completion", + "choices": [ + {"index": 0, "message": {"role": "assistant", "content": "hi"}, "finish_reason": "stop"} + ], + "usage": { + "prompt_tokens": prompt_tokens, + "completion_tokens": total_tokens - prompt_tokens, + "total_tokens": total_tokens, + }, + }).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *_args): + pass + + HTTPServer(("0.0.0.0", 3000), Handler).serve_forever() + networks: [demo] + + gateway-a: + build: + context: ${PRAXIS_AI_SRC:-../../../praxis-ai-trl-demo} + dockerfile: Containerfile + image: praxis-ai-trl-demo:local + environment: + TOKEN_RATE_LIMIT_VALKEY_URL: redis://valkey:6379 + volumes: + - ./config.yaml:/etc/praxis/config.yaml:ro + command: ["-c", "/etc/praxis/config.yaml"] + ports: + - "8080:8080" + depends_on: + valkey: + condition: service_healthy + backend: + condition: service_started + networks: [demo] + + # Same image, same config, same Valkey backend as gateway-a -- an + # independent replica, not a hot spare. Its only relationship to + # gateway-a is sharing the built image tag (see README.md). + gateway-b: + image: praxis-ai-trl-demo:local + environment: + TOKEN_RATE_LIMIT_VALKEY_URL: redis://valkey:6379 + volumes: + - ./config.yaml:/etc/praxis/config.yaml:ro + command: ["-c", "/etc/praxis/config.yaml"] + ports: + - "8081:8080" + depends_on: + valkey: + condition: service_healthy + backend: + condition: service_started + gateway-a: + condition: service_started + networks: [demo] + + # Same-origin browser dashboard + reverse proxy (see dashboard/nginx.conf). + # Recording-only convenience: exercises the exact same HTTP contract as the + # curl walkthrough above, it does not participate in the filter chain. + dashboard: + image: docker.io/library/nginx:alpine + volumes: + - ./dashboard/nginx.conf:/etc/nginx/conf.d/default.conf:ro + - ./dashboard/index.html:/usr/share/nginx/html/index.html:ro + ports: + - "3000:3000" + depends_on: + gateway-a: + condition: service_started + gateway-b: + condition: service_started + networks: [demo] + +networks: + demo: {} diff --git a/demos/token-rate-limit-per-app-budgets/k8s/00-namespace-rbac.yaml b/demos/token-rate-limit-per-app-budgets/k8s/00-namespace-rbac.yaml new file mode 100644 index 0000000..46c8d00 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/k8s/00-namespace-rbac.yaml @@ -0,0 +1,43 @@ +# Namespace + minimal RBAC so each app-* pod can (a) look up which real pod is +# currently serving "gateway-a"/"gateway-b" via the Kubernetes API, instead of +# a name invented client-side, and (b) stream that pod's real stdout via the +# pods/log subresource (the same data `kubectl logs -f` reads), for the +# dashboard's live gateway-log panel. Scoped to get/list Pods and get +# Pods/log in this namespace only -- no write access, no other namespaces. +apiVersion: v1 +kind: Namespace +metadata: + name: trl-demo +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: trl-demo-app + namespace: trl-demo +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: trl-demo-pod-reader + namespace: trl-demo +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list"] + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: trl-demo-app-pod-reader + namespace: trl-demo +subjects: + - kind: ServiceAccount + name: trl-demo-app + namespace: trl-demo +roleRef: + kind: Role + name: trl-demo-pod-reader + apiGroup: rbac.authorization.k8s.io diff --git a/demos/token-rate-limit-per-app-budgets/k8s/01-valkey.yaml b/demos/token-rate-limit-per-app-budgets/k8s/01-valkey.yaml new file mode 100644 index 0000000..b42d37d --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/k8s/01-valkey.yaml @@ -0,0 +1,35 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: valkey + namespace: trl-demo + labels: { app: valkey } +spec: + replicas: 1 + selector: + matchLabels: { app: valkey } + template: + metadata: + labels: { app: valkey } + spec: + containers: + - name: valkey + image: docker.io/valkey/valkey:8-alpine + command: ["valkey-server", "--save", ""] + ports: + - containerPort: 6379 + readinessProbe: + exec: { command: ["valkey-cli", "ping"] } + periodSeconds: 2 + failureThreshold: 15 +--- +apiVersion: v1 +kind: Service +metadata: + name: valkey + namespace: trl-demo +spec: + selector: { app: valkey } + ports: + - port: 6379 + targetPort: 6379 diff --git a/demos/token-rate-limit-per-app-budgets/k8s/02-backend.yaml b/demos/token-rate-limit-per-app-budgets/k8s/02-backend.yaml new file mode 100644 index 0000000..e380d45 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/k8s/02-backend.yaml @@ -0,0 +1,99 @@ +# Minimal OpenAI-shaped backend so admitted requests get a real 200 with +# token usage data, exercising reservation reconciliation as well as +# admission/denial. `usage.total_tokens` is deliberately tier-aware (read +# from the `x-tier` header, which the router forwards through unmodified) +# rather than a single fixed constant -- it must exactly equal whatever +# ../config.yaml sets as that tier's `estimate_tokens` (15 gold, 7 silver), +# or reconciliation's actual-vs-estimate delta silently over/under-drains +# the budget relative to the demo's intended arithmetic (see +# ../config.yaml's comment and recording/RECORDING.md's "Correction" +# section). Everything else (id, created, and a small realistic processing +# delay) is randomized per request so the wire response is never +# byte-identical twice. +apiVersion: v1 +kind: ConfigMap +metadata: + name: backend-script + namespace: trl-demo +data: + backend.py: | + import json, random, time, uuid + from http.server import BaseHTTPRequestHandler, HTTPServer + + # Must track config.yaml's estimate_tokens per tier exactly -- see the + # ConfigMap-level comment above for why a mismatch matters. + TOTAL_TOKENS_BY_TIER = {"gold": 15, "silver": 7} + + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + length = int(self.headers.get("Content-Length", 0)) + if length: + self.rfile.read(length) + # Real, non-zero, varying processing latency -- not instantaneous, + # not identical between requests. + time.sleep(random.uniform(0.008, 0.045)) + tier = self.headers.get("x-tier", "") + total_tokens = TOTAL_TOKENS_BY_TIER.get(tier, 10) + prompt_tokens = total_tokens // 2 + body = json.dumps({ + "id": f"chatcmpl-{uuid.uuid4().hex[:24]}", + "object": "chat.completion", + "created": int(time.time()), + "choices": [ + {"index": 0, "message": {"role": "assistant", "content": "hi"}, "finish_reason": "stop"} + ], + "usage": { + "prompt_tokens": prompt_tokens, + "completion_tokens": total_tokens - prompt_tokens, + "total_tokens": total_tokens, + }, + }).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *_args): + pass + + HTTPServer(("0.0.0.0", 3000), Handler).serve_forever() +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: backend + namespace: trl-demo + labels: { app: backend } +spec: + replicas: 1 + selector: + matchLabels: { app: backend } + template: + metadata: + labels: { app: backend } + spec: + containers: + - name: backend + image: docker.io/library/python:3.13-alpine + command: ["python3", "/srv/backend.py"] + volumeMounts: + - name: script + mountPath: /srv + ports: + - containerPort: 3000 + volumes: + - name: script + configMap: + name: backend-script +--- +apiVersion: v1 +kind: Service +metadata: + name: backend + namespace: trl-demo +spec: + selector: { app: backend } + ports: + - port: 3000 + targetPort: 3000 diff --git a/demos/token-rate-limit-per-app-budgets/k8s/03-gateways.yaml b/demos/token-rate-limit-per-app-budgets/k8s/03-gateways.yaml new file mode 100644 index 0000000..b37f161 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/k8s/03-gateways.yaml @@ -0,0 +1,94 @@ +# gateway-a and gateway-b: two independent replicas of the *same* image and +# the *same* config.yaml (mounted from the `gateway-config` ConfigMap, created +# by deploy.sh from the sibling ../config.yaml -- single source of truth, +# same file the docker-compose version used), pointed at the same shared +# Valkey backend. Their only relationship is sharing that image/config; each +# runs as its own Deployment/Pod so it has its own real, independently +# discoverable pod identity. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gateway-a + namespace: trl-demo + labels: { app: gateway-a } +spec: + replicas: 1 + selector: + matchLabels: { app: gateway-a } + template: + metadata: + labels: { app: gateway-a } + spec: + containers: + - name: gateway + image: docker.io/library/praxis-ai-trl-demo:local + imagePullPolicy: Never + args: ["-c", "/etc/praxis/config.yaml"] + env: + - name: TOKEN_RATE_LIMIT_VALKEY_URL + value: "redis://valkey:6379" + volumeMounts: + - name: config + mountPath: /etc/praxis + readOnly: true + ports: + - containerPort: 8080 + volumes: + - name: config + configMap: + name: gateway-config +--- +apiVersion: v1 +kind: Service +metadata: + name: gateway-a + namespace: trl-demo +spec: + selector: { app: gateway-a } + ports: + - port: 8080 + targetPort: 8080 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gateway-b + namespace: trl-demo + labels: { app: gateway-b } +spec: + replicas: 1 + selector: + matchLabels: { app: gateway-b } + template: + metadata: + labels: { app: gateway-b } + spec: + containers: + - name: gateway + image: docker.io/library/praxis-ai-trl-demo:local + imagePullPolicy: Never + args: ["-c", "/etc/praxis/config.yaml"] + env: + - name: TOKEN_RATE_LIMIT_VALKEY_URL + value: "redis://valkey:6379" + volumeMounts: + - name: config + mountPath: /etc/praxis + readOnly: true + ports: + - containerPort: 8080 + volumes: + - name: config + configMap: + name: gateway-config +--- +apiVersion: v1 +kind: Service +metadata: + name: gateway-b + namespace: trl-demo +spec: + selector: { app: gateway-b } + ports: + - port: 8080 + targetPort: 8080 diff --git a/demos/token-rate-limit-per-app-budgets/k8s/04-apps.yaml b/demos/token-rate-limit-per-app-budgets/k8s/04-apps.yaml new file mode 100644 index 0000000..25a13f5 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/k8s/04-apps.yaml @@ -0,0 +1,381 @@ +# app-a, app-b, app-c: each a *real*, separate pod that makes its own +# server-side call to its gateway -- not a header value faked from one +# browser tab. Each returns its own real pod name (from the Downward API) +# plus the *actual current* pod name behind whichever gateway Service it +# called, resolved live via the Kubernetes API (see 00-namespace-rbac.yaml), +# so "which pod served this" is never invented client-side. +apiVersion: v1 +kind: ConfigMap +metadata: + name: app-script + namespace: trl-demo +data: + app.py: | + import json, os, re, ssl, time, urllib.error, urllib.parse, urllib.request, uuid + from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + + APP_ID = os.environ["APP_ID"] + TIER = os.environ["TIER"] + POD_NAME = os.environ["POD_NAME"] + POD_IP = os.environ["POD_IP"] + NAMESPACE = os.environ["POD_NAMESPACE"] + + K8S_TOKEN_PATH = "/var/run/secrets/kubernetes.io/serviceaccount/token" + K8S_CA_PATH = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" + K8S_API = "https://kubernetes.default.svc" + ANSI_RE = re.compile(r"\x1b\[[0-9;]*m") + + def k8s_request(path, timeout=2): + with open(K8S_TOKEN_PATH) as f: + token = f.read().strip() + ctx = ssl.create_default_context(cafile=K8S_CA_PATH) + req = urllib.request.Request(f"{K8S_API}{path}", headers={"Authorization": f"Bearer {token}"}) + return urllib.request.urlopen(req, context=ctx, timeout=timeout) + + def resolve_gateway_pod_name(gw_label): + try: + with k8s_request( + f"/api/v1/namespaces/{NAMESPACE}/pods?labelSelector=app%3D{gw_label}" + ) as resp: + data = json.loads(resp.read()) + items = data.get("items", []) + if items: + return items[0]["metadata"]["name"] + return "no-pod-found" + except Exception as exc: + return f"lookup-failed:{exc.__class__.__name__}" + + def list_namespace_pods(): + try: + with k8s_request(f"/api/v1/namespaces/{NAMESPACE}/pods") as resp: + data = json.loads(resp.read()) + pods = [] + for item in data.get("items", []): + labels = item.get("metadata", {}).get("labels", {}) + statuses = item.get("status", {}).get("containerStatuses", [{}]) + ready = bool(statuses) and all(s.get("ready") for s in statuses) + pods.append({ + "role": labels.get("app", "?"), + "pod_name": item["metadata"]["name"], + "phase": item.get("status", {}).get("phase", "Unknown"), + "ready": ready, + }) + pods.sort(key=lambda p: p["role"]) + return pods + except Exception as exc: + return [{"role": "?", "pod_name": f"lookup-failed:{exc.__class__.__name__}", "phase": "Unknown", "ready": False}] + + class Handler(BaseHTTPRequestHandler): + def _cors(self): + self.send_header("Access-Control-Allow-Origin", "*") + self.send_header("Access-Control-Allow-Methods", "POST, OPTIONS") + self.send_header("Access-Control-Allow-Headers", "Content-Type") + + def do_OPTIONS(self): + self.send_response(204) + self._cors() + self.end_headers() + + def _json(self, payload): + body = json.dumps(payload).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self._cors() + self.end_headers() + self.wfile.write(body) + + def do_GET(self): + # No-cost identity lookups (never call the gateway, so these never + # touch a rate-limit budget): the dashboard uses these at load + # time to show real pod names before any request is sent. + parsed = urllib.parse.urlsplit(self.path) + if parsed.path == "/whoami": + self._json({"app_pod": POD_NAME, "app_id": APP_ID, "tier": TIER, "pod_ip": POD_IP}) + return + if parsed.path == "/gateway-pod": + gw = urllib.parse.parse_qs(parsed.query).get("gw", ["a"])[0] + self._json({"gateway": gw, "gateway_pod": resolve_gateway_pod_name(f"gateway-{gw}")}) + return + if parsed.path == "/cluster-pods": + # Live "kubectl get pods -n trl-demo" equivalent, so the + # dashboard can show the whole real deployment topology (not + # just the one pod each card already links to) -- the + # single strongest "this isn't a mock" signal available, + # since it's the entire namespace's actual live state. + self._json({"namespace": NAMESPACE, "pods": list_namespace_pods()}) + return + if parsed.path == "/gw-logs": + gw = urllib.parse.parse_qs(parsed.query).get("gw", ["a"])[0] + self._stream_gateway_logs(gw) + return + self.send_response(404) + self._cors() + self.end_headers() + + def _sse_send(self, data): + for line in data.splitlines() or [""]: + self.wfile.write(f"data: {line}\n".encode()) + self.wfile.write(b"\n") + self.wfile.flush() + + def _stream_gateway_logs(self, gw): + # Streams the *real* stdout of the gateway- pod -- the exact + # same data `kubectl logs -f` reads -- via the Kubernetes API's + # pods/log subresource (?follow=true), as Server-Sent Events. + # ANSI color codes are stripped for plain display; nothing else + # about the log content is altered or synthesized. + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Cache-Control", "no-cache") + self.send_header("Connection", "keep-alive") + self._cors() + self.end_headers() + pod = resolve_gateway_pod_name(f"gateway-{gw}") + if pod.startswith("lookup-failed") or pod == "no-pod-found": + self._sse_send(f"[dashboard] could not resolve gateway-{gw} pod: {pod}") + return + self._sse_send(f"[dashboard] streaming real stdout from pod {pod} (kubectl logs -f equivalent)") + try: + # tailLines=0: no historical backlog. The gateway pod may have + # been running (and logging requests from earlier app-pod + # generations with now-recycled IPs) long before this browser + # session's ipToApp map was built, so replaying old lines + # would show unresolvable stale IPs instead of app names. + # Starting empty and filling live as "Run scenario" executes + # is both correct and a clearer signal that this is real. + with k8s_request( + f"/api/v1/namespaces/{NAMESPACE}/pods/{pod}/log" + f"?follow=true&tailLines=0×tamps=false", + timeout=None, + ) as resp: + for raw_line in resp: + line = ANSI_RE.sub("", raw_line.decode(errors="replace")).rstrip("\n") + if line: + self._sse_send(line) + except (BrokenPipeError, ConnectionResetError): + pass + except Exception as exc: + try: + self._sse_send(f"[dashboard] log stream ended: {exc.__class__.__name__}") + except Exception: + pass + + def do_POST(self): + length = int(self.headers.get("Content-Length", 0)) + raw = self.rfile.read(length) if length else b"{}" + try: + req_body = json.loads(raw or b"{}") + except Exception: + req_body = {} + gw = req_body.get("gateway", "a") + gw_host = f"gateway-{gw}" + + gw_req = urllib.request.Request( + f"http://{gw_host}:8080/v1/chat/completions", + data=json.dumps( + {"model": "gpt-4", "messages": [{"role": "user", "content": "hi"}]} + ).encode(), + headers={ + "Content-Type": "application/json", + "x-tier": TIER, + "x-app-id": APP_ID, + }, + method="POST", + ) + t0 = time.monotonic() + try: + with urllib.request.urlopen(gw_req, timeout=5) as resp: + status = resp.status + resp_headers = {k.lower(): v for k, v in resp.getheaders()} + resp_body = resp.read() + except urllib.error.HTTPError as e: + status = e.code + resp_headers = {k.lower(): v for k, v in e.headers.items()} + resp_body = e.read() + except urllib.error.URLError as e: + status = 503 + resp_headers = {} + resp_body = json.dumps({"error": f"gateway unreachable: {e.reason}"}).encode() + latency_ms = (time.monotonic() - t0) * 1000 + + try: + backend_body = json.loads(resp_body) if resp_body else None + except Exception: + backend_body = None + + out = { + "app_pod": POD_NAME, + "app_id": APP_ID, + "tier": TIER, + "gateway": gw, + "gateway_pod": resolve_gateway_pod_name(gw_host), + "status": status, + "latency_ms": round(latency_ms, 1), + "rate_limit": { + "limit_tokens": resp_headers.get("x-ratelimit-limit-tokens"), + "remaining_tokens": resp_headers.get("x-ratelimit-remaining-tokens"), + "retry_after": resp_headers.get("retry-after"), + "reset_tokens": resp_headers.get("x-ratelimit-reset-tokens"), + }, + "backend_body": backend_body, + "request_id": str(uuid.uuid4()), + } + self._json(out) + + def log_message(self, *_args): + pass + + # Threading server: a long-lived /gw-logs SSE stream must not block + # concurrent /send requests from the same pod. + ThreadingHTTPServer(("0.0.0.0", 4000), Handler).serve_forever() +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app-a + namespace: trl-demo + labels: { app: app-a } +spec: + replicas: 1 + selector: + matchLabels: { app: app-a } + template: + metadata: + labels: { app: app-a } + spec: + serviceAccountName: trl-demo-app + containers: + - name: app + image: docker.io/library/python:3.13-alpine + command: ["python3", "/srv/app.py"] + env: + - name: APP_ID + value: "app-a" + - name: TIER + value: "gold" + - name: POD_NAME + valueFrom: { fieldRef: { fieldPath: metadata.name } } + - name: POD_NAMESPACE + valueFrom: { fieldRef: { fieldPath: metadata.namespace } } + - name: POD_IP + valueFrom: { fieldRef: { fieldPath: status.podIP } } + volumeMounts: + - name: script + mountPath: /srv + ports: + - containerPort: 4000 + volumes: + - name: script + configMap: { name: app-script } +--- +apiVersion: v1 +kind: Service +metadata: + name: app-a + namespace: trl-demo +spec: + selector: { app: app-a } + ports: + - port: 4000 + targetPort: 4000 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app-b + namespace: trl-demo + labels: { app: app-b } +spec: + replicas: 1 + selector: + matchLabels: { app: app-b } + template: + metadata: + labels: { app: app-b } + spec: + serviceAccountName: trl-demo-app + containers: + - name: app + image: docker.io/library/python:3.13-alpine + command: ["python3", "/srv/app.py"] + env: + - name: APP_ID + value: "app-b" + - name: TIER + value: "silver" + - name: POD_NAME + valueFrom: { fieldRef: { fieldPath: metadata.name } } + - name: POD_NAMESPACE + valueFrom: { fieldRef: { fieldPath: metadata.namespace } } + - name: POD_IP + valueFrom: { fieldRef: { fieldPath: status.podIP } } + volumeMounts: + - name: script + mountPath: /srv + ports: + - containerPort: 4000 + volumes: + - name: script + configMap: { name: app-script } +--- +apiVersion: v1 +kind: Service +metadata: + name: app-b + namespace: trl-demo +spec: + selector: { app: app-b } + ports: + - port: 4000 + targetPort: 4000 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app-c + namespace: trl-demo + labels: { app: app-c } +spec: + replicas: 1 + selector: + matchLabels: { app: app-c } + template: + metadata: + labels: { app: app-c } + spec: + serviceAccountName: trl-demo-app + containers: + - name: app + image: docker.io/library/python:3.13-alpine + command: ["python3", "/srv/app.py"] + env: + - name: APP_ID + value: "app-c" + - name: TIER + value: "gold" + - name: POD_NAME + valueFrom: { fieldRef: { fieldPath: metadata.name } } + - name: POD_NAMESPACE + valueFrom: { fieldRef: { fieldPath: metadata.namespace } } + - name: POD_IP + valueFrom: { fieldRef: { fieldPath: status.podIP } } + volumeMounts: + - name: script + mountPath: /srv + ports: + - containerPort: 4000 + volumes: + - name: script + configMap: { name: app-script } +--- +apiVersion: v1 +kind: Service +metadata: + name: app-c + namespace: trl-demo +spec: + selector: { app: app-c } + ports: + - port: 4000 + targetPort: 4000 diff --git a/demos/token-rate-limit-per-app-budgets/k8s/05-dashboard.yaml b/demos/token-rate-limit-per-app-budgets/k8s/05-dashboard.yaml new file mode 100644 index 0000000..f2b66bd --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/k8s/05-dashboard.yaml @@ -0,0 +1,45 @@ +# Same-origin dashboard + reverse proxy, now fronting the app-a/b/c pods +# (not the gateways directly) so every scenario request routes through +# that app's own real pod. The `dashboard-assets` ConfigMap is created by +# deploy.sh from the sibling ../dashboard/{index.html,nginx.conf} -- single +# source of truth, same files used locally. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: dashboard + namespace: trl-demo + labels: { app: dashboard } +spec: + replicas: 1 + selector: + matchLabels: { app: dashboard } + template: + metadata: + labels: { app: dashboard } + spec: + containers: + - name: dashboard + image: docker.io/library/nginx:alpine + volumeMounts: + - name: assets + mountPath: /etc/nginx/conf.d/default.conf + subPath: nginx.conf + - name: assets + mountPath: /usr/share/nginx/html/index.html + subPath: index.html + ports: + - containerPort: 3000 + volumes: + - name: assets + configMap: { name: dashboard-assets } +--- +apiVersion: v1 +kind: Service +metadata: + name: dashboard + namespace: trl-demo +spec: + selector: { app: dashboard } + ports: + - port: 3000 + targetPort: 3000 diff --git a/demos/token-rate-limit-per-app-budgets/k8s/deploy.sh b/demos/token-rate-limit-per-app-budgets/k8s/deploy.sh new file mode 100644 index 0000000..d866180 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/k8s/deploy.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Deploys the token-rate-limit-per-app-budgets demo to a kind cluster. +# +# Prereqs: +# - A kind cluster already exists (KIND_EXPERIMENTAL_PROVIDER=podman kind +# create cluster --name trl-demo) and images are loaded into it -- see +# ../README.md's "Run on Kubernetes (kind)" section. +# - kubectl context points at that cluster. +set -euo pipefail +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +demo_root="$(dirname "$here")" + +kubectl apply -f "$here/00-namespace-rbac.yaml" + +kubectl -n trl-demo create configmap gateway-config \ + --from-file=config.yaml="$demo_root/config.yaml" \ + --dry-run=client -o yaml | kubectl apply -f - + +kubectl -n trl-demo create configmap dashboard-assets \ + --from-file=index.html="$demo_root/dashboard/index.html" \ + --from-file=nginx.conf="$demo_root/dashboard/nginx.conf" \ + --dry-run=client -o yaml | kubectl apply -f - + +kubectl apply -f "$here/01-valkey.yaml" +kubectl apply -f "$here/02-backend.yaml" +kubectl apply -f "$here/03-gateways.yaml" +kubectl apply -f "$here/04-apps.yaml" +kubectl apply -f "$here/05-dashboard.yaml" + +kubectl -n trl-demo rollout status deployment/valkey --timeout=60s +kubectl -n trl-demo rollout status deployment/backend --timeout=60s +kubectl -n trl-demo rollout status deployment/gateway-a --timeout=60s +kubectl -n trl-demo rollout status deployment/gateway-b --timeout=60s +kubectl -n trl-demo rollout status deployment/app-a --timeout=60s +kubectl -n trl-demo rollout status deployment/app-b --timeout=60s +kubectl -n trl-demo rollout status deployment/app-c --timeout=60s +kubectl -n trl-demo rollout status deployment/dashboard --timeout=60s + +echo "" +echo "All pods ready:" +kubectl -n trl-demo get pods -o wide diff --git a/demos/token-rate-limit-per-app-budgets/recording/RECORDING.md b/demos/token-rate-limit-per-app-budgets/recording/RECORDING.md new file mode 100644 index 0000000..635edbc --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/recording/RECORDING.md @@ -0,0 +1,359 @@ +# Recording spike: token-rate-limit-per-app-budgets via traffic-theater + +## What this is + +A narrated, evidence-backed recording of the per-app token budget scenario +(see `../README.md`), produced with +[nerdalert/traffic-theater](https://github.com/nerdalert/traffic-theater) +(`feat/reusable-recording-toolkit` branch), driving a real browser against a +live two-gateway + Valkey docker-compose stack -- not a mockup or a scripted +screen capture. This recording covers the **mixed-algorithm** scenario: +`gold-tier` (`sliding_window`) and `silver-tier` (`token_bucket`), matched by +an `x-tier` header, per ai#789/praxis#551. + +**Watch:** `output/k8s-real-pods-token-rate-limit.mp4` (1920x1080, h264/aac, +~143s) + +`slides/proof-agenda.html`'s palette/fonts were restyled to a generic, +non-branded look after this video was recorded, so the shipped clip's intro +still shows the earlier Red Hat-styled cards -- flagged here pending a +re-record. + +An earlier, single-algorithm (`sliding_window`-only) recording of this same +demo existed before the mixed-algorithm scenario landed; it has been +superseded and removed since it no longer matches the current +`config.yaml`/`dashboard/` (which only support the two-rule, `x-tier`-matched +scenario). + +## Update: real Kubernetes deployment, retuned token values, live cluster topology + +The stack now runs on a real `kind` Kubernetes cluster (`k8s/`, `deploy.sh`) +instead of `docker-compose.yml` for recording purposes (the compose file is +kept in sync and still works for the `curl` walkthrough in `../README.md`, +but the video itself is produced against the K8s deployment). This +addressed feedback that the previous recording "read like a fake video": + +- **Real pod identities everywhere**, not placeholder names: every + `budget-card`, gateway-log panel, and the new namespace/pod-status strip + at the top of the dashboard shows actual `metadata.name` pod names + resolved live via the Kubernetes API (see `k8s/00-namespace-rbac.yaml`'s + RBAC grant and `k8s/04-apps.yaml`'s `/whoami`, `/gateway-pod`, and + `/cluster-pods` endpoints) -- app-a/b/c keep their role labels as the + primary identifier (they're the real `x-app-id` rate-limit key), but the + pod name shown alongside each is the literal, currently-scheduled pod. +- **Live gateway stdout**, streamed via Server-Sent Events from the + Kubernetes `pods/log` subresource (`k8s/04-apps.yaml`'s `/gw-logs` + endpoint), client-side reformatted for legibility and with `client_ip` + resolved back to an app name via each app's Downward-API `pod_ip`. +- **Retuned `estimate_tokens`** for more visible traffic: `gold-tier` + 15/40 (was 40/40) and `silver-tier` 7/10 (was 10/10), so a single + request no longer exhausts either budget outright -- see + `../config.yaml`'s comment and `../README.md`'s "Validate the request + flow" for the exact admit/deny/recover math this produces, and why + `estimate_tokens` must still exactly match the stub backend's (now + tier-aware, see `k8s/02-backend.yaml`) reported `usage.total_tokens`. + The scenario grew from 7 to 8 requests: gold-tier now shows a genuine + 3-request burst (admit, admit, deny) instead of a 1-request cliff-edge, + and silver-tier's recovery is demonstrably much faster (a couple of + seconds) than gold-tier's (the full 10s window). +- **A live namespace/pod-status strip** (`#cluster-strip` in + `dashboard/index.html`) shows every pod in the `trl-demo` namespace -- + not just the ones each card already links to -- fetched from the real + Kubernetes API and polled every 5s, as the single strongest "this isn't + a mock" signal available (the entire namespace's actual live state, not + a curated subset). + +All narration/timing (`narration/narration.txt`, `.srt`) and +`playwright/record.mjs`'s `requireGate` assertions were updated to match +the new 8-request scenario; see their headers/comments for specifics. The +"Seven live HTTP assertions" and per-algorithm numeric details in the +sections below describe the *original* mixed-algorithm recording's +original `capacity: 40, estimate_tokens: 40` / `capacity: 10, +estimate_tokens: 10` numbers -- they are not the current `estimate_tokens` +values (see this section and `../config.yaml` for those), and the +"reconciliation asymmetry" language they and the following section +originally used has since been corrected (see "Correction" section below). +The scenario is now 8 requests, not 7. + +## Dashboard v2: live per-app gauges/charts + narration-paced timeline + +The first mixed-algorithm cut fired all 7 requests within the first ~15s +after `dashboard/`'s "Run scenario" button was clicked, then held on a +static log for the remaining ~60s of the ~77s narration -- correct evidence, +but visually inert for most of the clip, and it gave viewers no way to see +*how* the two algorithms differ, only their end states (200/429). Fixed by: + +- **Narration-paced offsets.** Each request now fires at an offset (from + `dashboard/index.html`'s `runScenario()`) chosen against + `narration/narration.srt`'s actual cue timestamps, so the dashboard keeps + producing new, narrated action for the full ~74s of the scenario instead + of finishing in the first 15s. A short non-request "algorithm assignment" + beat and a closing recap beat (which checks off the "what this proves" + list live) were added so there's always something happening on screen, + not just a static end state. +- **Live per-app budget gauges + 30s rolling sparklines.** A client-side + `BudgetModel` per app replays the *same* arithmetic the Lua + reserve/reconcile scripts perform server-side -- + `sliding_window`: capacity minus reservations still inside the trailing + window; `token_bucket`: continuous linear refill up to capacity, drained + per reservation -- driven off each request's real timestamp and cost. + A `requestAnimationFrame` loop samples it every frame, so gold-tier's + budget visibly sits flat at zero until the window slides (a step + function/"cliff"), while silver-tier's visibly ramps upward continuously + as it refills -- the exact visual contrast between the two algorithms + that a plain admit/deny log couldn't show. This needs no extra requests + to the gateways (which would themselves cost budget); every *transition* + the model predicts is still cross-checked against the real HTTP + status/headers of each actual request via `requireGate`, so the model is + a visualization layer on top of the live evidence, not a replacement for + it. +- **A caught timing bug, found by dry-running the new offsets first.** + The original gap between silver-tier's exhaustion (its first admitted + request) and the cross-instance denial check that was supposed to prove + it (5.3s) was *longer* than `token_bucket`'s exact full-refill time + (`capacity: 10 / refill_rate: 2` = 5.0s) -- so by the time of the check, + the bucket had already silently refilled and the "expected 429" request + returned 200 instead. Caught with a plain curl replay of the new offsets + against the live stack *before* spending an actual browser recording run + on it (see the dry-run transcript below); fixed by tightening the gap to + 2.0s (4/10 tokens refilled at that point, comfortably below the 10 + needed for admission). + +```text +$ curl replay of the new offsets, live stack, before re-recording: +t=9732ms app-a@a (gold) -> 200 (admitted) +t=13166ms app-a@b (gold) -> 429 (denied, cross-instance) +t=20297ms app-c@a (gold) -> 200 (admitted, own untouched budget) +t=25328ms app-b@a (silver) -> 200 (admitted) +t=30658ms app-b@b (silver) -> 200 (BUG: expected 429, bucket already refilled) +t=44687ms app-a@a (gold) -> 200 (recovered) +t=51718ms app-b@b (silver) -> 200 (recovered) + +after tightening the exhaustion->check gap from 5.3s to 2.0s: +t=... app-a@a (gold) -> 200 +t=... app-a@b (gold) -> 429 +t=... app-c@a (gold) -> 200 +t=... app-b@a (silver) -> 200 +t=... app-b@b (silver) -> 429 (fixed) +t=... app-a@a (gold) -> 200 +t=... app-b@b (silver) -> 200 +``` + +## What it proves (evidence-manifest.json) + +Seven live HTTP assertions (`requireGate` in `playwright/record.mjs`), each +checked against the real running gateways during the recording, not after +the fact: + +1. app-a (`gold-tier`, `sliding_window`) admitted on Gateway A (`200`). +2. app-a denied on Gateway B, the *other* process (`429`, + `X-RateLimit-Remaining-Tokens: 0`) -- proves the shared Valkey ledger, + not per-replica state. +3. app-c (`gold-tier`, `sliding_window`) admitted on Gateway A (`200`) on + its own untouched budget -- a third independent app, same rule, same + Valkey namespace. +4. app-b (`silver-tier`, `token_bucket`) admitted on Gateway A (`200`) -- a + different rule entirely, matched purely on `x-tier`. +5. app-b denied on Gateway B (`429`) -- proves the shared-Valkey, + cross-instance guarantee holds for `token_bucket` too, not just + `sliding_window`. +6. app-a, previously denied, admitted again on Gateway A (`200`) once the + 10s sliding window ages its earlier reservation out -- proves the + window *slides* (budget recovers continuously as usage ages out) rather + than requiring a manual reset or a fixed reset boundary. +7. app-b, previously denied, admitted again on Gateway B (`200`) once + `silver-tier`'s bucket refilled continuously past its `estimate_tokens` + cost -- a *different* recovery mechanism than app-a's window slide, also + with no manual reset. + +This is the exact same causal sequence as the curl walkthrough in +`../README.md`, driven through a browser dashboard (`../dashboard/`) instead +of curl, so it can be recorded. + +## Why a dashboard exists + +`traffic-theater`'s `scripts/record.sh` hard-requires a Playwright script +driving a real browser against a `baseUrl` web app -- there is no +terminal/CLI recording path in the toolkit as shipped. This demo had no +browser surface, so `dashboard/` (a static page + nginx same-origin reverse +proxy, see `../dashboard/nginx.conf`) was added purely for recording. It +wraps the identical HTTP contract as the curl walkthrough; it is not part of +the Praxis AI filter chain and does not change anything about the filter +under test. + +`exact_trace`, the evidence gate every one of `traffic-theater`'s three +shipped examples requires, was deliberately **not** used here: this scenario +is a single-hop admission decision (no cross-service routing), so there is +no distributed trace to correlate. `evidence.required` in `production.yaml` +was scoped instead to `per_rule_algorithm_choice`, +`cross_instance_shared_budget`, `per_app_isolation`, `reservation_admission`, +`window_recovery`, and `token_bucket_recovery` -- matched 1:1 to the +`requireGate` assertions above. + +## Correction: an earlier version of this doc claimed a false algorithm asymmetry + +An earlier pass of this recording's tuning notes claimed that +`token_bucket` credits an estimate/actual refund back into its balance +while `sliding_window` does not retroactively shrink what's counted +against its window -- framed as a genuine, previously-undocumented +asymmetry between the two algorithms. **That claim was wrong** and has +been removed from this doc, `../README.md`, `../config.yaml`, and +`evidence-manifest.json`. Re-reading the source branch's actual reconcile +implementations (`token_rate_limit::ledger::Ledger::reconcile` and +`token_rate_limit::token_bucket_ledger::TokenBucketLedger::reconcile`) +shows both apply the identical `estimate`-vs-`actual` delta: `sliding_window` +records the *settled* usage entry at the actual token count (not the +estimate), and since window usage is summed fresh from settled + active +entries on every call, an overestimate genuinely does shrink what's counted +against the window -- there is a dedicated passing unit test for exactly +this, +`reconcile_releases_unused_tokens_on_overestimate` (`filters/src/token_rate_limit/tests.rs`). +Both algorithms reconcile symmetrically as currently implemented. + +What actually needed fixing, and what the retuning above was really about: +this demo's stub backend originally reported a single **fixed** +`usage.total_tokens` value regardless of which tier's request it was +serving, which could mismatch whatever `estimate_tokens` a tier configured +-- a demo-configuration bug, not an algorithm-level design difference. It's +now fixed by making the stub backend tier-aware (see `../k8s/02-backend.yaml` +and `../docker-compose.yml`), so `estimate_tokens` always matches +`usage.total_tokens` exactly for both tiers and reconciliation's refund/ +overage math nets to ~0 either way. There is no known open design question +here upstream; this section previously implied one that doesn't hold up +against the actual code. + +## What "window recovery" and "bucket recovery" mean, and what they don't + +`gold-tier`'s `window: 10s` and `silver-tier`'s `refill_rate: 2` tokens/sec +are both deliberately small so recovery is watchable; a production +deployment would use much larger/slower values (the demo's own +`../config.yaml` comments this accordingly). What's demonstrated is each +algorithm's *own* recovery mechanic: + +- **`sliding_window`**: no reset event, no restart, no cron job -- app-a's + budget recovers purely because its earlier reservation ages past the + trailing 10s window as real time elapses. +- **`token_bucket`**: no reset event either, but the mechanism is + different -- app-b's bucket refills continuously at a fixed rate + (2 tokens/sec here) up to `capacity`, so it can become admissible again + well before a fixed window would elapse. + +See `../README.md`'s "Open design questions" section for why the choice +between these algorithms (and which one, if either, should be the default) +is still open upstream. + +## Known deviations from the toolkit's default pipeline + +`scripts/generate-narration.sh` and `scripts/generate-captions.sh` both +hard-require `OPENAI_API_KEY` (real OpenAI TTS + Whisper transcription). +**No such key was available in this environment**, so: + +- `narration/narration.wav` was generated with a local, offline + substitute -- macOS `say -v Samantha`, converted to WAV via `ffmpeg` -- + instead of the toolkit's OpenAI TTS call. +- `narration/narration.srt` caption timing is an **approximation** + (proportional to sentence character count over the measured audio + duration), not a real Whisper word-level transcription, via a small ad + hoc substitute script (not part of the upstream toolkit). + +Everything else -- the production schema, the live browser recording, the +`requireGate` evidence assertions, the ffmpeg assembly, and the final media +validation (`src/validation/validate-media.js`) -- used the toolkit exactly +as shipped, unmodified, against a genuinely live stack. + +**To get the toolkit's canonical pipeline output** (real OpenAI TTS + +Whisper captions), re-run with `OPENAI_API_KEY` set: + +```bash +export OPENAI_API_KEY=... +./scripts/generate-narration.sh examples/token-rate-limit-per-app-budgets +./scripts/generate-captions.sh examples/token-rate-limit-per-app-budgets +./scripts/assemble.sh examples/token-rate-limit-per-app-budgets +``` + +## Recording environment + +This mixed-algorithm recording was produced on a remote RHEL 9 lab host +(Podman + Compose, not Docker Desktop) rather than the local macOS spike +environment used for the original single-algorithm recording -- macOS +Podman machine networking was unreliable in this session's local +environment. Two host-specific things came up that aren't part of the +demo itself: + +- No `ffmpeg`/`ffprobe` package is available via `dnf` on this host (no + RPM Fusion/EPEL configured); a static build from + [johnvansickle.com](https://johnvansickle.com/ffmpeg/) was used instead. +- Podman on an SELinux-enforcing host denies containers read access to + bind-mounted files that aren't labeled for container access. Rather than + changing the repo's `docker-compose.yml` (which would add an SELinux-only + `:z`/`:Z` mount flag that other contributors on non-SELinux hosts don't + need), the bind-mounted files were relabeled on the host directly: + `chcon -Rt container_file_t config.yaml dashboard/nginx.conf + dashboard/index.html`. Podman/Docker on non-SELinux hosts (macOS, most + default Linux distros) need neither of these workarounds. + +## NDA scrubbing + +App names throughout (`app-a`, `app-b`, `app-c`) are placeholders. The real +customer scenario's application names are excluded here and enforced via +`production.yaml`'s `redaction.deny` list. + +## Reproduce from scratch + +Self-contained -- assumes nothing already checked out locally. Requires +Docker or Podman with Compose, Git, Node.js/npm, and ffmpeg/ffprobe on +`PATH` (`traffic-theater`'s own `scripts/doctor.sh` checks these). On an +SELinux-enforcing host, see "Recording environment" above for the extra +`chcon` step. + +```bash +# 1. Clone this repo and the filter's source branch as siblings, then bring +# up the demo stack (see ../README.md for the full walkthrough) +git clone https://github.com/praxis-proxy/experimental.git +git clone --branch jordigilh/token-rate-limit-per-app-budgets \ + https://github.com/jordigilh/praxis-ai.git praxis-ai-trl-demo + +cd experimental/demos/token-rate-limit-per-app-budgets +export PRAXIS_AI_SRC=../../../praxis-ai-trl-demo +podman compose up --build -d # or: docker compose up --build -d +podman exec token-rate-limit-per-app-budgets-valkey-1 valkey-cli FLUSHALL + +# 2. Clone traffic-theater and install deps (one-time) +git clone --depth 1 -b feat/reusable-recording-toolkit \ + https://github.com/nerdalert/traffic-theater.git +cd traffic-theater +npm install +npx playwright install chromium + +# 3. Copy this example's recording assets into the clone's examples/ dir +mkdir -p examples/token-rate-limit-per-app-budgets +cp -r ../recording/{production.yaml,slides,narration,playwright} \ + examples/token-rate-limit-per-app-budgets/ + +# 4. Validate, record, assemble, validate media +node src/validation/validate-production.js \ + examples/token-rate-limit-per-app-budgets/production.yaml +node examples/token-rate-limit-per-app-budgets/playwright/record.mjs +./scripts/assemble.sh examples/token-rate-limit-per-app-budgets +node src/validation/validate-media.js \ + examples/token-rate-limit-per-app-budgets/output/final.mp4 + +# 5. Tear down the demo stack when done +cd ../../experimental/demos/token-rate-limit-per-app-budgets +podman compose down -v +``` + +## Confidence + +High (>=90%) on everything demonstrated: every HTTP status/header claim in +the video was asserted live against the real stack, not staged, and the +final media passed the toolkit's own validation gate. The flagged +deviations (local TTS, approximate captions, the lab-host ffmpeg/SELinux +workarounds) are cosmetic/environment-layer, not evidentiary -- they don't +affect what was actually proven about the filter's behavior. An earlier +pass of this doc also claimed a "reconciliation asymmetry" between the two +algorithms as a finding from tuning this recording; that claim did not hold +up against the source branch's actual reconcile code and tests, and has +been corrected above -- flagged here as a reminder that this doc's own +design-adjacent claims (as opposed to the live HTTP assertions) need the +same evidentiary bar, not just plausibility. diff --git a/demos/token-rate-limit-per-app-budgets/recording/evidence-manifest.json b/demos/token-rate-limit-per-app-budgets/recording/evidence-manifest.json new file mode 100644 index 0000000..b372e18 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/recording/evidence-manifest.json @@ -0,0 +1,44 @@ +{ + "production": "token-rate-limit-per-app-budgets", + "demonstrated": [ + "per_rule_algorithm_choice: app-a/app-c matched to gold-tier (sliding_window) via x-tier: gold, app-b matched to silver-tier (token_bucket) via x-tier: silver -- same token_rate_limit filter instance, two algorithms chosen per rule", + "cross_instance_shared_budget: app-a admitted on Gateway A (HTTP 200, 15/40 reserved), admitted again on Gateway B (HTTP 200, 30/40 reserved -- the OTHER process, same shared ledger), then denied on a third request back on Gateway A (HTTP 429, X-RateLimit-Remaining-Tokens: 0) once the shared budget is exhausted (sliding_window)", + "cross_instance_shared_budget: app-b admitted on Gateway A (HTTP 200, 7/10 reserved), then denied on Gateway B immediately after (HTTP 429) for the same shared Valkey ledger -- proving the cross-instance guarantee holds for token_bucket too, not just sliding_window", + "per_app_isolation: app-c admitted on Gateway A (HTTP 200) on its own independent, untouched budget, immediately after app-a's exhaustion on the same rule/algorithm and Valkey namespace", + "reservation_admission: each admitted request reserved its estimate_tokens cost atomically via the shared ledger before the response was known, for either algorithm", + "bucket_recovery: app-b, previously denied, is admitted again on Gateway B (HTTP 200) once silver-tier's token_bucket refilled continuously past its estimate_tokens cost -- just 2 seconds (4 tokens) is already enough (3 + 4 = 7) -- no manual reset", + "window_recovery: app-a, previously denied, is admitted again on Gateway A (HTTP 200) once the full 10s sliding window aged its earlier reservations out -- a much longer wait than silver-tier needed, no manual reset, no restart, budget recovers purely because the window advanced" + ], + "configured": [ + "gold-tier (sliding_window): window: 10s, capacity: 40 tokens, estimate_tokens: 15 -- deliberately lower than capacity so a single request never exhausts the budget outright: two requests admit (30/40 reserved), a third is denied (needs 15, only 10 remain)", + "silver-tier (token_bucket): capacity: 10, refill_rate: 2 tokens/sec, estimate_tokens: 7 -- one request admits (7/10 reserved, 3 remaining), a second immediately after is denied (needs 7, only 3 remain), but a ~2s wait refills 4 tokens (3 + 4 = 7), exactly enough for a retry to succeed", + "both tiers' estimate_tokens are set to exactly match the stub backend's tier-aware usage.total_tokens (k8s/02-backend.yaml reads the x-tier header) -- reconciliation debits actual-minus-estimate as an extra charge whenever actual exceeds estimate, symmetrically for both algorithms (verified against the source branch's Ledger::reconcile/TokenBucketLedger::reconcile and their unit tests), so a mismatch here would silently over-drain either budget past this arithmetic (see RECORDING.md's 'Correction' section, which supersedes an earlier incorrect asymmetry claim)", + "bucket_key_header: x-app-id (one independent budget per app, within each rule)", + "backend: valkey, shared namespace across both gateway replicas, for both rules" + ], + "inferred": [], + "trace_ids": [], + "validation": { + "media": "1920x1080 h264/aac, ~143s -- passed src/validation/validate-media.js", + "production_schema": "passed src/validation/validate-production.js", + "live_gates": "8 requireGate assertions passed during recording against a live Kubernetes (kind) deployment (see playwright/record.mjs): app-a admitted on A, app-a admitted again on B (shared ledger accumulating), app-a denied on a third request back on A (sliding_window exhausted), app-c admitted on A (isolation), app-b admitted on A, app-b denied on B (token_bucket), app-b re-admitted on B after its bucket refilled (~2s), app-a re-admitted on A after the full 10s window aged out" + }, + "dashboard_v3": [ + "Migrated the recording stack from docker-compose to a real kind Kubernetes cluster (k8s/, deploy.sh) so the dashboard could show literal, currently-scheduled pod names (from the Downward API and live Kubernetes API lookups) instead of static app-a/b/c placeholders -- addressing feedback that the previous recording 'read like a fake video'.", + "Added a live namespace/pod-status strip (#cluster-strip) showing every pod in the trl-demo namespace (all 8: three apps, two gateways, backend, dashboard, valkey), fetched from the real Kubernetes API via each app pod's new /cluster-pods endpoint and polled every 5s.", + "Added live gateway stdout log panels, streamed via Server-Sent Events from the Kubernetes pods/log subresource, with client-side reformatting (key fields surfaced first) and client_ip resolved back to an app name via each app's Downward-API pod_ip, since raw log lines were otherwise illegible/unattributable when displayed at panel width.", + "Removed the 'Gateway A'/'Gateway B' topology cards and 'Send as app-X' buttons, now redundant with the gateway-log panels and the automated scenario respectively, freeing vertical space.", + "Retuned estimate_tokens (gold-tier 40->15, silver-tier 10->7) so a single request no longer exhausts either budget outright -- more visible admitted traffic before each denial, and gold-tier's/silver-tier's recovery speeds now contrast sharply (full 10s window vs ~2s). Required making the stub backend's usage.total_tokens tier-aware (was a single fixed constant) to keep estimate == actual and avoid reconciliation silently over-draining either budget (this reconcile math is symmetric between the two algorithms, not an asymmetry -- see RECORDING.md's 'Correction' section). The scenario grew from 7 to 8 requests; found via live E2E dry-runs, not just recomputed by hand -- an initial choice of offsets let gold-tier's first reservation age back out of its 10s window before the expected denial fired, flipping it from 429 to 200 (fixed by firing the 3-request gold-tier burst within ~1s instead of spread across narration pacing).", + "Fixed a slide-transition rendering bug in recording/slides/proof-agenda.html: the intro deck's three sections cross-faded via a 0.4s opacity transition, but since they share the same absolute position with unrelated layouts, this produced several recorded frames of both slides' text overlapping/garbled mid-transition (visible as flicker in the assembled video). Fixed by making the section switch instant (no transition) instead of cross-fading." + ], + "limitations": [ + "No OPENAI_API_KEY was available in either recording environment: narration audio used a local macOS `say` TTS fallback instead of the toolkit's OpenAI TTS (scripts/generate-narration.sh), and narration.srt caption timing is an approximate proportional-duration estimate rather than a real Whisper word-level transcription (a small ad hoc substitute script, not part of the upstream toolkit). Both are flagged deviations, not canonical toolkit output -- re-run with OPENAI_API_KEY set to get the toolkit's default pipeline.", + "No exact_trace evidence gate: this scenario is a single-hop admission decision with no cross-service routing, so there is no distributed trace to correlate. Evidence is instead the exact HTTP status/headers returned by the live gateways, asserted via requireGate during recording.", + "The browser dashboard (../dashboard/) is a demo-only recording convenience wrapping the same HTTP contract as the docker-compose README's curl walkthrough; it is not part of the Praxis AI filter chain and ships only with this recording, not upstream.", + "App names (app-a/app-b/app-c) are placeholders. The real customer scenario's application names are intentionally excluded (NDA) and enforced via production.yaml's redaction.deny list.", + "This remains early exploratory work validating ai#658's sliding-window design, the ai#129 bucket-key proposal, and the new per-rule algorithm choice (ai#789/praxis#551), not a substitute for their own review threads -- see the parent README's \"Current scope\" section.", + "An earlier version of this manifest and the parent README/RECORDING.md claimed the two algorithms' reconciliation semantics differ (token_bucket crediting refunds back into a live balance vs. sliding_window not retroactively shrinking window usage). That claim was incorrect -- re-reading the source branch's Ledger::reconcile and TokenBucketLedger::reconcile (and their unit tests, e.g. reconcile_releases_unused_tokens_on_overestimate) shows both apply the identical estimate-vs-actual delta; sliding_window does shrink counted window usage on an overestimate, via recording the settled entry at the actual token count rather than adjusting a live balance. Corrected in RECORDING.md's \"Correction\" section; there is no known open design question here upstream.", + "This recording was produced on a remote Linux (RHEL 9) lab host, not the local macOS spike environment used for the first (single-algorithm) recording -- required a static ffmpeg/ffprobe build (no matching dnf package on this host). An earlier pass on the same host used Podman + Compose directly and needed an SELinux relabel (chcon -Rt container_file_t) of bind-mounted config files; this pass instead runs a kind (Kubernetes-in-Podman) cluster (see k8s/, deploy.sh), which sidesteps that bind-mount path entirely (config/scripts are delivered via ConfigMaps, not host bind mounts).", + "The dashboard's app pods need RBAC (get/list on pods, get on pods/log; see k8s/00-namespace-rbac.yaml) to resolve gateway pod names, stream gateway logs, and list the namespace's pods live -- a demo-recording-only requirement, not something the Praxis AI filter chain itself needs at runtime." + ] +} diff --git a/demos/token-rate-limit-per-app-budgets/recording/narration/narration.srt b/demos/token-rate-limit-per-app-budgets/recording/narration/narration.srt new file mode 100644 index 0000000..2e69a72 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/recording/narration/narration.srt @@ -0,0 +1,71 @@ +1 +00:00:00,000 --> 00:00:03,487 +Before the live proof, here's what's under the hood. + +2 +00:00:03,487 --> 00:00:12,609 +Two independent Praxis AI gateways sit in front of three applications: app-a, app-b, and app-c, each governed by a per-app token budget. + +3 +00:00:12,609 --> 00:00:14,889 +Every request carries two headers. + +4 +00:00:14,889 --> 00:00:21,127 +X-tier selects one of two rules: gold-tier, a sliding window, or silver-tier, a token bucket. + +5 +00:00:21,127 --> 00:00:26,090 +X-app-id then selects that app's own budget within whichever rule matched. + +6 +00:00:26,090 --> 00:00:37,559 +Both rules reserve tokens against the same shared Valkey ledger, so the budget is shared across every gateway replica, not local to whichever instance handled the request. + +7 +00:00:37,559 --> 00:00:53,991 +Watch a few example requests move through this path: matched to a rule, reserved against the shared ledger, and only then forwarded to the backend, or stopped cold, before ever reaching a backend, if the ledger says that budget is already spent. + +8 +00:00:53,991 --> 00:01:05,997 +This demo proves per app token budgets are enforced consistently across gateway replicas, for two independently chosen rate limiting algorithms, and that each recovers on its own. + +9 +00:01:05,997 --> 00:01:10,423 +Two independent Praxis AI gateways share one Valkey backed ledger. + +10 +00:01:10,423 --> 00:01:18,875 +App-a and app-c are matched to the gold tier rule, a sliding window; app-b is matched to the silver tier rule, a token bucket. + +11 +00:01:18,875 --> 00:01:22,496 +App-a's first request, through Gateway A, is admitted. + +12 +00:01:22,496 --> 00:01:31,483 +Its next request, through Gateway B, a separate process, is also admitted: both gateways are drawing down the very same shared budget. + +13 +00:01:31,483 --> 00:01:39,330 +A third request, back on Gateway A, is denied: gold-tier's budget is now exhausted, no matter which gateway is asked. + +14 +00:01:39,330 --> 00:01:44,696 +App-c, on the same sliding window rule, is admitted on its own untouched budget. + +15 +00:01:44,696 --> 00:01:56,098 +Now app-b, on the token bucket rule, is admitted through Gateway A, then immediately denied through Gateway B: a single burst already leaves too little for a second call. + +16 +00:01:56,098 --> 00:02:04,348 +Watch app-b recover first: its bucket refills continuously, and after only a couple of seconds it already has enough again. + +17 +00:02:04,348 --> 00:02:13,536 +App-a takes longer: its sliding window has to wait for the earlier reservations to age out of the full window before it's admitted again. + +18 +00:02:13,536 --> 00:02:22,859 +Same shared ledger, same reservation guarantee for both algorithms, but very different recovery speeds, and no manual reset for either one. diff --git a/demos/token-rate-limit-per-app-budgets/recording/narration/narration.txt b/demos/token-rate-limit-per-app-budgets/recording/narration/narration.txt new file mode 100644 index 0000000..f2104c0 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/recording/narration/narration.txt @@ -0,0 +1 @@ +Before the live proof, here's what's under the hood. Two independent Praxis AI gateways sit in front of three applications: app-a, app-b, and app-c, each governed by a per-app token budget. Every request carries two headers. X-tier selects one of two rules: gold-tier, a sliding window, or silver-tier, a token bucket. X-app-id then selects that app's own budget within whichever rule matched. Both rules reserve tokens against the same shared Valkey ledger, so the budget is shared across every gateway replica, not local to whichever instance handled the request. Watch a few example requests move through this path: matched to a rule, reserved against the shared ledger, and only then forwarded to the backend, or stopped cold, before ever reaching a backend, if the ledger says that budget is already spent. This demo proves per app token budgets are enforced consistently across gateway replicas, for two independently chosen rate limiting algorithms, and that each recovers on its own. Two independent Praxis AI gateways share one Valkey backed ledger. App-a and app-c are matched to the gold tier rule, a sliding window; app-b is matched to the silver tier rule, a token bucket. App-a's first request, through Gateway A, is admitted. Its next request, through Gateway B, a separate process, is also admitted: both gateways are drawing down the very same shared budget. A third request, back on Gateway A, is denied: gold-tier's budget is now exhausted, no matter which gateway is asked. App-c, on the same sliding window rule, is admitted on its own untouched budget. Now app-b, on the token bucket rule, is admitted through Gateway A, then immediately denied through Gateway B: a single burst already leaves too little for a second call. Watch app-b recover first: its bucket refills continuously, and after only a couple of seconds it already has enough again. App-a takes longer: its sliding window has to wait for the earlier reservations to age out of the full window before it's admitted again. Same shared ledger, same reservation guarantee for both algorithms, but very different recovery speeds, and no manual reset for either one. diff --git a/demos/token-rate-limit-per-app-budgets/recording/narration/narration.wav b/demos/token-rate-limit-per-app-budgets/recording/narration/narration.wav new file mode 100644 index 0000000..822f533 Binary files /dev/null and b/demos/token-rate-limit-per-app-budgets/recording/narration/narration.wav differ diff --git a/demos/token-rate-limit-per-app-budgets/recording/output/checksums.sha256 b/demos/token-rate-limit-per-app-budgets/recording/output/checksums.sha256 new file mode 100644 index 0000000..a61f334 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/recording/output/checksums.sha256 @@ -0,0 +1,2 @@ +d6cc9ca4ae055186c5ab12e6769c5dd3647ffc2c5290083b996f85d34eb87926 k8s-real-pods-token-rate-limit.mp4 +b5ef88fdd62d6c8d65d92ba43e90839da41e40819c5e34c2b157d470e6d04321 poster.png diff --git a/demos/token-rate-limit-per-app-budgets/recording/output/k8s-real-pods-token-rate-limit.mp4 b/demos/token-rate-limit-per-app-budgets/recording/output/k8s-real-pods-token-rate-limit.mp4 new file mode 100644 index 0000000..676b2c1 Binary files /dev/null and b/demos/token-rate-limit-per-app-budgets/recording/output/k8s-real-pods-token-rate-limit.mp4 differ diff --git a/demos/token-rate-limit-per-app-budgets/recording/output/poster.png b/demos/token-rate-limit-per-app-budgets/recording/output/poster.png new file mode 100644 index 0000000..3237777 Binary files /dev/null and b/demos/token-rate-limit-per-app-budgets/recording/output/poster.png differ diff --git a/demos/token-rate-limit-per-app-budgets/recording/playwright/record.mjs b/demos/token-rate-limit-per-app-budgets/recording/playwright/record.mjs new file mode 100644 index 0000000..e1d3b22 --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/recording/playwright/record.mjs @@ -0,0 +1,116 @@ +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { execFileSync } from 'node:child_process'; +import { openRecordingBrowser } from '../../../src/browser/live.js'; +import { requireGate } from '../../../src/evidence/gates.js'; + +const here = path.dirname(fileURLToPath(import.meta.url)); +const exampleDir = path.resolve(here, '..'); +const baseUrl = process.env.DEMO_URL || 'http://127.0.0.1:3000'; +const narrationWav = path.join(exampleDir, 'narration', 'narration.wav'); + +function narrationSeconds() { + try { + const out = execFileSync( + 'ffprobe', + ['-v', 'error', '-show_entries', 'format=duration', '-of', 'default=noprint_wrappers=1:nokey=1', narrationWav], + { encoding: 'utf8' }, + ); + return Number(out.trim()); + } catch { + return 0; + } +} + +// Total on-screen time must be >= narration length, or scripts/assemble.sh's +// `-shortest` ffmpeg flag truncates the audio to match a shorter video. +const targetMs = Math.max(narrationSeconds() + 3, 30) * 1000; +const start = Date.now(); +const holdRemaining = async page => { + const remaining = targetMs - (Date.now() - start); + if (remaining > 0) await page.waitForTimeout(remaining); +}; + +const outputDir = process.env.OUTPUT_DIR || path.join(exampleDir, 'output', 'raw'); +const { browser, context, page } = await openRecordingBrowser({ videoDir: outputDir }); +try { + // proof-agenda.html is a self-advancing 3-section deck (title/cards -> + // architecture -> animated topology preview) timed to narration.srt's + // first 7 cues; 53991ms is that segment's measured duration with the + // Ava (Premium) narration voice (see RECORDING.md's "Intro deck" + // section) plus no extra padding, since the deck's own last caption + // already holds through its final frame. + await page.goto(`file://${path.join(exampleDir, 'slides', 'proof-agenda.html')}`, { waitUntil: 'load' }); + await page.waitForTimeout(53991); + + await page.goto(baseUrl, { waitUntil: 'domcontentloaded' }); + await page.waitForSelector('#run-scenario'); + await page.waitForTimeout(1500); + await page.click('#run-scenario'); + + // The re-paced scenario (see dashboard/index.html) now spreads its 8 + // requests across ~79s to track narration.srt's cues instead of firing + // them all in the first ~15s, so this deadline needs enough headroom + // above that, not just above a single request's round-trip time. + const deadline = Date.now() + 100000; + let results = null; + while (Date.now() < deadline) { + results = await page.evaluate(() => (window.__scenarioDone ? window.__scenarioResults : null)); + if (results) break; + await page.waitForTimeout(500); + } + if (!results) throw new Error('scenario did not complete within timeout'); + + const [appAOnA, appAOnB, appAOnADenied, appCOnA, appBOnA, appBOnB, appBRecovered, appARecovered] = results; + requireGate( + appAOnA.status === 200, + 'app-a (gold-tier, sliding_window) admitted on gateway A -- 15/40 reserved, 25 remaining', + appAOnA, + ); + requireGate( + appAOnB.status === 200, + 'app-a admitted again on gateway B, the OTHER process -- the same shared Valkey budget keeps accumulating: 30/40 reserved, 10 remaining', + appAOnB, + ); + requireGate( + // X-RateLimit-Remaining-Tokens is a hardcoded "0" on every 429 regardless + // of algorithm or actual usage (an MVP shortcut, not computed from real + // remaining capacity -- see filters/src/token_rate_limit/mod.rs's + // HEADER_RATELIMIT_REMAINING_TOKENS doc comment), so this only proves + // "denied", not the specific 10-tokens-short-of-15 shortfall. + appAOnADenied.status === 429 && appAOnADenied.rate_limit?.remaining_tokens === '0', + 'app-a denied on a third request, back on gateway A -- gold-tier now exhausted (needs 15, only 10 remain), proven consistent across both gateways', + appAOnADenied, + ); + requireGate( + appCOnA.status === 200, + 'app-c (gold-tier, sliding_window) admitted on gateway A on its own untouched budget', + appCOnA, + ); + requireGate( + appBOnA.status === 200, + 'app-b (silver-tier, token_bucket) admitted on gateway A -- 7/10 reserved, 3 remaining', + appBOnA, + ); + requireGate( + appBOnB.status === 429 && appBOnB.rate_limit?.remaining_tokens === '0', + 'app-b denied on gateway B immediately after -- a single burst already leaves too little (3 remaining) for a second 7-token call', + appBOnB, + ); + requireGate( + appBRecovered.status === 200, + "app-b admitted again on gateway B once silver-tier's token_bucket refilled continuously -- just 2 seconds (4 tokens) would already have been enough, with no manual reset", + appBRecovered, + ); + requireGate( + appARecovered.status === 200, + 'app-a admitted again on gateway A once the sliding window aged its earlier reservations out of the full 10s window -- a much longer wait than silver-tier needed, with no manual reset', + appARecovered, + ); + + await holdRemaining(page); +} finally { + await page.close(); + await context.close(); + await browser.close(); +} diff --git a/demos/token-rate-limit-per-app-budgets/recording/production.yaml b/demos/token-rate-limit-per-app-budgets/recording/production.yaml new file mode 100644 index 0000000..46d35ae --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/recording/production.yaml @@ -0,0 +1,22 @@ +production: + id: token-rate-limit-per-app-budgets + title: Per-app token budgets across gateway replicas + output: { width: 1920, height: 1080, fps: 30, videoCodec: h264, audioCodec: aac } +browser: { baseUrl: http://127.0.0.1:3000, viewport: { width: 1920, height: 1080 } } +scenes: + - { id: agenda, kind: slide, source: slides/proof-agenda.html, narration: narration/narration.txt } + - { id: live-scenario, kind: browser, script: playwright/record.mjs, narration: narration/narration.txt } +evidence: + required: + [ + per_rule_algorithm_choice, + cross_instance_shared_budget, + per_app_isolation, + reservation_admission, + window_recovery, + token_bucket_recovery, + ] +# No redaction.deny list: app-a/app-b/app-c (and every other identifier in +# this scenario) are already fully generic placeholders, not stand-ins for +# real customer/internal names, so there's nothing here that needs scrubbing +# from narration/output. diff --git a/demos/token-rate-limit-per-app-budgets/recording/slides/proof-agenda.html b/demos/token-rate-limit-per-app-budgets/recording/slides/proof-agenda.html new file mode 100644 index 0000000..bde3d2e --- /dev/null +++ b/demos/token-rate-limit-per-app-budgets/recording/slides/proof-agenda.html @@ -0,0 +1,254 @@ + + + + +Per-app token budgets — proof agenda + + + +
+
TOKEN RATE LIMIT DEMONSTRATION
+

Per-app token budgets, mixed algorithms, shared across gateway replicas

+

Two independent Praxis AI gateways, three applications, one shared budget per app.

+
+

Three apps

app-a, app-b, and app-c each draw against their own token budget.

+

Two gateways

Independent replicas — same config, same Valkey namespace.

+

Two rules, two algorithms

gold-tier (sliding_window) and silver-tier (token_bucket), matched by x-tier.

+

One shared ledger

Budgets tracked in Valkey, not per-replica memory.

+
+
+ +
+
HOW THE PIECES WORK TOGETHER
+

Match a rule. Reserve a budget. Then forward.

+

Every request is matched to a rule and its cost reserved before it ever reaches the backend.

+
+
Gateway A or B
token_rate_limit filter
+
→
+
Rule match
by x-tier header
+
→
+
Shared Valkey ledger
budget keyed by x-app-id
+
→
+
Backend
only reached if admitted
+
+
+
+

3 applications

+
    +
  • app-a → gold-tier
    • +
  • app-c → gold-tier
    • +
  • app-b → silver-tier
    • +
+
+
+

2 rules, 2 algorithms

+
    +
  • gold-tier: sliding_window, window 10s, capacity 40
    • +
  • silver-tier: token_bucket, capacity 10, refill 2/s
    • +
+
+
+

1 shared backend

+
    +
  • Valkey, one namespace
    • +
  • Both gateway replicas, both rules
    • +
+
+
+
+ +
+
A PREVIEW, BEFORE THE LIVE PROOF
+

Watch two requests move through this path

+

Matched to a rule, reserved against the shared ledger, and only then forwarded — or stopped cold before ever reaching a backend.

+
+
app-a
gold-tier
+
Gateway A
independent replica
+
Gateway B
independent replica
+
Shared Valkey ledger
app-a's gold-tier budget
+
Backend
only reached if admitted
+
+
+
+
+ + + +