Operational characterization: memory profile, autoscaling signals, and resource sizing for the AI Gateway (v1)

[shaneutt]

Summary

The AI Gateway inherits every operational characteristic of the Praxis data plane (see the companion Praxis issue) and then amplifies the hardest one: memory. AI traffic is large and wildly variable (long prompts, large model responses, multimodal inputs, uploaded files and documents), and the gateway holds far more state than a generic proxy: response store and conversation history, streaming accumulation, tool-call state, compaction, agentic loops, MCP sessions, and per-tenant data. That combination produces memory usage that is both higher and much less predictable than any ordinary proxy, driven by conversation shape and model behavior rather than request rate. Sizing this on CPU or RPS alone is not viable.

Before a v1 of the AI Gateway we need a measured, documented grasp of these characteristics: how memory and CPU behave across AI features and workload profiles, what signals autoscaling must key on (including inference- and token-aware signals), and exactly what an operator should provision at a given scale. This is broader than the AI feature benchmarks in #1, which it can build on.

Goals

• Measure and document the AI Gateway's memory and CPU behavior across representative AI workloads, with emphasis on worst-case and spike behavior driven by conversation and payload shape.
• Define the autoscaling signals appropriate to an inference gateway (in-flight requests, buffered/accumulated bytes, token throughput, backend saturation) and how to export them.
• Publish resource-sizing guidance at specific, named scales and workload profiles.
• Back all of the above with automated performance, soak, and spike tests on a regular cadence, with memory-regression gating.
• Ship operator-facing documentation prior to v1.

Operational considerations to cover

AI-specific memory behavior

• Response store and conversation-history growth, including the full-history-rebuild cost (see Process conversation history without full-history rebuilds or non-spec limits #532) and incremental-storage options (Consider incremental storage for Responses history #65).
• Streaming accumulation without an aggregate ceiling (Streaming Responses accumulation has no aggregate byte ceiling #556) and unbounded tool-call/argument state growth (Streaming tool-call state grows without an aggregate limit #412, function_call_arguments.done bypasses the configured argument-size cap #559).
• Compaction memory behavior and its callout path.
• Agentic-loop memory: long multi-turn runs holding state, and MCP session lifecycle/state (MCP sessions: lifecycle management and distributed store #156).
• Buffering entire request/response bodies for API translation (OpenAI, Anthropic, Responses) when model outputs are large.
• Token-counting and metering overhead on the hot path.
• Storage-backed APIs (Files, Containers, Skills, Vector Stores) and their in-process memory during upload/stream/resolve.
• Per-tenant memory under multi-tenancy (Epic: Multi-Tenancy Support #301) and KV/Valkey memory for counters, sessions, and shared buckets.
• Local engines where applicable: embedding engine (Candle) model loading and residency, and any co-located inference image footprint.
• Long-lived SSE streams held open for long-running responses.

Inherited proxy behavior

• All memory, CPU, buffering, spike, hot-reload, and allocator considerations from the Praxis data plane (see the companion Praxis issue). The AI build should be characterized as an actual deployed configuration, not just in the abstract.

Scaling and autoscaling

• Signals appropriate to an inference gateway: in-flight requests, accumulated/buffered bytes, token throughput, response-store pressure, and backend (model/provider) saturation, in addition to CPU and memory.
• Custom/derived metrics for Kubernetes HPA and KEDA, and how the gateway exposes them.
• Behavior under overload and backpressure, and how it should interact with autoscaling and with backend rate limits and fallback (Epic: Inference Provider Fallback #249).
• Scale-up latency vs spike speed, and the headroom needed to absorb conversation- and payload-driven spikes.
• Connection draining and in-flight-response handling during scale-down.
• Vertical vs horizontal guidance, including when response-store and buffered-body memory force vertical scaling.
• Container memory requests vs limits, cgroup awareness, and OOMKill avoidance.

Observability

• Metrics required to see AI memory behavior in real time: RSS/heap, accumulated bytes per stream, response-store size, active streams, per-model in-flight counts.
• Token/usage and cost metrics and their cardinality budget (per model, per tenant) so observability does not itself become the memory problem.
• Tracing for long agentic loops and slow/large responses.
• Alerting thresholds and SLOs for memory saturation, response-store growth, and overload.
• Reference dashboards for capacity and cost planning.

Resource sizing and capacity planning

• Documented resource requirements at specific, named scales and workload profiles (for example: N concurrent conversations at a given prompt/response token distribution, with and without the response store, plus the embedding engine).
• Sizing rules of thumb per feature (marginal cost of the response store, compaction, guardrails callouts, embeddings, file/container APIs).
• Reference profiles: single-user/standalone MVP (Epic: Standalone AI Gateway MVP ("single-user ready") #758), shared multi-tenant, and full agentic deployment.
• Explicit headroom recommendations for spike absorption.

Failure modes

• Behavior at memory exhaustion: fail-closed vs crash, and recovery, including response-store consistency after an OOM.
• Interaction with the response store, KV/Valkey, and backend inference under sustained pressure.
• Cascading-failure behavior across gateway, store, and backends.

Deliverables

• Automated performance, spike/burst, large-payload, long-conversation, and soak tests over real AI workloads, on a regular (nightly) cadence with memory-regression gating (building on Benchmarks for AI proxy builds #1).
• A reproducible AI load harness and named workload profiles.
• Published measurement results (this issue or a linked doc).
• Operator documentation: an AI Gateway operations/sizing guide, an autoscaling guide (which signals, HPA/KEDA config, thresholds), and an observability/alerting runbook with capacity and cost tables at named scales.

Definition of done (pre-v1)

• Memory, CPU, and spike behavior are measured and published across the reference workload profiles.
• Inference-aware autoscaling signals are defined, exported, and documented with a working HPA/KEDA example.
• A sizing guide exists that an operator can plan an AI Gateway deployment against.
• Automated performance and soak tests run on a cadence and fail the build on regression.

[shaneutt]

Related:

• Operational characterization: memory profile, autoscaling signals, and resource sizing for v1 praxis#1067 (companion Praxis data-plane operational characterization)