Summary
Praxis is a general-purpose L4/L7 proxy, but its architecture makes its resource behavior fundamentally different from a "classic" proxy. Body access and buffering, a composable filter pipeline (compression, decompression, body rewriting, payload processing, AI filters), in-memory caching and counters, embedded Wasm runtimes, and hot-reload (which briefly holds two pipelines at once) all mean memory can be large and, more importantly, can spike hard and unpredictably with the shape of traffic rather than just request count. A proxy sized purely on CPU or connection count will get surprised. Before v1 we need a firm, measured, and documented grasp of these characteristics: how memory and CPU behave across features and traffic profiles, what signals autoscaling must key on, and exactly what resources an operator should provision at a given scale.
This is broader than raw throughput/latency benchmarking (see #453). It is about characterizing the full operational envelope, proving it with automated tests, exposing the right signals, and shipping documentation an operator can plan against. #453 can serve as the raw performance-benchmark input to this work.
Goals
- Measure and document Praxis's memory and CPU behavior across representative feature sets and traffic profiles, with particular emphasis on worst-case and spike behavior.
- Define the signals operators should autoscale on (CPU alone is insufficient for a memory-spike-prone proxy) and how to export them.
- Publish resource-sizing guidance at specific, named scales.
- Back all of the above with automated performance, soak, and spike tests that run on a regular cadence and gate regressions.
- Ship operator-facing documentation prior to v1.
Operational considerations to cover
Memory behavior
- Baseline (idle) footprint per worker/process and how it scales with worker count.
- Per-connection and per-request memory cost across HTTP/1.1, HTTP/2, and HTTP/3.
- Body-buffering memory: request/response body access, buffering thresholds, and configured max buffered body sizes.
- Memory amplification from payload-processing and transformation filters (compression, decompression, body rewrite, JSON transforms) and how it compounds when chained.
- Streaming vs buffering paths (SSE codec, streaming compression) and their contrasting memory profiles.
- Cache storage backend memory (in-memory response cache), growth and eviction behavior under pressure.
- Distributed in-memory counters and KV-store growth over time.
- Worst-case and tail spikes: large bodies, many concurrent large requests, slow clients (slowloris-style), and backpressure buildup.
- Memory under connection-pool growth, keepalive, and H2 stream multiplexing.
- Wasm filter memory (fuel and memory limits), per-instance overhead, and pool sizing.
- Allocator behavior: fragmentation, RSS not returning to the OS, and whether an allocator choice/tuning (jemalloc, mimalloc) is warranted.
- Hot-reload memory behavior: old and new pipelines coexisting during the atomic swap (Arc), and the transient footprint doubling that implies.
- Long-running leak detection via soak tests.
CPU and throughput
- CPU cost per filter and per request phase.
- Throughput and latency across concurrency levels and payload-size distributions.
- Cost of TLS termination, HTTP/2, and HTTP/3 on CPU and memory.
Scaling and autoscaling
- Which signals to autoscale on: memory pressure, in-flight requests, buffered bytes, queue depth, and overload-manager/backpressure state (not just CPU).
- Custom/derived metrics for Kubernetes HPA and KEDA, and how Praxis exposes them.
- Behavior under overload (overload manager, load shedding, 503s, backpressure) and how it should interact with autoscaling.
- Scale-up latency vs spike speed: can horizontal autoscaling react fast enough to memory spikes, and what headroom is required to absorb them in the meantime?
- Graceful degradation and admission control at capacity.
- Connection draining during scale-down and endpoint removal.
- Vertical vs horizontal guidance, including when buffered-body memory forces vertical scaling.
- Container memory requests vs limits guidance, cgroup awareness, and OOMKill avoidance.
Observability
- Metrics required to see memory behavior in real time: RSS/heap, buffered bytes in flight, per-filter allocation, active-requests and active-connections gauges.
- Metric cardinality budget (label sets) so observability does not itself become a memory problem.
- Tracing hooks for slow or large requests.
- Alerting thresholds and SLOs for memory saturation and overload.
- Reference dashboards for capacity planning.
Resource sizing and capacity planning
- Documented resource requirements at specific, named scales (for example: at X RPS, Y concurrent connections, and a given body-size distribution, provision N cores and M GiB RAM).
- Sizing rules of thumb per feature (the marginal cost of enabling body buffering, caching, compression, or payload processing).
- Reference profiles: minimal L4/L7 proxy, full-feature, and AI-gateway data plane.
- Explicit headroom recommendations for spike absorption.
Failure modes
- Behavior at memory exhaustion: fail-closed vs crash, and OOM recovery.
- Interaction with Pingora's H2 backpressure and connection-pool safety.
- Cascading-failure behavior under sustained memory pressure.
Deliverables
- Automated performance, spike/burst, large-payload, slow-client, and long-running soak tests, wired into a regular (nightly) cadence with memory-regression gating alongside the existing perf-regression checks.
- A reproducible load harness and named load profiles.
- Published measurement results (this issue or a linked doc).
- Operator documentation: an operations/sizing guide, an autoscaling guide (which signals, HPA/KEDA config, thresholds), and an observability/alerting runbook with capacity tables at named scales.
Definition of done (pre-v1)
- Memory, CPU, and spike behavior are measured and published across the reference profiles.
- Autoscaling signals are defined, exported, and documented with a working HPA/KEDA example.
- A sizing guide exists that an operator can plan a deployment against.
- Automated performance and soak tests run on a cadence and fail the build on regression.
Summary
Praxis is a general-purpose L4/L7 proxy, but its architecture makes its resource behavior fundamentally different from a "classic" proxy. Body access and buffering, a composable filter pipeline (compression, decompression, body rewriting, payload processing, AI filters), in-memory caching and counters, embedded Wasm runtimes, and hot-reload (which briefly holds two pipelines at once) all mean memory can be large and, more importantly, can spike hard and unpredictably with the shape of traffic rather than just request count. A proxy sized purely on CPU or connection count will get surprised. Before v1 we need a firm, measured, and documented grasp of these characteristics: how memory and CPU behave across features and traffic profiles, what signals autoscaling must key on, and exactly what resources an operator should provision at a given scale.
This is broader than raw throughput/latency benchmarking (see #453). It is about characterizing the full operational envelope, proving it with automated tests, exposing the right signals, and shipping documentation an operator can plan against. #453 can serve as the raw performance-benchmark input to this work.
Goals
Operational considerations to cover
Memory behavior
CPU and throughput
Scaling and autoscaling
Observability
Resource sizing and capacity planning
Failure modes
Deliverables
Definition of done (pre-v1)