feat: add OTel observability benchmark demo - #13
Conversation
|
Unsigned commits: 74990af. Please sign your commits. |
|
PR too large: 1500 lines added (limit: 750, excludes Cargo files, tests, docs, examples, and benchmarks). Please split into smaller PRs. Add |
74990af to
476fc27
Compare
praxis-bot
left a comment
praxis-bot
left a comment
There was a problem hiding this comment.
4 findings (1 Large, 3 Medium)
Overall: well-structured demo with good reproducibility design -- pinned chart versions, configurable benchmark parameters, multiple runs with warmup. The OTel collector pipeline, Tempo integration, and benchmark methodology are sound. The three-way comparison (baseline / noop / full) is the right approach for isolating overhead. A few issues to address.
| f.write(f"**Platform:** KIND (1 node) | **Backend:** Fortio echo\n\n") | ||
| f.write(f"## Summary\n\n") | ||
| f.write(f"| Config | P50 (us) | P99 (us) | RPS | Runs | P50 delta | P99 delta |\n") | ||
| f.write(f"|--------|----------|----------|-----|------|-----------|-----------||\n") |
There was a problem hiding this comment.
[Medium] Trailing || on separator lines creates a ghost 8th column, breaking Markdown table rendering. This line has 7 header columns but 8 separator segments. The same issue appears on line 96 (the "Previous Results" table).
Fix both separator lines by removing the trailing extra |:
f.write(f"|--------|----------|----------|-----|------|-----------|-----------|\n")and line 96:
f.write(f"|--------|----------|----------|-----|-----------|-----------|\n")|
|
||
| CLUSTER_NAME="${CLUSTER_NAME:-otel-bench-local}" | ||
| CTX="kind-${CLUSTER_NAME}" | ||
| GATEWAY_URL="http://localhost:18080" |
There was a problem hiding this comment.
[Medium] Neither benchmark.sh nor benchmark-ai.sh validate that vegeta is installed before starting. With set -euo pipefail, the first vegeta command will fail, but only after the script has already modified the Kubernetes deployment (created configmap, changed image, scaled pods). This leaves the cluster in a partially modified state.
Add a prerequisite check near the top of both scripts:
for cmd in vegeta kubectl; do
command -v "$cmd" >/dev/null 2>&1 || { echo "Error: $cmd not found"; exit 1; }
done
76b9126 to
6988b33
Compare
6988b33 to
b232cf7
Compare
praxis-bot
left a comment
praxis-bot
left a comment
There was a problem hiding this comment.
PR Review (re-review)
Summary: Previous findings addressed -- ARG FEATURES replaces the fragile sed approach, vegeta prerequisite check added, report table formatting fixed. One new issue found.
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 0 |
| Medium | 1 |
| RESULTS_PREFIX="" | ||
| TARGET_FN=core_target | ||
| SCENARIO_TITLE="" | ||
| REPORT_SCRIPT="report.sh" |
There was a problem hiding this comment.
[Medium] REPORT_SCRIPT is set to "report-ai.sh" for the AI scenario, but that file does not exist -- the only report script is report.sh (which auto-detects the scenario via the results directory contents). After running bash scripts/benchmark.sh --scenario ai, line 154 prints an instruction to run a nonexistent script.
Change to:
REPORT_SCRIPT="report.sh"
b232cf7 to
cf132ab
Compare
Full observability stack on KIND for benchmarking Praxis experimental server OTel tracing overhead across two scenarios: - AI gateway (500 RPS, 22 spans/request, POST /v1/chat/completions) - Core proxy (2000 RPS, 10 spans/request, GET to echo backend) AI scenario uses extended filter chain: request_id, access_log, model_to_header, token_usage_headers, token_count, time_to_first_token, intelligent_route, and load_balancer — demonstrating the full AI gateway pipeline in the trace waterfall. Each scenario runs baseline/otel-noop/otel-full configurations with vegeta load testing via a single parameterized benchmark.sh script. Stack: Prometheus, Grafana 11.x, Tempo, Loki, OTel Collector, MLflow (file-backed SQLite), Fortio echo, llm-d inference-sim, plus 5 Grafana dashboards. Also adds: - otel feature to praxis-experimental-server (praxis-core/otel + praxis-filter/otel + praxis-ai-proxy/opentelemetry) - FEATURES build-arg in Containerfile - Patches praxis crates to main rev 1b439271 for Tokio runtime fix and filter/otel feature not yet in v0.5.3 Signed-off-by: Ladislav Smola <lsmola@redhat.com>
cf132ab to
746594d
Compare
3 participants
Summary
experimental AI gateway's OTel tracing overhead
with intelligent_route, token_count, time_to_first_token,
model_to_header)
otelfeature to praxis-experimental-server(praxis-core/otel + praxis-filter/otel + praxis-ai-proxy/opentelemetry)
FEATURESbuild-arg in Containerfile for opt-in OTel buildsDependencies
extraPortMappingssupportfor exposing KIND NodePorts to the host)
Demo structure
Test plan
praxis-forge doctorpassespraxis-forge up --config forge.yamlcreates KIND clusterskip/pr-conventionslabel needed (1436 lines — demo withJSON dashboards and configs, not splittable)
Known limitations
[patch.crates-io]pins praxis crates to main rev1b439271forthe Tokio runtime fix and filter/otel feature not yet in v0.5.3.
Remove once praxis publishes v0.5.4+.