benchmarking/locust: harvest server-side telemetry from Prometheus

[Nishanth Kotla (Nishanth29)]

Fixes #1590

Note

Stacked on #1723. Until #1723 merges into main, GitHub displays both commits in the overall diff. To review only the 5 files changed in this PR, click commit bb6d09b9.

What this PR does

Locust measures the client side only. This adds a post-run harvest of server-side ground truth from Prometheus, written to a new server_summary.json and summarized as one row in stats.jsonl.

Proposed Changes

Steady-state scoping

server_telemetry.py derives the steady-state window from stats_history.csv, taking the first sample at 90% of peak user count as the window start, and scopes its queries to that range so ramp-up does not skew the results.

Harvested metrics

Cluster packing, from ate_workerpool_workers: assigned workers over total workers, reported as min, p50, p90, p99, max and mean (avg), plus the underlying timeseries so transient spikes remain visible.

Host kernel pressure, from cAdvisor PSI: CPU, memory and IO stall percentages, each as the same percentile set over the window.

Snapshots: sizes at p50/p90/p95 and mean, checkpoint counts both in-window and cumulative, restore and checkpoint latency at p50/p95 from the AteomHerder RPC histograms, and checkpoint throughput as checkpoint_mb_s.

Constraints and failure behavior

The module uses only the standard library, since the locust image is distroless, and every request carries a timeout. An unreachable Prometheus records nulls and does not fail the run. --prometheus-url overrides the in-cluster default.

Unmeasured fields are null and a measured zero is 0, consistent with the rest of the runner. Prometheus exposes no byte counter on the restore path, so no restore throughput field is emitted rather than deriving one indirectly.

Output

server_summary.json holds the full nested artifact. stats.jsonl receives a single server_summary row with the flat subset intended for graphing: the three packing percentiles, the three PSI p90s, snapshot_size_p50_mb, checkpoints_in_window, restore_p50_s, checkpoint_p50_s and checkpoint_mb_s.

status.json is unchanged.

How this was tested

11 unit tests in test_server_telemetry.py covering steady-state detection, percentile boundaries, the range-query window guard, malformed Prometheus responses, the packing and checkpoint arithmetic, a missing denominator returning null, snapshot fields returning null rather than zero, and telemetry surviving a missing stats CSV.

Verified against live Prometheus on a benchmark cluster. Every emitted value was re-derived by hand from the raw queries and matched. A run with --prometheus-url pointed at an unreachable address completes normally with the affected fields null.

References

Agent Substrate: Actor Density Benchmark Specs

• Tests pass
• Appropriate changes to documentation are included in the PR