Skip to content

feat: add global p95 latency guardrail - #6

Merged
brandonwise merged 1 commit into
mainfrom
auto/pm-presence-20260421
Apr 22, 2026
Merged

feat: add global p95 latency guardrail#6
brandonwise merged 1 commit into
mainfrom
auto/pm-presence-20260421

Conversation

@brandonwise

Copy link
Copy Markdown
Owner

Summary

Adds a global max_p95_latency_ms guardrail so release decisions can fail on tail-latency regressions, not just average latency drift.

Why

Recent telemetry keeps highlighting production agent pain around slow long-tail responses and regression detection. Average latency can look healthy while p95 degrades.

What changed

  • Added global.max_p95_latency_ms to gate spec parsing (GateSpec).
  • Added p95 latency computation and gate enforcement in evaluator:
    • fail when p95 exceeds configured limit
    • fail when p95 limit is configured but latency telemetry is missing
  • Added p95_latency_ms to JSON and markdown report output.
  • Updated README + example spec for new config.
  • Added evaluator tests for:
    • p95 limit catches tail outlier regressions
    • telemetry-required behavior when p95 guardrail is configured

Validation evidence

All checks run from repo root (/Users/bwise/.openclaw/workspace/projects/agent-release-gate):

  • python3 -m pytest → ✅ 19 passed in 0.17s (and rerun after cleanup: 19 passed in 0.04s)
  • python3 -m pytest tests/test_evaluator.py → ✅ 11 passed in 0.03s
  • python3 -m ruff check . → ✅ All checks passed!
  • python3 -m mypy src → ✅ Success: no issues found in 5 source files
  • python3 -m build → ✅ built sdist + wheel successfully
  • Smoke/integration (new behavior):
    • command: PYTHONPATH=src python3 -m agent_release_gate.cli evaluate --spec <tmp>/spec.yaml --results <tmp>/results.json
    • result: ✅ expected gate failure with exit_code=1, gate_passed=False, p95_latency_ms=2500.0, reason P95 latency 2500.0ms exceeds limit 1000ms

Risk

Low. Additive, opt-in config only; existing specs continue to behave as-is unless max_p95_latency_ms is set.

@brandonwise
brandonwise merged commit 4609275 into main Apr 22, 2026
2 checks passed
@brandonwise
brandonwise deleted the auto/pm-presence-20260421 branch April 22, 2026 01:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant