Skip to content

feat: add per-case latency/cost guardrails - #1

Merged
brandonwise merged 1 commit into
mainfrom
auto/pm-presence-20260320
Mar 21, 2026
Merged

feat: add per-case latency/cost guardrails#1
brandonwise merged 1 commit into
mainfrom
auto/pm-presence-20260320

Conversation

@brandonwise

Copy link
Copy Markdown
Owner

Summary

Adds optional per-case max_latency_ms and max_cost_usd guardrails so a single outlier can fail the gate even when global averages still look healthy.

Why this matters

Current gate checks only global average latency/cost. That can hide catastrophic case-level outliers (exactly the kind of failure teams hit in production with agent workflows).

What shipped

  • Added max_latency_ms and max_cost_usd fields to GateCase.
  • Enforced per-case limits during scoring:
    • fails case when metric exceeds limit
    • fails case when metric is missing but a limit is configured
  • Added evaluator tests for:
    • latency/cost outlier failures
    • missing telemetry failures when case limits are set
  • Updated README + example spec with case-level guardrail docs.

Validation

Command Result
python3 -m pytest ✅ PASS (5 passed)
python3 -m pytest tests/test_evaluator.py -k case_limits -q ✅ PASS (2 passed)
python3 -m compileall src tests ✅ PASS
python3 -m build --sdist --wheel ✅ PASS
PYTHONPATH=src python3 -m agent_release_gate.cli evaluate --spec examples/spec.yaml --results examples/results.json --output /tmp/argate-report.json --markdown /tmp/argate-report.md ✅ PASS (gate_passed=true)
node /Users/bwise/.openclaw/workspace/projects/humanizer/src/cli.js score README.md ✅ PASS (13/100, mostly human-sounding)

Risk / rollback

Low risk, additive schema change only. Rollback with:

  • Revert commit f16b25b

@brandonwise
brandonwise merged commit c934fcb into main Mar 21, 2026
2 checks passed
@brandonwise
brandonwise deleted the auto/pm-presence-20260320 branch March 21, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant