Skip to content

feat: add baseline latency/cost regression caps to release gate - #3

Merged
brandonwise merged 1 commit into
mainfrom
auto/pm-presence-20260402
Apr 3, 2026
Merged

feat: add baseline latency/cost regression caps to release gate#3
brandonwise merged 1 commit into
mainfrom
auto/pm-presence-20260402

Conversation

@brandonwise

@brandonwise brandonwise commented Apr 3, 2026

Copy link
Copy Markdown
Owner

Summary

Adds optional baseline drift guardrails so releases fail when average latency or cost regresses too far versus baseline, not just pass rate.

Why

Agent teams keep reporting quality can look stable while latency/cost quietly degrade. This closes that gap and keeps release gates aligned with production economics.

What changed

  • Added max_avg_latency_regression_pct and max_avg_cost_regression_pct global spec options
  • Enforced failures when current run exceeds configured baseline drift thresholds
  • Added explicit failure reasons for missing/invalid baseline telemetry
  • Added evaluator tests for latency drift, cost drift, and telemetry-missing paths
  • Updated README + example spec with new config options

Validation evidence

  • python3 -m pytest10 passed
  • python3 -m pytest tests/test_evaluator.py -k baseline3 passed, 6 deselected
  • python3 -m ruff check .All checks passed
  • python3 -m mypy srcSuccess: no issues found in 4 source files
  • python3 -m buildBuilt sdist + wheel successfully
  • Smoke/integration check (expected fail path):
    • Command: PYTHONPATH=src python3 -m agent_release_gate.cli evaluate --spec <tmp>/spec.yaml --results <tmp>/results.json --baseline <tmp>/baseline.json
    • Result: exit_code=1, gate_passed=false, reason includes Latency regression detected: current 1300.0ms vs baseline 1000.0ms (30.00% increase, allowed 10.00%)

Docs tone check

  • node src/cli.js score -f /Users/bwise/.openclaw/workspace/projects/agent-release-gate/README.md (run from projects/humanizer) → 🟢 0/100

@brandonwise
brandonwise merged commit 48f53ae into main Apr 3, 2026
2 checks passed
@brandonwise
brandonwise deleted the auto/pm-presence-20260402 branch April 3, 2026 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant