Skip to content

feat: add cross-run pass-rate trend gating - #5

Merged
brandonwise merged 1 commit into
mainfrom
auto/pm-presence-20260410
Apr 11, 2026
Merged

feat: add cross-run pass-rate trend gating#5
brandonwise merged 1 commit into
mainfrom
auto/pm-presence-20260410

Conversation

@brandonwise

Copy link
Copy Markdown
Owner

Summary

Adds a GateHistoryAnalyzer flow to persist per-run gate summaries and detect sustained pass-rate drift over time.

This closes the blind spot where each run can pass point-in-time baseline checks while quality declines gradually across many runs.

Why now

  • AI agent teams are asking for runtime drift/eval-over-time workflows (Reddit + Product Hunt + GitHub signals).
  • Existing --baseline checks only compare one run vs one baseline.
  • This enables CI to fail on directional decline, not just single-step drops.

What changed

  • Added src/agent_release_gate/history.py:
    • record_history() to append run summaries as JSON.
    • analyze_history() to compute OLS slope over recent pass rates.
    • Direction labeling: improving | stable | declining.
    • Machine-readable trend_to_dict() output.
  • Extended CLI:
    • evaluate --record-history <dir> records each run summary.
    • New trend subcommand:
      • --history <dir>
      • --window <n>
      • --output <file>
      • --fail-on-regression (exit 1 on declining trend)
  • Added/expanded tests:
    • tests/test_history.py for record + trend behavior.
    • tests/test_cli.py for parser coverage and CLI behavior (record-history, regression fail exit).
  • Updated README with trend recording/analyzing examples.

Validation

1) Repo-level test suite

  • python3 -m pytest
  • 17 passed in 0.04s

2) Targeted tests for changed modules

  • python3 -m pytest tests/test_history.py tests/test_cli.py
  • 8 passed in 0.02s

3) Lint/type/build

  • python3 -m ruff check .
  • All checks passed!
  • python3 -m mypy src
  • Success: no issues found in 5 source files
  • python3 -m build
  • Successfully built agent_release_gate-0.1.0.tar.gz and agent_release_gate-0.1.0-py3-none-any.whl

4) Smoke/integration checks

  • Command:
    • PYTHONPATH=src python3 -m agent_release_gate.cli trend --history "$tmpdir/history" --fail-on-regression
  • Fixture: three historical runs with pass rates 0.92 -> 0.88 -> 0.84
  • ✅ Exit code 1 (expected)
  • ✅ Output includes:
    • "pass_rate_slope": -0.04
    • "pass_rate_direction": "declining"
    • "any_regression": true

Docs tone check (humanizer)

  • node /Users/bwise/.openclaw/workspace/projects/humanizer/src/cli.js score --file README.md --ignore-code
  • 🟢 0/100

Closes #4.

@brandonwise
brandonwise merged commit 378553e into main Apr 11, 2026
2 checks passed
@brandonwise
brandonwise deleted the auto/pm-presence-20260410 branch April 11, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cli): GateHistoryAnalyzer — cross-run pass_rate trend detection via OLS slope

1 participant