Skip to content

feat(budget): remember validator latency, and warn before a doomed run - #8

Merged
rschumann merged 1 commit into
mainfrom
feat/latency-ledger-budget-fit
Aug 9, 2026
Merged

rschumann merged 1 commit into
mainfrom
feat/latency-ledger-budget-fit

Conversation

@rschumann

Copy link
Copy Markdown
Contributor

Why

Lope measured per-validator latency and printed it, then forgot it. Every run re-derived its ceiling from a flag with no way to know whether that number was survivable — so an unfittable budget was discovered by spending it, and the killed call's work was discarded.

The motivating incident: a caller passed --timeout 240 for a validator whose configured budget is 600s and whose real p90 is ~178s. It was silently clamped to 240, died at the wall twice, and the retry re-used the same 240 — so it failed identically. Nothing in any output said "240 is too small" or even "we cut your 600 to 240".

What

  • lope/latency.py — bounded per-validator sample of observed durations at $LOPE_HOME/latency.json. Timed-out calls are kept as censored lower bounds, biasing estimates upward (the safe direction). Advisory only; every entry point swallows its own errors, so a corrupt or unwritable ledger degrades to "no advice", never a failed run.
  • Pre-launch budget advice on the request plan, reporting two previously invisible problems with different fixes: a clamp (call ceiling silently cutting a provider's configured budget) and a misfit (enforced ceiling below observed p90 + variance room). It prints, it does not block — a first-ever call has no history and must still run.
  • --respect-provider-timeout inverts stricter-wins per invocation. Default behaviour is unchanged and deliberate: provider timeouts are shorter safety caps and team test --timeout 10 must stay bounded.
  • tests/conftest.py — the suite was writing fixture data (stub, a, b) into the operator's real ledger. It no longer does.

The 1.5 safety factor is calibrated, not guessed: the same prompt took 136s then 178s (+31%), then exceeded 240s twice. A 1.25 factor would have called that 240s ceiling a good fit.

Verification

  • 24 new tests; full suite 870 passed.
  • Each new guard mutation-tested individually — safety factor, sample-window bounding, censored-timeout recording, clamp detection, and the escape hatch all confirmed to fail the suite when broken.
  • End-to-end on the real CLI: clamp advice printed; ledger recorded under the real validator name; the misfit warning fired (p90 13s over 3 call(s) — predicted timeout; raise --timeout to >= 20s), the call it predicted then did time out, and that timeout was stored as a censored sample; --respect-provider-timeout suppressed the clamp and the call succeeded.

Pre-existing flake noted, unrelated: test_invocation_limits.py::test_term_exiting_leader_does_not_leave_term_ignoring_grandchild fails intermittently on clean HEAD too (6/10 clean vs 4/10 with this branch).

🤖 Generated with Claude Code

…unch

Lope measured per-validator latency and printed it, then forgot it. Every run
re-derived its ceiling from a flag with no way to know whether that number was
survivable, so an unfittable budget was discovered by spending it: the call was
killed at the wall and all of its work discarded. A caller — increasingly
another agent — picks 240 because 240 looks reasonable, and never learns the
validator's p90 is 178s.

Three changes, no behaviour flipped by default:

- lope/latency.py: a bounded per-validator sample of observed call durations
  at $LOPE_HOME/latency.json. Timed-out calls are kept as censored lower
  bounds, which biases estimates upward — the safe direction for a budget.
  Advisory only, and every entry point swallows its own errors: a corrupt or
  unwritable ledger degrades to "no advice", never to a failed run.

- Before launch, the request plan reports two distinct problems that were
  previously invisible: a *clamp* (the call ceiling is silently cutting a
  provider's own configured budget) and a *misfit* (the enforced ceiling sits
  below observed p90 plus variance room, so the call is predicted to be
  killed). It prints; it does not block. A first-ever call has no history and
  must still be allowed to run.

- --respect-provider-timeout inverts the stricter-wins rule per invocation.
  The default is unchanged and deliberate: provider timeouts exist as shorter
  safety caps, and `team test --timeout 10` must stay bounded to 10s. The flag
  is for the opposite case, a provider configured slow on purpose being
  clamped by a ceiling somebody guessed.

The 1.5 safety factor is calibrated against the incident that motivated this:
the same prompt on the same validator took 136s then 178s (+31%), then
exceeded 240s twice under load. A 1.25 factor would have called that 240s
ceiling a good fit, which is exactly the wrong answer.

tests/conftest.py stops the suite writing fixture data ("stub", "a", "b") into
the operator's real ledger, which it did until this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rschumann
rschumann merged commit 91da447 into main Aug 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant