feat(budget): remember validator latency, and warn before a doomed run - #8
Merged
Merged
Conversation
…unch
Lope measured per-validator latency and printed it, then forgot it. Every run
re-derived its ceiling from a flag with no way to know whether that number was
survivable, so an unfittable budget was discovered by spending it: the call was
killed at the wall and all of its work discarded. A caller — increasingly
another agent — picks 240 because 240 looks reasonable, and never learns the
validator's p90 is 178s.
Three changes, no behaviour flipped by default:
- lope/latency.py: a bounded per-validator sample of observed call durations
at $LOPE_HOME/latency.json. Timed-out calls are kept as censored lower
bounds, which biases estimates upward — the safe direction for a budget.
Advisory only, and every entry point swallows its own errors: a corrupt or
unwritable ledger degrades to "no advice", never to a failed run.
- Before launch, the request plan reports two distinct problems that were
previously invisible: a *clamp* (the call ceiling is silently cutting a
provider's own configured budget) and a *misfit* (the enforced ceiling sits
below observed p90 plus variance room, so the call is predicted to be
killed). It prints; it does not block. A first-ever call has no history and
must still be allowed to run.
- --respect-provider-timeout inverts the stricter-wins rule per invocation.
The default is unchanged and deliberate: provider timeouts exist as shorter
safety caps, and `team test --timeout 10` must stay bounded to 10s. The flag
is for the opposite case, a provider configured slow on purpose being
clamped by a ceiling somebody guessed.
The 1.5 safety factor is calibrated against the incident that motivated this:
the same prompt on the same validator took 136s then 178s (+31%), then
exceeded 240s twice under load. A 1.25 factor would have called that 240s
ceiling a good fit, which is exactly the wrong answer.
tests/conftest.py stops the suite writing fixture data ("stub", "a", "b") into
the operator's real ledger, which it did until this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Lope measured per-validator latency and printed it, then forgot it. Every run re-derived its ceiling from a flag with no way to know whether that number was survivable — so an unfittable budget was discovered by spending it, and the killed call's work was discarded.
The motivating incident: a caller passed
--timeout 240for a validator whose configured budget is 600s and whose real p90 is ~178s. It was silently clamped to 240, died at the wall twice, and the retry re-used the same 240 — so it failed identically. Nothing in any output said "240 is too small" or even "we cut your 600 to 240".What
lope/latency.py— bounded per-validator sample of observed durations at$LOPE_HOME/latency.json. Timed-out calls are kept as censored lower bounds, biasing estimates upward (the safe direction). Advisory only; every entry point swallows its own errors, so a corrupt or unwritable ledger degrades to "no advice", never a failed run.--respect-provider-timeoutinverts stricter-wins per invocation. Default behaviour is unchanged and deliberate: provider timeouts are shorter safety caps andteam test --timeout 10must stay bounded.tests/conftest.py— the suite was writing fixture data (stub,a,b) into the operator's real ledger. It no longer does.The 1.5 safety factor is calibrated, not guessed: the same prompt took 136s then 178s (+31%), then exceeded 240s twice. A 1.25 factor would have called that 240s ceiling a good fit.
Verification
p90 13s over 3 call(s) — predicted timeout; raise --timeout to >= 20s), the call it predicted then did time out, and that timeout was stored as a censored sample;--respect-provider-timeoutsuppressed the clamp and the call succeeded.Pre-existing flake noted, unrelated:
test_invocation_limits.py::test_term_exiting_leader_does_not_leave_term_ignoring_grandchildfails intermittently on cleanHEADtoo (6/10 clean vs 4/10 with this branch).🤖 Generated with Claude Code