fix: stop reconsidering a payment already handed to a human - #21
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while reading a case brief for the explainer: one payment listed
ESCALATE_HUMANtwelve times. Pulling the distribution showed 48 escalationsacross four payments — twelve each — against a run cap of fifty. ₹5,280 spent
handing over cases that had already been handed over, and two more repeats on any
one of them would have exhausted the pool for every payment that genuinely
warranted a human.
The root cause
Escalation hands a case to a person. The executor records it, writes
STOPPED,and returns
succeeded=False, because no money came back. The runner's rule was:So the payment came back a minute later. Nothing about it had changed, so the
policy proposed escalation again, the gate allowed it, and the loop ran until the
per-payment decision cap of twelve stopped it.
Executionnow carriesterminalalongsidesucceeded. They answer differentquestions — handing a case to a human did not recover the money and is also not a
failed attempt worth retrying — and collapsing them into one boolean is what
created the loop.
And a rule that should have existed anyway
The compliance gate now separately refuses a second escalation on the same
payment. This is the more important half. "One payment, one human" is a
policy, and a policy that holds only because of the shape of a scheduling loop
somewhere else is not enforced — it is coincidental. The mandate bug already in
FAILURES.md was a compliance rule that looked present and did nothing; this is the
same lesson from the other direction.
That rule does not fire on the committed seed, because with the runner fixed a
second escalation is never proposed. A rule that never fires is exactly the shape
of the bug it guards against, so it is covered by unit tests that construct the
condition directly rather than trusted because the run looks clean.
Why nothing caught it
Every individual piece was correct. The executor did stop touching the payment.
The gate did enforce its run cap — 48 is under 50, so it never fired. The policy
priced each escalation correctly.
actions_by_kindshowedESCALATE_HUMAN: 48,and 48 looks like forty-eight escalated payments, which would be reasonable. The
failure was visible only in the distribution, and nothing summarised that.
It also cost nothing in recovery, because escalation never recovers money in this
simulator — so the totals moved by ₹5,280 of cost and not one rupee of revenue. A
bug that only makes you slightly poorer is much harder to notice than one that
breaks something.
What moved
Incremental recovery, the coverage/judgment split and the LLM ablation are
unchanged to the rupee, because all three are measured on recovery.
The full sweep re-ran and
reports/sensitivity.jsoncame back byte-identical —so this is cost-only across all fifteen points in the parameter space, not just at
the base case. Twenty minutes to prove a negative, which is what the sweep is for.
reproduceflagged exactly the six figures that should have changed — cost anddigest for the three escalating arms — and nothing else. First time it has caught
a real change rather than confirming a null one.
Also
summarise()said "The agent acted 1 times" and then "stopped when no option wasworth its cost" on a case that was in fact escalated. It now names the handover and
says the outcome of that review is outside this record, which is the honest
statement: whether a person resolved it is not something this run may claim credit
for.
Verification
pytest— 469 passed, 4 new.ruff check .clean.recoup reproducegreenagainst regenerated claims. Run, explanations, claims and sweep all regenerated;
FAILURES.md and the README figures updated.