feat: explain a case in prose, grounded in its own record - #20
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Case Detail shows the arithmetic — every candidate action, its expected value as a
checkable sum, the compliance rules that fired. That is the right artifact for
auditing a decision and the wrong one for asking "so what happened to this payment,
and why did nobody chase it?"
This adds the second thing, above the first.
The rule that makes it safe
The model may not use a number it was not given. Every digit run in the
generated text must appear in the brief it was handed. Grouping is normalised
first, so ₹5,41,724 and 541724 count as the same figure and correct prose is not
rejected for reformatting.
That is a much simpler check than asking whether a claim is true, and it catches
the failure that matters: a narrative confidently reporting "three retry attempts"
on a payment that had one is worse than no narrative at all, on the one screen
whose entire purpose is showing that the numbers add up.
Same reasoning as the copywriter, reached from the other side. There the model
never sees a number; here it may only echo the ones it was shown.
Two narrower checks alongside it — an explanation cannot mention a channel that was
not used, and cannot claim a recovery that did not happen. The recovery check keys
on specific phrases rather than the word "recovered", because "we stopped once no
further attempt was worth what it could recover" is a true sentence about a payment
that was never recovered.
Selection by rule, not by eye
Picking cases by hand would make this a highlight reel: the ones that read well
would get the prose. So the selection is the largest payment of each distinct
shape — recovered without contact, recovered after contact, chased and lost,
never actioned, never diagnosed.
Two of those five are cases where the agent achieved nothing, and they are in the
list on purpose. A test asserts the selection is not all wins.
The fifth shape returns nothing on the agent arm, because the LLM classifier
resolves every cause — so this run explains four cases, not five. That is the
selection working, not failing.
Never a model call on a page load
One batched call covers the whole selection, run once when
demowrites the run.Explaining on demand would be a call per page view — a judge clicking through ten
cases would spend half the daily Gemini allowance on a read-only screen, and two
visits to the same case could disagree with each other.
Cases outside the selection get
summarise(), composed from the same facts with nomodel involved. It is not a stub — it is assembled from the case's own shape, so a
payment that was refused everything reads as a sentence about being refused
everything. The screen always has something to say.
A bug the tests caught
A test runs
summarise()over 40 real cases and puts the output through thevalidator — holding our own prose to the standard we hold the model's. It failed on
pay_002520: the summary said "the agent acted 12 times" and 12 appeared nowhere inthe brief, because the brief listed
ESCALATE_HUMANtwelve times and never statedthe total.
Both halves were wrong. Actions are now tallied (
12 in total — ESCALATE_HUMAN ×12)and identical decision sentences deduplicated. Repeating one decision twelve times
cost tokens, read badly, and invited the model to narrate a sequence of distinct
events that did not happen.
Also fixed: a 35-paise WhatsApp send rendered as
spent on recovery: ₹0, whichwould have had the model reporting that nothing was spent. It now keeps its paise,
which is exactly what that formatting rule exists for.
Not in the ablation
The explainer reads the ledger; it does not participate in it. It cannot change a
decision or an outcome, and
reproduceconfirms all 37 figures including everyper-arm digest are unchanged.
Verification
pytest— 465 passed, 30 of them new.ruff check .clean. Generated live againstthe real model: 4 of 4 accepted, one call, zero rejections. The committed cache
replays with both API keys removed from the environment.