docs: lead with the measured result - #17
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README opened with "Status: in development. Day 1 of 16. Headline results are
not in yet." That has been false for a while, and it was the first thing anyone
cloning the repo read.
It now opens with the number, and with the number's own caveats.
README
Leads with ₹5,41,724 recovered on 1,605 failures, ₹3,94,791 of it incremental
over naive retry, at ₹8,336 across 926 contacts, with 903 actions refused on
compliance grounds. Seed
20260902, reproducible withmake evaland no API key.Three things it deliberately does not hide:
The lift decomposition.
contact_onlyexists to split the result intocoverage (67.7%) and judgment (32.3%). Two thirds of the lift is available to
anyone who notices that 76.3% of failures cannot be retried and have to be asked
about. A single headline number would have quietly claimed all of it for the
policy engine.
The LLM ablation: −₹238. Reported as measured. The model classifies every
unresolved symptom correctly and is still not worth much here, because 46 of the
51 failures it resolves are outages where the right answer is "do not chase" —
close to what the pessimistic default already produced.
What this is not. One error archetype is verified against live test mode;
twenty are assumptions, and the first one checked was half wrong. The simulator
cannot model whether better copy recovers more money, so the copywriter is
excluded from the ablation rather than credited with a lift it has no evidence
for.
Also corrects the quickstart, which advertised
make reproduce— still a stub.It now lists the commands that actually run.
DECISIONS.md
Ten decisions, each against the alternative a reasonable person would have picked:
EV over a learned policy, compliance as a veto rather than a term in the sum,
append-only enforced by database triggers rather than by convention, common random
numbers across arms, two holdouts instead of one.
Then a second half on what was deliberately not built, which is the more useful
part:
cache/llm/is a better trace than a trace. Exact prompt, exactresponse, content-addressed, committed, readable offline with no account. A
hosted dashboard behind a login is worse evidence than a file the reader already
has.
price the options, check the rules, act. One branch point, and it is an
argmaxover arithmetic. Wrapping that in a message-passing framework would add a
scheduler and a failure surface on top of a function call, and make the system
less explainable. Calling this multi-agent would be dressing.
parse would be text we wrote. That measures the model's ability to read our own
strings.
SWITCH_ROUTE— Razorpay is the PSP here, and comparative acquirer successrates are exactly the data nobody publishes. The simulator's route model would be
invention, and so would any lift from it.
Cleanup
.env.examplelisted Langfuse and Resend keys for integrations that never shipped.It now lists only variables something actually reads, and states up front that none
of them are needed to reproduce the numbers — the world comes from a seed and every
model response is committed.
duckdbwas a declared dependency and is imported nowhere. Dropped.Verification
pytest— 382 passed.ruff check .clean. Every figure in the README traced backto
data/run.jsonandreports/sensitivity.jsonrather than retyped from memory.