Skip to content

docs: lead with the measured result - #17

Merged
iamsiddhesh-dev merged 3 commits into
mainfrom
docs-results
Sep 3, 2026
Merged

iamsiddhesh-dev merged 3 commits into
mainfrom
docs-results

Conversation

@iamsiddhesh-dev

Copy link
Copy Markdown
Owner

The README opened with "Status: in development. Day 1 of 16. Headline results are
not in yet." That has been false for a while, and it was the first thing anyone
cloning the repo read.

It now opens with the number, and with the number's own caveats.

README

Leads with ₹5,41,724 recovered on 1,605 failures, ₹3,94,791 of it incremental
over naive retry, at ₹8,336 across 926 contacts, with 903 actions refused on
compliance grounds. Seed 20260902, reproducible with make eval and no API key.

Three things it deliberately does not hide:

The lift decomposition. contact_only exists to split the result into
coverage (67.7%) and judgment (32.3%). Two thirds of the lift is available to
anyone who notices that 76.3% of failures cannot be retried and have to be asked
about. A single headline number would have quietly claimed all of it for the
policy engine.

The LLM ablation: −₹238. Reported as measured. The model classifies every
unresolved symptom correctly and is still not worth much here, because 46 of the
51 failures it resolves are outages where the right answer is "do not chase" —
close to what the pessimistic default already produced.

What this is not. One error archetype is verified against live test mode;
twenty are assumptions, and the first one checked was half wrong. The simulator
cannot model whether better copy recovers more money, so the copywriter is
excluded from the ablation rather than credited with a lift it has no evidence
for.

Also corrects the quickstart, which advertised make reproduce — still a stub.
It now lists the commands that actually run.

DECISIONS.md

Ten decisions, each against the alternative a reasonable person would have picked:
EV over a learned policy, compliance as a veto rather than a term in the sum,
append-only enforced by database triggers rather than by convention, common random
numbers across arms, two holdouts instead of one.

Then a second half on what was deliberately not built, which is the more useful
part:

  • Langfusecache/llm/ is a better trace than a trace. Exact prompt, exact
    response, content-addressed, committed, readable offline with no account. A
    hosted dashboard behind a login is worse evidence than a file the reader already
    has.
  • LangGraph / supervisor-worker — the work is a decision per payment: classify,
    price the options, check the rules, act. One branch point, and it is an argmax
    over arithmetic. Wrapping that in a message-passing framework would add a
    scheduler and a failure surface on top of a function call, and make the system
    less explainable. Calling this multi-agent would be dressing.
  • Promise-to-pay parsing — there is no inbound channel, so the only text to
    parse would be text we wrote. That measures the model's ability to read our own
    strings.
  • SWITCH_ROUTE — Razorpay is the PSP here, and comparative acquirer success
    rates are exactly the data nobody publishes. The simulator's route model would be
    invention, and so would any lift from it.
  • Resend, WhatsApp Business API, Sarvam voice, learned retry timing, live mode.

Cleanup

.env.example listed Langfuse and Resend keys for integrations that never shipped.
It now lists only variables something actually reads, and states up front that none
of them are needed to reproduce the numbers — the world comes from a seed and every
model response is committed.

duckdb was a declared dependency and is imported nowhere. Dropped.

Verification

pytest — 382 passed. ruff check . clean. Every figure in the README traced back
to data/run.json and reports/sensitivity.json rather than retyped from memory.

@iamsiddhesh-dev
iamsiddhesh-dev merged commit d41cb3a into main Sep 3, 2026
1 check passed
@iamsiddhesh-dev
iamsiddhesh-dev deleted the docs-results branch September 3, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant