Orders ↔ PG ledger ↔ Bank statement, tied out to the paisa. Automatically. Three-way payment settlement reconciliation for an Indian online merchant, plus a forward cash position.
A personal project: run the books and the cash position, automatically.
The problem. In payment reconciliation, one bank credit is rarely one payment. It's usually a batch: the combined settlement of several transactions, minus refunds that were issued, minus chargebacks that came in, plus or minus whatever adjustments the gateway applied along the way. The UTR or reference number that's supposed to tie that credit back to the original orders is often mangled by the bank's own systems, truncated to a handful of characters, or missing from the narration entirely. Today, a finance team works through this by hand in a spreadsheet, every settlement cycle, trying to reconstruct which payments actually landed and which didn't.
The solution. Kosh automates that reconstruction end to end, tying orders, the payment gateway ledger, and the bank statement out to the paisa in 40 milliseconds for 10,000 records. But speed isn't the part that matters most. Kosh is built to refuse rather than guess: when the evidence for a match is genuinely ambiguous, it declines to assert a link and raises an honest exception for a human to review, instead of risking a wrong match that would quietly corrupt the books.
make demo output, unmodified. Every panel shown on this page is a crop of the same committed report.
| Auto-match rate | 97.83% @ 10,000 records · 97.74% @ 2,000 · 97.61% @ 500 |
| False-match rate | 0.00%, every scale, 11 seeds, 7 adversarial attacks, and a live LLM run |
| Link scoring | 19,610 of 19,610 links correct at 10k, precision and recall both 100.00% |
| Money found | ₹76,227.57 in gateway fee / tax / FX overcharges, across 219 records |
| Throughput | ~146k records/sec sustained, verified linear out to 93,030 records |
| LLM cost | $0.056 per 1,000 records; only 0.31% of records ever reach a model |
| Verification | 257 tests · 88% coverage · reproduced byte-for-byte from a clean clone |
The bar this project holds itself to: throughput, plus measured accuracy, plus an honest exception list. One cherry-picked match proves nothing.
Every number on this page is measured against a machine-readable
ground_truth.jsonwith injected, labelled defects, never asserted, never hand-picked. Kosh is frozen at 500 / 2,000 / 10,000 records and validated out to 93,030.
Contents: Quick Start · Architecture · Results · Why you can trust these numbers · Why we generate our own data · Live API + dashboard · Limitations
git clone https://github.com/bodapatisaikrishna/kosh && cd kosh
pip install -e .
make demoOpens benchmarks/run_demo.html: the full 4-panel dashboard (headline strip, layer waterfall, exception queue with evidence-chain and agent-trace drill-down, cash position) from a fresh 2,000-record run. Verified in an isolated clone on a clean venv, not assumed.
Want to look before running anything? benchmarks/freeze_10000.html and freeze_2000.html are the same dashboard, committed: identical engine, fixture, and numbers. Open either straight from the repo; the screenshots on this page are unmodified crops of them. (run_demo.html is regenerated output and deliberately not committed, same as data/fixtures/: everything reproducible from a seed stays out of git.)
make freeze # regenerate all 3 scales + phase benchmarks
make multiseed # the 6-seed sweep
make adversarial # the 7 attacks
make verify-deterministic
make demo-cash # cash forecast from an operator-chosen viewpoint
pytest # 257 testsDeeper dives, once the extras are installed (pip install -e ".[dev]", add .[llm] for SDK-backed tests, .[api] for the API):
python -m data.generator.trace --fixtures data/fixtures/run_2000 --pick-clean # hand-verify one full chain
python -m engine.l2_subset --profile --trials 2000 --seed 42 # L2 solver timing
export NIM_API_KEY=...
python -m engine.l3_agent --profile --backend nim --model nvidia/nemotron-3-ultra-550b-a55bmake gen, sample, test, trace, eval-null, eval-oracle, eval-l0l1, eval-l0l1l2, eval-full, l2-profile, l3-profile, demo, demo-cash, multiseed, adversarial, verify-deterministic, freeze, api, dashboard wrap the same commands.
Deterministic first, LLM last: five layers, each seeing only what the one above couldn't resolve (shares from run_2000):
Every box names the file that implements it. An interactive, clickable version with guided views is at docs/kosh-architecture.html. GitHub shows raw source for .html files by default. This link renders it as a page instead.
L0 Deterministic joins (exact keys) → 99.33% of matched links
L1 Tolerance matching (±amount, ±date) → 0.26%
L2 Combinatorial solver (subset-sum) → 0.41%
L3 LLM agent (residual only) → 6 records (0.32%), 29 (0.31%) at 10k
L4 Exception ledger (honest remainder) → 139 exceptions, every category covered
Each layer refuses rather than guesses when evidence is ambiguous: that's what holds false-match at zero. L0 won't pick between two settlements sharing a UTR prefix. L1 won't pick the "closest" of two candidates in tolerance. L2 returns AMBIGUOUS rather than choosing one of several valid subsets. L3's tool layer structurally rejects any ID it didn't hand the model, any match under 0.85 confidence, and recomputes severity itself rather than trusting the model. The prompt asks, the tool layer enforces.
The generator is deliberately adversarial to its own engine: it injects consolidated payouts (one credit, several settlements, no per-settlement reference, solvable only by subset-sum) and compound fee+tax errors (two overlapping causes, so no single-cause hypothesis can decompose them: genuinely unexplained, real work for L3). That's what makes the layer shares above real measurements rather than a diagram.
Stack: Python 3.11+ and zero runtime dependencies, plain dataclasses, stdlib csv/json throughout (deliberately, so no float formatting ever gets near money), integer paise everywhere (enforced by an AST lint). pytest is a dev extra; the anthropic/openai SDKs are an optional extra behind a provider-agnostic LLMClient interface, needed only for a live L3 run. pip install -e . pulls in nothing at all.
data/generator/ synthetic dataset generator + injected, labelled defects
engine/ L0-L4: matching layers, the LLM adapter, the exception ledger
eval/ scoring against ground truth, the 4-panel HTML dashboard
cash/ forward cash position: SLA forecast, stuck cash, book-vs-reconciled
tests/ 257 tests, incl. adversarial suite and frozen regression baselines
benchmarks/ committed reports at every phase + the 3-scale freeze + real agent traces
api/ post-freeze stretch goal: FastAPI layer over eval.report.run_eval
dashboard/ post-freeze stretch goal: Next.js + Recharts interactive dashboard
docs/ report screenshots + the architecture diagrams used on this page
Full design rationale, and every bug with the reasoning that caught it, in ARCHITECTURE.md.
Complete pipeline, three scales, all committed (benchmarks/freeze_*.json):
| Records | Auto-match | Precision / Recall | False-match | Exceptions | Wall clock |
|---|---|---|---|---|---|
| 500 | 97.61% | 100.00% / 99.91% | 0.00% | 48 | ~3 ms |
| 2,000 | 97.74% | 100.00% / 99.95% | 0.00% | 139 | ~8 ms |
| 10,000 | 97.83% | 100.00% / 100.00% | 0.00% | 550 | ~40 ms |
False-match rate is the headline metric, not auto-match rate. In finance a wrong match is worse than no match: it silently corrupts the books, where an unmatched item merely sits in a queue for review. It reads 0.00% at every scale tested, including a real, non-scripted LLM run, checked directly against ground truth, not asserted.
At 10,000 records the link scoring is a clean sweep: 19,610 links asserted, 19,610 correct, 19,610 actually true. Nothing wrong, nothing missing. That's a raw count, not a rounded rate.
The generator injects 14 labelled defect types. Nine must be caught; five must be resolved silently: those five exist specifically to punish an engine that flags everything it doesn't instantly recognise.
| Fixture | Injected defects | Correct | Wrong |
|---|---|---|---|
run_500 |
78 | 78 (100%) | - |
run_2000 |
203 | 203 (100%) | - |
run_10000 |
722 | 717 (99.31%) | 4 mislabelled, 1 missed |
At 500 and 2,000 records every injected defect lands in exactly the right bucket: zero misses, zero false alarms. At 10,000 it does not: three fx_variance and one gst_variance are mislabelled into a neighbouring fee category, and one boundary-case period_cutoff is missed outright. Stated here rather than left to be discovered, because the two smaller fixtures alone would imply a perfection that 722 defects doesn't support.
Full breakdown on run_2000, where every class is exact:
| Correctly flagged as exceptions | Correctly resolved silently | ||
|---|---|---|---|
missing_settlement |
20/20 | rounding_drift |
25/25 |
duplicate_payment |
20/20 | utr_mangled |
20/20 |
fee_mismatch_wrong_tier |
19/19 | settlement_split |
11/11 |
gst_variance |
15/15 | consolidated_payout |
8/8 |
period_cutoff |
14/14 | ||
orphan_chargeback |
14/14 | ||
unidentified_credit |
14/14 | ||
refund_misallocation |
9/9 | ||
fx_variance |
8/8 | ||
compound_fee_tax_error |
3–6/6* |
* compound_fee_tax_error is L3's live residual, not deterministic, so it varies run to run. The best evidence is the larger sample: against run_10000's 29-record residual, L3 scored 25/29 (86%); see L3 at full scale. The four run_2000 runs (6, 4, 5, 3 out of 6) are the same behaviour on a sample too small to draw a rate from.
₹76,227.57 in fee leakage at 10,000 records (₹10,475.40 at 2,000): the industry-standard reconciliation metric, and the number a finance team actually cares about. That's what the merchant was overcharged in gateway fees, tax on those fees, and FX. Only FEE_VARIANCE / TAX_VARIANCE / FX_VARIANCE count; timing, duplication, and attribution problems are deliberately excluded, since folding them in would inflate the number into meaninglessness.
Always reported as a lower bound: a compound error coerced to UNEXPLAINED_VARIANCE contains real leakage this number cannot isolate to a single fee leg.
And the cash position, decomposed to the paisa. Book cash ₹6,94,27,826.66; reconciled cash (money actually evidenced end to end) ₹6,52,77,273.37. The ₹41,50,553.29 gap is not a rounding difference: it's named component by component (settled-but-not-credited, refunds, chargebacks, unidentified credits). ₹12,44,126.19 is stuck across 198 named payment IDs: captured, past SLA, not yet in the bank. Those are IDs a human can chase this afternoon, not an aggregate.
550 exceptions, ₹49,33,264.88 at risk, each with a category, an owner, an age, and a click-through evidence chain. This is the queue a controller works on Monday morning, not a number quietly absorbed into a match rate.
Exception aging against the industry 48-hour SLA: median 45 days, max 89, 525 of 550 breaching (at 2,000 records: 35 / 89 / 129 of 139). This is not a live queue. The fixture is a fixed, historical 3-month dataset scored against its own end date, so aging this large is the expected result of scoring a static snapshot, not a finding about operational neglect. Stated rather than quietly omitted, because the number looks alarming and isn't.
Measured, not asserted: share of correctly-matched links contributed by each layer:
| Layer | 500 | 2,000 | 10,000 | What only it can do |
|---|---|---|---|---|
| L0 exact-key | 97.76% | 99.33% | 99.82% | UTR / FK joins |
| L1 tolerance | 0.78% | 0.26% | 0.06% | UTR rekeyed with a transposed digit |
| L2 subset-sum | 1.47% | 0.41% | 0.12% | consolidated payouts, one credit, 2-4 settlements, no per-settlement UTR |
| L3 agent | 1 record | 6 records | 29 records | variances no deterministic rule can decompose |
L3 saw 6 of 1,858 records (0.32%) on run_2000 and 29 of 9,317 (0.31%) on run_10000. That ratio holding flat across a 5× scale increase is the deterministic-first thesis quantified: the residual grows linearly, not explosively, so LLM cost stays a rounding error at any scale. The other 99.7% of records cost zero tokens.
The freeze stops at 9,317 records, so the pipeline was run out to 93,030 to check the claim actually holds. False-match stayed 0.00% at every scale:
| Scored records | Wall clock | rec/s | Auto-match | False-match |
|---|---|---|---|---|
| 9,317 | 40 ms | 231,191 | 97.83% | 0.00% |
| 23,228 | 111 ms | 209,943 | 97.83% | 0.00% |
| 46,506 | 318 ms | 146,364 | 97.82% | 0.00% |
| 93,030 | 638 ms | 145,773 | 97.82% | 0.00% |
The final doubling has a scaling exponent of 1.006, linear to three decimal places, and the rate plateaus around 146k records/sec. The higher rate at small N is amortisation, not a faster path: 146k/sec is the honest sustained figure, not the 231k a 9,317-record run reports. L2's subset-sum is the only superlinear component, and it stays bounded by design (≤40 candidates, 250 ms deadline), which is why the curve flattens rather than exploding. Source: scaling_100k.json.
Seed-robustness and scale, tested together. The 6-seed sweep is at 2,000 records and the original 10,000 result was a single seed, so "0.00% at scale" rested on one seed. Five seeds at 10,000 records each (one of them the original seed=42): 97.83–97.84% auto-match, 100% recall on every seed, 0.00% false-match on every seed, a tighter spread than at 2,000, as a larger sample should be. Source: multiseed_10k.json.
| Engine mode | Fixture | Auto-match | False-match | Precision / Recall | Cost |
|---|---|---|---|---|---|
| Null (matches nothing) | run_2000 |
0.00% | 0.00% | 0.00% / 0.00% | $0 |
| L0 + L1 | run_2000 |
92.84% | 0.00% | 100.00% / 99.54% | $0 |
| L0 + L1 + L2 | run_2000 |
97.74% | 0.00% | 100.00% / 99.95% | $0 |
| Full (+ L3 + L4) | run_2000 |
97.74% | 0.00% | 100.00% / 99.95% | $0 (deterministic CLI path) |
| All-LLM (L3 only, L0-L2 and L4 bypassed) | sample_200* |
81.52% | 0.00% | 100.00% / 77.36% | $6.58 → $35.76 / 1000 records |
* 348 non-order records (payments + settlements + bank), not the 1,858-record run_2000; routing everything through a live LLM at 2,000-record scale costs materially more for the same architectural point. Real run: nvidia/nemotron-3-ultra-550b-a55b via NVIDIA NIM, 1,207 LLM calls, 348/348 records, zero crashes. Source: ablation_llm_only.json.
The all-LLM row is the actual evidence for the architecture, not a knock against the model. False-match holds at 0.00% even with every deterministic layer disabled. The "refuse rather than guess" discipline survives. What drops is recall: more genuinely ambiguous records get correctly refused rather than confidently matched, at 637× the cost ($35.76 per 1,000 records here against the $0.056 per 1,000 the deterministic-first pipeline actually spends, both above, both real) for the 99.68% of records that never needed judgment in the first place. Deterministic-first isn't a shortcut around the LLM; it's what reserves LLM judgment for the cases that actually need it.
6 records that L0–L2 and L4's deterministic classifier genuinely could not resolve, sent live to nvidia/nemotron-3-ultra-550b-a55b via NVIDIA NIM (phase5_live_residual.json). Anthropic's Claude was the original target model; no key was available, so NIM's Nemotron ran instead: a documented deviation, not a silent one.
| Record | Outcome |
|---|---|
pay_dGxUjmPIxeeXo4 |
Correctly matched to its settlement, independently checked against ground_truth.json. Its 88-paise fee/GST anomaly judged immaterial, not separately flagged. |
pay_OyvjU0Hc7g7Bi2 |
Correctly raised UNEXPLAINED_VARIANCE, ₹2,286.93, exact, the largest of the six |
pay_RMejvzSwrh9QXa |
Correctly raised UNEXPLAINED_VARIANCE, ₹1.83, exact to the true net delta |
pay_Yw6hEZsEyvZMNn |
Correctly raised UNEXPLAINED_VARIANCE, ₹1.42, exact |
pay_ymzQx3u8WEhd7G |
Correctly raised UNEXPLAINED_VARIANCE, ₹41.19, exact |
pay_3egKQ6BCralBAI |
Raised TAX_VARIANCE, ₹0.90, exact amount, single-cause label: the model checked only the GST leg this run, so the multi-leg coercion correctly did not fire on one data point |
Zero AGENT_INCOMPLETE, zero false matches, zero invented categories, verified by hand against ground truth for every asserted link, category, and amount, not read off the summary. 51 real LLM calls, 487 s wall clock, $0.104 total → $0.056 per 1,000 records (well under the self-imposed <$0.50/1000 target), computed from real token counts against NIM's published rate, not estimated. Full traces: sample_traces_live/.
The structural fix behind it: L3's tool layer recomputes the category to UNEXPLAINED_VARIANCE, and the amount to the true net delta (not one leg's), whenever the model's own tool-call history shows 2+ comparisons it couldn't decompose. The prompt asks; the tool layer enforces. It fires when the evidence supports it and correctly refuses on weaker evidence, rather than papering over real run-to-run variance by guessing.
The run_2000 residual is only 6 records, too few to claim a rate from. So the agent was also run live against run_10000's 29-record residual: 245 LLM calls, 20 minutes. Source: phase5_live_residual_10k.json.
Headline metrics did not move. Auto-match 97.83%, false-match 0.00%, precision 100%, recall 100%, 550 exceptions, byte-identical to the deterministic freeze_10000 run. L3 touches 29 of 9,317 records; it cannot and did not shift the top-line numbers.
On its own defect class, L3 scored 25/29 (86%), a real rate from a real sample, replacing four noisy readings off six records. The 4 it got wrong, verified by hand against ground_truth.json:
| Record | Outcome | Amount |
|---|---|---|
pay_4cXgekcH1NC0sO |
labelled TAX_VARIANCE, not UNEXPLAINED_VARIANCE |
₹1.30, exact |
pay_l23vdlUo60FAXP |
labelled FEE_VARIANCE |
₹205.79, exact |
pay_z8mbHMXa8wIMih |
labelled TAX_VARIANCE |
₹0.33, exact |
pay_wkjO7N4t4iTfU1 |
AGENT_INCOMPLETE, exhausted its 12-turn budget |
fallback hint, not a computed claim |
Three of the four are label-only misses with the money exactly right: the model found the real variance, then named one cause instead of "multiple causes, undecomposable," because the multi-leg coercion correctly declined to fire on insufficient evidence.
Two findings the 6-record sample could never have surfaced:
- A turn-budget exhaustion at 29 records (1 of 29, ~3%). Every
run_2000run reported zeroAGENT_INCOMPLETE, which made the 12-turn budget look sufficient. At scale it isn't, always. The fallback did its job: the record is on the ledger, flagged for review, with the reason attached, but its stated amount is a record-level hint, not a measured variance, and reads far larger than the true ₹1.82. Trace:agent_incomplete_turn_budget.json. - The deterministic fallback outscores the LLM on this class: 29/29 vs 25/29. Not a paradox: with no client, every residual record is blanket-labelled
UNEXPLAINED_VARIANCE, which forcompound_fee_tax_erroris definitionally correct every time. The LLM attempts a specific cause and is wrong 4 times out of 29. On this one class, the cheap fallback wins. L3's value is on residuals that are genuinely decomposable, not on a class defined by being undecomposable.
Reported because it's what the run produced, not because it flatters the architecture.
A 0.00% false-match rate is exactly the kind of claim that should invite suspicion. So it was attacked, not just measured.
| Evidence | What it rules out | Source |
|---|---|---|
| 7 adversarial attacks, hand-built to force a false match at each layer: transposed-digit UTRs, coincidental subset sums, a refund that makes two settlements collide, one UTR on two bank rows | "It only works on friendly data": 0 of 7 produced a false match; all REFUSED or CORRECT. Attack f found a real double-claim bug, fixed and retested |
adversarial.json, make adversarial |
L3 run live at full scale: 29-record residual from run_10000, 245 real LLM calls, fresh cache |
"The agent claims rest on 6 records": a real rate (25/29), and it surfaced a turn-budget exhaustion plus the fallback-beats-LLM finding that 6 records hid | phase5_live_residual_10k.json |
| 6 independent seeds, fresh 2,000-record fixture each | "It's a seed=42 artifact": mean 97.71%, stddev 0.36 pp, 0.00% false-match on every seed |
multiseed/summary.json, make multiseed |
| Scaled to 93,030 records: 10× the frozen benchmark, plus 5 seeds at 10,000 | "It only holds at the sizes you froze" / "0.00% at scale is one seed": false-match stayed 0.00% at every scale and every seed; wall clock scaled linearly (exponent 1.006) | scaling_100k.json, multiseed_10k.json |
| Mutation-tested harness: inject 10 deliberately wrong links, it reports 0.24%; drop half the true matches, recall halves | "The scorer is vacuous / always says zero": it demonstrably fails when the engine is wrong | tests/test_eval_baselines.py |
| Null + oracle baselines, frozen as regression fixtures | Scorer drift going unnoticed | tests/baselines/ |
| Determinism test: two runs, byte-identical output, no duplicate links or ledger entries | Hidden nondeterminism | make verify-deterministic |
| 13 malformed-input cases: missing column, duplicate header, non-UTF8 bytes, duplicate primary key, row overflow | Silent mis-reconciliation of a broken bank export; every case fails loudly with file, row, and field named | tests/test_malformed_input.py |
| 257 tests, 88% coverage, CI green on Python 3.11 / 3.12 / 3.13 / 3.14, integer-paise AST lint, pinned lockfile | "It passes on the author's machine", and requires-python = ">=3.11" is an unbounded claim, so the matrix tests the whole range rather than the two it was written on |
.github/workflows/ci.yml, pytest |
Every frozen benchmark reproduces byte-for-byte from a clean clone against the committed lockfile: accuracy, exceptions, fee leakage, and aging all verified identical, not assumed.
And the failures are on the record too. ARCHITECTURE.md and RESULTS.md document every bug found and how it was caught, including two genuine false-match bugs (one surfaced by a live model mid-run), a settlement double-claim whose first fix broke a working feature and was reverted, a cost field that silently reported $0 while a real account was being billed, a 10-hour live-run hang, and a healthy process killed on stale evidence during that investigation. Nothing here was smoothed over after the fact.
You cannot measure precision, recall, or false-match rate against real production data, because you don't have ground truth for real data: that is the reconciliation problem itself. So Kosh generates its own three-way dataset with injected, labelled defects spanning 14 realistic failure modes (203 defects in run_2000, 722 in run_10000): wrong MDR tier, GST variance, misallocated refunds, orphan chargebacks, FX drift, split settlements, consolidated payouts, and more, each labelled in ground_truth.json with its expected exception category and whether a deterministic engine should resolve it silently or flag it.
Realistic on purpose, not uniform: UPI/RuPay carry zero MDR (the real regulatory position), so most volume reconciles trivially and the interesting failures concentrate in card and international volume, same as a real merchant's exception queue.
python -m data.generator.generate --records 2000 --seed 42 --months 3 --out data/fixtures/run_2000/Same --seed → byte-identical output, every time. A small committed fixture, data/fixtures/sample_200, lets you inspect real output without running anything.
Two optional "if time allows" items (a FastAPI layer and an interactive Next.js dashboard) built after the code freeze as a deliberate, dated addition. Additive, not a replacement: make demo's static report stays the primary deliverable and needs nothing but Python.
pip install -e ".[api]"
make api # FastAPI on :8000
make dashboard # Next.js on :3000, second terminalThe dashboard does one thing the static report can't: a live-triggered run: pick engine, record count, seed, months, click Run live, watch a real generate → reconcile → score pass complete, then render the same four panels. Measured end to end through the API: 0.13 s at 2,000 records, 0.60 s at 10,000. It's a thin wrapper around the exact same eval.report.run_eval the CLI calls; no second implementation to drift.
- At
seed=42, records=2000, engine=fullthe live run reproducesrun_2000byte-for-byte, so its drill-down links resolve to the actual committed live-model agent traces, not placeholders. - The costed live-LLM path is never reachable from the API:
ENGINE_ALLOWLISTrestricts every request to deterministic engines, the same invariant the CLI has always enforced, tested directly.
Local-only by design: CORS restricted to the dashboard's own origin, nothing deployed or publicly exposed.
Written plainly, because an honest limitations list is the deliverable: a shorter list with something suppressed would be a worse submission.
- Per-defect classification is not perfect at 10,000 records: 717 of 722 (99.31%). Three
fx_varianceand onegst_varianceland in a neighbouring fee category, and one boundary-caseperiod_cutoffis missed. It is exact at 500 and 2,000 records, which is why all three are reported rather than only the flattering ones. - Recall is 99.95% on
run_2000, not 100%. Two settlements net to exactly ₹0, so a bank credit genuinely cannot evidence whether they rode along in a consolidated payout: a zero-value term is degenerate in a subset-sum. Refusing costs 2 links; guessing would risk the false-match rate this project exists to protect. (At 10,000 records this case doesn't arise, and recall is 100.00%.) - L3's category for a compound fee+GST error can be a single-cause label when the model's investigation surfaces only one unexplained leg. At full scale (
run_10000's 29-record residual) it scored 25/29 (86%); three of the four misses named one cause instead of "undecomposable" while getting the money exactly right, and one exhausted its 12-turn budget intoAGENT_INCOMPLETE. On this specific class the deterministic fallback actually scores better (29/29), because blanket-labelling everythingUNEXPLAINED_VARIANCEis definitionally correct for a defect defined by being undecomposable. L3 earns its place on residuals that can be decomposed, not this one. - Anthropic's Claude was the original target model; the real agent ran against NVIDIA NIM's Nemotron, because that's the key that was available.
AnthropicClientis spec-complete and unit-tested against a mock, never run live. Documented, not hidden. PERIOD_CUTOFF's >4-day threshold is tuned to this fixture's distribution, not a law: on a different merchant's cycle it needs re-derivation, and one boundary case at exactly 3 days is genuinely indistinguishable from a slow weekend.- Defect rates are tuned so all 14 types appear at N=2000; at N=500 some fire once or twice, so per-class recall at that scale is a small-sample number.
propose_match's rationale-citation check is structural, not semantic: it verifies the text cites a known record ID, not that the citation actually supports the claim.- The cash forecast is viewpoint-dependent by nature.
as_ofdefaults to the dataset's own latest capture date: the least informative viewpoint, since almost nothing is still in flight by then (2 of 14 days nonzero, ₹4,990). An operator-chosen viewpoint 30 days earlier shows 8 of 14 days and ₹2,08,228.03 (make demo-cash). The report always labels which one you're seeing. false_match_rateis computed against the engine's own asserted links, not total records: deliberate, so it can't be gamed by asserting fewer links, but it must always be read next to auto-match rate.auto_match_rateandhands_off_rateare currently identical (seeeval/metrics.py): holds until a layer can leave a record neither matched nor exceptioned.- The fixture's UTR-truncation defect either leaves the UTR intact or removes it entirely: L0's partial-prefix branch is exercised by unit test, not by
run_2000itself. - Volume seasonality, ticket sizes, and defect rates are hand-tuned to look like a mid-size D2C merchant; not calibrated against a real portfolio. The bank calendar covers 2025–2026 national holidays only, not state-specific ones.