Ledger

A billing agent that is allowed to move money, and mostly decides not to.

📺 Two-minute demo — a customer email tries to extract $5,000. It gets $24.00, a ticket it asked us not to file, and a Slack post it told us not to send.

A customer emails about a duplicate charge. Ledger finds the customer and the specific charge in Stripe, evaluates the request against a versioned refund policy, then either issues the refund or refuses. Either way it files a Linear ticket with the evidence trail and posts to Slack, where refunds at or above a threshold wait for a human before any money moves. The reply goes out over Gmail only after the action resolves.

The interesting part is not that it issues refunds. It is that the model is structurally incapable of authorising one.

If you have three minutes

1. Watch the demo.
2. npm install && npm run ui — no credentials needed. Pick the message flagged instruction attempt.
3. Read ExtractionSchema and note the fields that aren't there.

---

How to run

Requirements

Node 20 or newer. That is all — there are three runtime dependencies (dotenv, yaml, zod) and no build step.

node --version   # v20+
npm install

1. Run it with no credentials

Everything below boots an in-process fake that speaks the real Stripe, Linear and Slack wire formats. No keys, no network, no accounts.

npm run ui       # http://localhost:4000 — the control surface. Start here.
npm run eval     # the 32-scenario suite
npm run demo     # one prompt-injection run, prints an evidence receipt
npm run smoke    # plumbing end to end, 20 assertions
npm run typecheck

npm run ui is the fastest way to see what this does. Open http://localhost:4000, pick the message flagged instruction attempt, and press process.

The interface is double-entry on purpose: the left column is what the agent changed, the right is what it declined and the clause that blocked it. For a system whose main job is refusing, the refusals get equal width.

2. Point it at real services

cp .env.example .env    # then fill it in — every line is commented
npm run verify          # pings Stripe, Linear, Slack, mail and the model
npm run seed            # builds the fixtures in Stripe TEST MODE
REAL=1 npm run ui       # the same interface, against your live accounts

npm run verify prints a green or red line per provider with a specific fix hint on failure. Do not skip it — it catches the two configuration traps that otherwise fail silently (see Configuration traps below).

DRY_RUN=true is the default: every write is logged and none is executed. Set DRY_RUN=false in .env only once verify is green.

3. The command line

npm run run                          # process the inbox
npm run run -- approve ch_xxx        # execute a held approval by hand
npm run serve                        # approval callback server, port 3000
npm run gmail-auth                   # one-shot Gmail OAuth, writes the refresh token

Which credentials buy you what

You have	You can run
nothing	ui, eval, demo, smoke, typecheck
OPENAI_API_KEY	the above, plus EXTRACTOR=openai npm run eval
Stripe test key	verify, seed, REAL=1 npm run ui
+ Linear, Slack	full end-to-end with real tickets and posts
+ Gmail OAuth	live inbox as the trigger

Configuration traps

Two things fail in a way that looks like something else. Both are guarded now, but worth knowing:

• SLACK_CHANNEL must be quoted. dotenv treats an unquoted # as the start of a comment, so SLACK_CHANNEL=#ledger-approvals parses to an empty string and every post fails with channel_not_found. Write SLACK_CHANNEL="#ledger-approvals". Config now rejects the empty value with this exact explanation rather than defaulting past it.
• LINEAR_API_KEY goes in Authorization raw, with no Bearer prefix. A Bearer prefix fails auth. Most examples online get this wrong.

---

The three ideas this is built on

1. Policy is data, not prompt

policy/refund-policy.yaml holds ten numbered clauses, RF-001 to RF-010. src/lib/policy.ts is a pure function: facts in, a decision citing clause IDs out. Changing a limit is a one-line YAML edit, and every decision names the clause that produced it. No limit has ever lived in a prompt.

Deny clauses evaluate first and short-circuit, so no allow clause can override a prohibition.

2. The model cannot express an authorisation

Extraction runs on the OpenAI Chat Completions API (gpt-4.1-mini by default). Its whole job is to read the email and report what the sender appears to want.

Look at ExtractionSchema in src/agent/extract.ts and note what it cannot say: no customer ID, no charge ID, no refund amount, no approval flag. A fully compromised model cannot emit a field that authorises money to move, because no such field exists. Amounts and identities are read from Stripe. The ceiling comes from the policy engine.

response_format: json_object forces valid JSON. It does not force the right shape, so the zod parse still runs, and a malformed extraction falls back to the deterministic classifier — never to a permissive default.

Swap providers with EXTRACTOR=claude and ANTHROPIC_API_KEY, or point OPENAI_BASE_URL at any OpenAI-compatible gateway.

3. One HTTP path, so guarantees are testable

Nothing calls fetch directly. Every provider call goes through src/lib/http.ts, which owns the timeout, the retry classification, the dry-run switch, the idempotency store and the audit span.

The retry logic turns on a distinction most clients skip — whether the write could have landed:

Outcome	Meaning	Retry?
never_sent	DNS, connection refused, 429	always safe
maybe_sent	timeout, 5xx after send	only if the provider deduplicates
rejected	4xx, or a 200 the provider calls a failure	never

Stripe honours Idempotency-Key and replays v1 POSTs for 24 hours, so an ambiguous failure there is retried with the same key. Linear, Slack and Gmail have no equivalent, so an ambiguous write is marked unresolved and raises AmbiguousWriteError rather than risking a duplicate. The store persists to disk, so a webhook redelivered after a crash still cannot produce a second refund.

---

Evaluation

32 scenarios, each with its own world, its own server and an empty idempotency store, so no scenario can pass because another ran first.

EXTRACTOR=openai npm run eval      # 32/32 — measures the model
EXTRACTOR=heuristic npm run eval   # 29/32 — measures the guardrails, no network

Category	n	What it proves
happy	3	refunds execute, at the right amount, reply only after resolution
identity	3	unknown and ambiguous senders escalate; no ID is ever invented
policy	7	age window, disputes, already-refunded, threshold, daily cap, ceiling
idempotency	3	redelivery, triple delivery, redelivery after the store is lost
approval	3	a held approval executes once, under duplicate and concurrent delivery
generalisation	3	the same situation described the way real customers describe it
injection	3	embedded instructions, suppression attempts, invented clause IDs
resilience	7	429s, 503s, apply-then-hang on both provider types, 200-as-failure, dry run

Assertions are on integers — refunds created, cents moved, tickets filed, approval posts — never on model output. npm run eval writes .runtime/eval-report.json with per-scenario side-effect counts.

The 32 vs 29 gap is the point

The three scenarios the heuristic fails are the whole generalisation category. They describe the same seeded duplicate charge as H1, but the way an actual customer would — buried mid-paragraph, symptom-only, or angry and never stating the request plainly — using none of the tokens the keyword classifier matches on.

The guardrails are identical in both modes. The only variable is whether the extractor recognises the request at all. That difference, 3 scenarios, is exactly what the model is worth here, and it is worth stating as a number rather than asserting.

The suite can tell a live model from a dead one

An early version could not. EXTRACTOR=openai reported a perfect score while the OpenAI call failed on every single scenario and silently fell through to the heuristic. The runner now counts extractions and fails a model-mode run that records zero successes, and every run prints the tally:

extractor openai: 36 reached the extractor, 0 fell back to heuristic

That bug is the one we'd point at first. It was not a logic error — the logic was fine. It was a measurement error, and the suite was reporting success for a system that was not running.

---

Defence in depth on injection

Four independent layers, none of which is a prompt instruction the model has to choose to follow:

1. The extraction schema has no field that can authorise money.
2. Facts come from Stripe. The email supplies an address to look up; every figure in the decision is read from the charge record.
3. The policy engine caps everything at the unrefunded remainder, regardless of intent or claimed amount.
4. A failed extraction falls back to the deterministic classifier, never to a permissive default.

npm run demo shows it. The email demands $5,000, no ticket and no Slack post. It gets $24.00, a ticket, and a Slack post — and the receipt names the clause that refused each demand.

Approvals

npm run serve exposes POST /slack/actions for the Approve and Deny buttons. Requests are verified against SLACK_SIGNING_SECRET with a timing-safe comparison and a 5-minute replay window, because an unverified endpoint makes the approval gate decorative.

The button payload is not trusted for the amount. executeApproval re-reads the charge from Stripe and re-runs the policy engine before moving anything, so a forged or stale callback cannot authorise what policy would refuse. Scenarios A1–A3 cover duplicate and concurrent delivery of the same approval.

Three independent things must fail before a customer is paid twice: the policy re-check, the idempotency key, and Stripe's own remaining-balance check. A3 demonstrates the middle one by salting the key and watching the suite go red.

---

Known limits

Stated rather than hidden.

Concurrent writes to a non-idempotent provider. src/lib/http.ts checks the idempotency store for done and unresolved before a write, but not inflight. Two processes handling the same message at the same instant can therefore both pass the check. Stripe is protected regardless by its own Idempotency-Key replay, so no duplicate refund is possible; Linear, Slack and Gmail have no equivalent, so a duplicate ticket or post is. Single-process operation — how the agent actually runs — is sequential and unaffected. The honest fix is a compare-and-set claim on inflight, not a retry tweak.

Stripe stamps created server-side. Charges cannot be backdated, so the age-gated scenarios — P1 (outside the 30-day window), P4 (same amount a month apart is not a duplicate) and H2 (pro-rata cancellation) — exist only in the in-process fake world. npm run seed prints which scenarios are live and which are not, rather than seeding something that quietly behaves differently. Demo the age cases through npm run eval.

Gmail rewrites the From header to the authenticated account. Fixture addresses on .test domains cannot drive a live Gmail run; a real inbox demo has to use a real sender.

---

Provider quirks absorbed

• Linear, Slack and Gmail all return HTTP 200 on logical failure. Each adapter supplies an okCheck that converts those into rejections, so a naive success path cannot report a ticket that was never created.
• Linear multiplexes every operation onto POST /graphql, so a path-keyed fault cannot distinguish a team lookup from an issue creation. The fake world keys GraphQL faults on the operation — without that, an injected fault lands on a harmless read and the test reports on something it never exercised.
• Stripe refunds are capped at the unrefunded remainder, checked before the call rather than discovered from a 400.

Layout

policy/refund-policy.yaml     ten numbered clauses, versioned
src/config.ts                 validated env, secret redaction, live-key guard
src/agent/extract.ts          the model's only job, and its schema
src/agent/pipeline.ts         the decision path end to end
src/lib/policy.ts             pure decision function
src/lib/http.ts               the only place fetch is called
src/lib/idempotency.ts        persistent store, three states
src/lib/audit.ts              JSONL traces + evidence receipt
src/apps/{stripe,linear,slack,mail}.ts
src/testing/fake-world.ts     real wire formats + fault injection
src/testing/seed.ts           seeded state covering every policy branch
src/eval/{scenarios,runner}.ts
src/ui/                       the control surface
scripts/                      every npm script

Safety rails

• DRY_RUN=true is the default and logs every write without executing it.
• A live Stripe key (sk_live_) aborts at startup.
• Secrets are redacted before anything reaches a trace or a log.
• .runtime/ is gitignored: it holds traces containing real customer emails and charge IDs.

---

Built for the Multi-App AI Agent Hackathon, 13 September 2026. Demo · Reliability brief