Damage photos, a scanned policy, and adjuster notes go in. An evidence-backed recommendation — with its reasoning, precedents, and audit trail — comes out.
The AI gathers and reasons. A human decides.
Note
Live demo: evincta.vercel.app → click “Enter as demo adjuster.” The API runs on a free-tier container that sleeps when idle, so the first load may take ~30s to wake — after that it's fast. Browsing and deciding claims costs nothing; recommendations are pre-computed and cached.
The adjuster's review console: queue, recommendation with cited precedents, evidence, and the approve / override gate.
A production-grade AI system, not a notebook demo. Vision · retrieval · tool-using agents · human-in-the-loop · held-out eval · deployed · hardened — end to end.
traced · evaluated · guarded · cost-controlled
- Why this project
- At a glance
- What it does
- Why it's technically interesting
- Architecture
- The agent design
- Evaluation
- What I actually debugged
- Production hardening
- Engineering standard
- Scope & boundaries
- Run it locally
- Tech stack
- Roadmap & limitations
- Repository map
Auto-claims triage is 20–30 minutes of multimodal context-gathering per claim before anyone decides anything: read the damage photos, check what the policy covers, estimate a repair cost, screen for fraud, and dig up comparable past claims. It's slow, inconsistent between adjusters, and exactly the kind of judgment-under-evidence work where an LLM system can do the gathering and synthesis — while a human keeps the decision.
Evincta is that system. It's deliberately built as a decision-support engine, not a classifier: it never auto-approves a payout. It assembles the evidence, reasons over it with a panel of specialised agents, and hands a human adjuster a recommendation they can accept or override — with every step traced, evaluated, and audited.
It is the flagship of a four-project portfolio, all built to one standard: traced · evaluated · guarded · cost-controlled.
| Domain | Auto / vehicle-damage insurance claims |
| Input | Damage photos · scanned policy · free-text adjuster notes |
| Output | approve / investigate / deny · payout range · fraud-risk · policy basis · cited precedents · rationale |
| Core pattern | Deterministic tools decide facts; a single constrained LLM synthesises the decision |
| Safety model | Structural human-in-the-loop gate + hash-chained audit + graceful degradation to human review |
| Proof | 18 held-out claims · asymmetric scoring · CI gate that fails the build on regression |
| Live | Deployed on Vercel + Render (free tier), secured and rate-limited |
A claim — damage photos + a scanned policy + free-text adjuster notes — flows through six stages:
📸 claim 🔍 extract 🧭 retrieve 🧠 reason 👤 decide 🧾 audit
photos + policy → vision → typed → multimodal → specialist agents → human approves → hash-chained
+ notes claim record precedent search + LLM synthesiser or overrides audit log
The output isn't a label — it's a structured recommendation: approve / investigate / deny, a payout range, a fraud-risk flag, the applicable policy clauses, the precedent claims it relied on, and a plain-language rationale. The adjuster sees all of it and makes the call.
This is the part most "AI projects" skip. Evincta is built like production software, and each choice below is deliberate, with a trade-off behind it:
| 🧩 Real MCP, not a metaphor | Four actual Model Context Protocol servers (policy · cost · fraud · precedent) behind one host, launched as subprocesses and reused across requests. The agents call tools over a real protocol — the primitive that lets these capabilities be swapped, versioned, or served remotely. |
| ⚖️ Hybrid agents — trust placed deliberately | The specialists are deterministic tool-callers (coverage, cost, fraud, precedent) — no LLM guessing where math and policy lookups belong. Exactly one LLM does the hard part: synthesising the evidence into a decision, constrained to a typed schema. LLMs synthesise; tools decide facts. |
| 🚦 Human-in-the-loop, enforced structurally | The graph cannot commit a decision. It runs to an interrupt, persists state via a checkpointer, and waits. The decision only exists when a human POSTs approve/override — which resumes the graph and writes the audit. It's not a convention; the machine can't skip it. |
| 📊 A held-out eval that gates CI | 18 claims are held out and never indexed. Every push runs the full pipeline against ground truth with asymmetric scoring (approving a should-deny claim is penalised far more than a cautious flag) and fails the build on regression. |
| 🔒 Tamper-evident audit | Every decision appends to a hash-chained JSONL log — each entry carries the previous entry's hash, so the record can't be silently edited after the fact. |
| 🛡️ Hardened and tested | Retry + circuit breaker with graceful degradation to human review, rate limiting, a daily spend cap, path-traversal and schema validation, structured logs, and dependency health checks — each verified by tests that run in CI. Infra-scale items are documented as roadmap, not faked. |
┌──────────────────────── Review console (React + Tailwind, Vercel) ─────────────────────────┐
│ claim queue · recommendation + cited precedents + evidence · approve / override │
└───────────────────────────────────────────┬───────────────────────────────────────────────┘
│ HTTPS (API key on writes · CORS locked)
┌────────────────────────────────────────────▼────────────────────────────────────────────────┐
│ FastAPI service (Docker → Render, free tier) │
│ │
📸 claim ──────────►│ ① INGEST ② INDEX ③ MCP HOST + 4 SERVERS ④ AGENT GRAPH │
photos + policy │ Claude vision → Voyage multimodal → policy · cost · fraud · → specialists (tools) │
+ notes │ → typed record embeddings + precedent (FastMCP) + LLM synthesiser │
│ Pinecone search one long-lived host → typed recommendation│
│ │ │
│ ⑤ HUMAN GATE (interrupt) ◄───────┘ │
│ │ approve / override │
│ ⑥ AUDIT (hash-chained JSONL) │
└───────────────────────────────────────────────────────────────────────────────────────────┬─┘
│
Langfuse tracing (every call, every cost) · held-out eval + CI gate (GitHub Actions) │
Every layer has an Architecture Decision Record explaining why it's built that way — MCP transport, agent topology, eval design, deploy/security posture, and the hardening line. The Langfuse trace below shows one real end-to-end run — coverage → cost → precedent → fraud → recommendation — in a single 26s span:
One claim, fully traced — every model call and its cost, observable end to end.
The single most important design decision in Evincta is where the LLM is allowed to think.
Most "multi-agent" systems hand every step to an LLM and hope. Evincta doesn't. The four specialists are deterministic — they call MCP tools and return facts:
- Coverage — is this incident covered? (policy exclusions, limits, deductible) → a hard fact.
- Cost — estimated repair range for this severity and damaged parts → a lookup, not a guess.
- Fraud — deterministic signals (pre-existing damage, inconsistencies) → rules, not vibes.
- Precedent — the k most similar past claims by multimodal similarity → retrieval.
Then exactly one LLM call — the synthesiser — takes those facts and produces the decision, forced into a typed schema (Decision and FraudRisk enums, a payout range, cited precedent IDs, a rationale). Its prompt applies rules in priority order: coverage is an absolute deny-gate; then fraud escalates to investigate; then below-deductible denies; else approve.
Why this matters: an LLM asked to "decide the claim" will cheerfully override a coverage exclusion or invent a payout. By making the facts deterministic and constraining the synthesis to a schema, a model hallucination can't produce an out-of-policy decision — the worst it can do is pick the wrong enum, which the eval catches and the human gate backstops. Deterministic tools decide facts; the LLM only synthesises within their constraints.
Evincta is scored end-to-end on 18 held-out claims — never indexed, never seen during retrieval — against ground truth, with a CI gate that fails the build on regression.
| Metric | Value | |
|---|---|---|
| Decision accuracy (weighted) | 0.722 | risk-weighted — the gated metric |
| Decision accuracy (exact) | 0.722 | |
| Payout in range | 0.769 | (n=13) |
| Extraction — severity | 0.889 | |
| Extraction — incident type | 0.833 | |
| Latency p50 / p95 | 14.9s / 16.6s | reported, not gated |
Weighted accuracy is the headline, and here's why it's strong: it penalises dangerous errors — approving a claim that should be denied — far more than cautious ones. Weighted equals exact here, which means the system makes no dangerous errors on the held-out set. When Evincta is wrong, it errs toward caution (flagging for review), never toward over-paying. For a claims system, that's the failure mode you want.
Scope & honesty on fraud recall (0.0, n=2) — click to expand
Fraud recall is reported separately and is not gated, deliberately. The synthetic frauds in the eval set are image-reuse type, whose only signal is image/incident inconsistency — indistinguishable, at the data level, from the noise in synthetically-paired images. Reliable detection needs perceptual image hashing across claims, which is roadmapped, not implemented. Gating a two-sample metric would be dishonest.
Two things make this an acceptable, honest position rather than a gap: the human approval gate means a missed fraud flag is never an automatic payout, and the limitation is named explicitly rather than hidden behind a flattering number. Knowing what real fraud detection requires — and scoping it deliberately — is the point.
Gated thresholds: eval/thresholds.yaml · Full results: eval/REPORT.md · Design rationale: ADR 0005
The eval didn't just produce a number — it caught real bugs, and tracing them one variable at a time is the part of this project I'd most want to talk through. Each was found by isolating the pipeline stage, fixing the root cause, and committing it as its own clean change:
| # | Symptom | Root cause | Fix |
|---|---|---|---|
| 1 | Minor claims "approved" for $0 | Ground-truth labels approved payouts below the deductible | Below-deductible → deny; regenerated data |
| 2 | Everything flagged fraud | A fraud signal fired on normal payout variance vs precedent | Removed the signal — spread isn't evidence |
| 3 | Model wrote prose in a risk field | fraud_risk was a free string |
Constrained to a typed enum |
| 4 | Excluded claims approved | Synthesiser overrode the coverage tool | Prompt makes coverage an absolute deny-gate |
| 5 | Over-investigate ↔ miss fraud | One signal was doing double duty (fraud + noise) | Decoupled it; scoped fraud honestly |
The throughline — bugs #3 and #4 are the same lesson twice: the LLM overriding a deterministic tool. The fix is consistent — constrain outputs to schemas, and make the prompt treat tool results as authoritative gates. That's the central discipline of building an LLM decision system, learned by hitting it, not reading it.
The git history preserves this as separate fix: commits with before/after numbers in the bodies — the debugging arc is legible in the log.
The live system is hardened across five areas. Application-level hardening is implemented and verified by tests that run in CI; infrastructure-scale items are documented as precise roadmap, not faked — the line between the two is itself a deliberate engineering decision (ADR 0008).
| Area | Implemented | Verified by |
|---|---|---|
| Resilience | Retry + timeout + backoff on every external call; circuit breaker; graceful degradation — a dependency failure routes the claim to human review, never a wrong automated decision | Unit tests (breaker opens/half-opens; forced failure → investigate) |
| Cost & abuse | Per-route rate limiting; a daily spend cap (a circuit breaker for the API bill); request-size limits | 429 on limit breach; cap refuses spend over ceiling; 413 on oversized body |
| Robustness | Path-traversal input validation; schema-validated model output; prompt-injection defense-in-depth (delimited untrusted input · constrained output enum · human gate) | 422 on traversal input; malformed output raises |
| Observability | Structured JSON logging (no secrets/PII); liveness + dependency-checking readiness probes | /health/ready reports per-dependency status |
| Concurrency | Shared-host tool calls serialised under a lock | Documented Postgres/Redis seam for multi-instance scale |
The safest failure mode for a decision system: on any partial failure, Evincta escalates to a human rather than deciding on incomplete evidence. A crash is bad; a confident wrong decision on missing data is worse. The circuit breaker degrades into the human gate — reusing the safety mechanism the whole system is built around.
Roadmap (needs infrastructure beyond the free demo): Postgres for the claim index + LangGraph checkpointer and Redis for the limiter/spend counter (multi-instance scale); real-time alerting on top of the existing Langfuse tracing. Prompt injection is mitigated via defense-in-depth, not claimed solved.
- Traceable — every model call and its cost is observable in Langfuse; one full trace is shown above.
- Evaluated — a held-out set, asymmetric scoring aligned to real risk, and a CI gate that blocks regressions.
- Guarded — resilience, rate limits, spend caps, and input/output validation, each with a test.
- Cost-controlled — token spend is estimated per call and hard-capped; the demo is cached to cost nothing to view.
- Decision-recorded — eight ADRs capture why each layer is built the way it is.
- Legible history — small, focused commits; the debugging arc is readable in the log.
- Honest — limitations (fraud recall, single-instance state, demo auth) are stated plainly, with a documented path forward for each.
The through-line: production-minded AI engineering — building systems that are observable, measurable, safe to fail, and honest about their edges.
Evincta is the adjudication engine, not a claimant intake portal. Claims come from a synthetic dataset that stands in for an upstream intake pipeline — the way intake and adjudication are separate systems in a real carrier. This is deliberate:
- The synthetic data has ground truth, which is what makes the held-out eval possible at all. Real user uploads have no labels to score against.
- The differentiating engineering is after the claim arrives — extraction, retrieval, agents, the gate. A file-upload form demonstrates none of it.
- No real PII; English + a single currency; not integrated with any real insurer; no fine-tuning.
git clone https://github.com/AttiR/evincta.git && cd evincta
# 1. environment
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # add your keys: ANTHROPIC, VOYAGE, PINECONE (+ optional LANGFUSE)
# 2. build the demo data + index (one-time)
python -m data.generate # synthetic claims with ground truth
python -m services.index.build # embed + upsert precedents to Pinecone
# 3. run the API (boots one MCP host, reused across requests)
uvicorn api.main:app --reload # → http://localhost:8000 (GET /health → ok)
# 4. run the review console
cd web && npm install && npm run dev # → http://localhost:5173Or the whole API in one container:
docker build -t evincta-api . && docker run --rm -p 8000:8000 --env-file .env evincta-apiRun the eval, the gate, and the hardening tests:
python -m eval.run_eval # scores the 18 held-out claims end-to-end
python -m eval.gate # exit 0 if every gated metric passes
pytest tests/ -v # eval + hardening tests (breaker, degradation, validation)| Layer | Choice |
|---|---|
| Vision + reasoning | Claude Sonnet (multimodal extraction + synthesis) |
| Embeddings | Voyage voyage-multimodal-3 (1024-dim, image + text in one space) |
| Vector DB | Pinecone (serverless) |
| Tools | Model Context Protocol — 4 FastMCP servers + a shared host |
| Orchestration | LangGraph (typed state · checkpointer · interrupt human gate) |
| API | FastAPI (async · lifespan-managed MCP host) |
| Frontend | React + TypeScript + Tailwind |
| Observability | Langfuse (per-call tracing + cost) |
| Resilience | tenacity (retry/backoff) · custom circuit breaker · slowapi (rate limit) |
| Eval / CI | Custom harness · pytest · GitHub Actions gate |
| Deploy | Docker → Render (API, free) · Vercel (frontend) |
| Language / CI | Python 3.12 · GitHub Actions |
- Phase 1 — Multimodal ingest (vision → typed claim record)
- Phase 2 — Multimodal vector DB + precedent retrieval
- Phase 3 — MCP servers (policy · cost · fraud · precedent) + host
- Phase 4 — Multi-agent recommender + human gate + audit
- Phase 5 — Held-out eval harness + asymmetric scoring + CI gate
- Phase 6 — FastAPI + React console · deploy · security · production hardening
- Next — perceptual-hash fraud detection · Postgres + Redis for horizontal scale · real-time alerting · SSO
Known limitations (stated plainly): fraud recall is limited without perceptual hashing; the demo runs single-instance (SQLite + JSON index); latency is ~15s/claim (sequential vision + synthesis calls); auth is demo-grade, SSO-ready at the same seam. None are hidden — each has a documented path forward.
evincta/
├─ services/
│ ├─ ingest/ # ① Claude vision → typed claim record
│ ├─ index/ # ② Voyage embeddings + Pinecone precedent search
│ ├─ mcp/ # ③ 4 FastMCP servers + the shared host
│ ├─ agents/ # ④ specialists (deterministic) + synthesiser (LLM) + graph + audit
│ └─ common/ # resilience (retry · breaker · budget) · logging
├─ api/ # FastAPI: lifecycle endpoints + human gate + auth
├─ web/ # React + Tailwind review console
├─ eval/ # ⑤ held-out eval · asymmetric metrics · thresholds · CI gate
├─ tests/ # eval + hardening tests (run in CI)
├─ docs/adr/ # architecture decision records (0001–0008)
└─ .github/workflows/ # CI · eval gate · hardening tests
Built by Atti Rehman
Traced · evaluated · guarded · cost-controlled.
The AI gathers and reasons. A human decides.