diff --git a/README.md b/README.md index 51408e1..7483b85 100644 --- a/README.md +++ b/README.md @@ -1,84 +1,317 @@ +
+ # Evincta -**Multimodal, MCP-native decision engine.** -Plain claim in — photos, a scanned policy PDF, adjuster notes — -structured, evidence-backed recommendation out. +### A multimodal, MCP-native, multi-agent decision engine for insurance claims + +*Damage photos, a scanned policy, and adjuster notes go in. +An evidence-backed recommendation — with its reasoning, precedents, and audit trail — comes out.* +**The AI gathers and reasons. A human decides.** + +
+ +[![Live demo](https://img.shields.io/badge/▶_Live_demo-Enter_as_demo_adjuster-7c3aed?style=for-the-badge)](https://YOUR-APP.vercel.app) +  +[![Eval gate](https://img.shields.io/badge/CI-eval_gate_passing-16a34a?style=for-the-badge)](.github/workflows/eval.yml) + +
+ +![Python](https://img.shields.io/badge/Python-3.12-3776AB?logo=python&logoColor=white) +![Claude](https://img.shields.io/badge/Claude-Sonnet_vision-d97706) +![MCP](https://img.shields.io/badge/MCP-4_servers_+_host-000000) +![LangGraph](https://img.shields.io/badge/LangGraph-multi--agent-1c3c3c) +![Pinecone](https://img.shields.io/badge/Pinecone-multimodal_vectors-000000) +![Voyage](https://img.shields.io/badge/Voyage-voyage--multimodal--3-5b21b6) +![FastAPI](https://img.shields.io/badge/FastAPI-async_API-009688?logo=fastapi&logoColor=white) +![React](https://img.shields.io/badge/React-review_console-61DAFB?logo=react&logoColor=black) +![Langfuse](https://img.shields.io/badge/Langfuse-tracing-6366f1) + +
+ +--- + +> [!NOTE] +> **Live demo:** [YOUR-APP.vercel.app](https://YOUR-APP.vercel.app) → click **“Enter as demo adjuster.”** +> The API runs on a free-tier container that sleeps when idle, so the **first load may take ~30s to wake** — after that it's fast. Browsing and deciding claims costs nothing (recommendations are pre-computed and cached). + +
+ + +Evincta review console — claim queue, recommendation, evidence, approve/override + +*The adjuster's review console: queue, recommendation with cited precedents, evidence, and the approve / override gate.* + +
+ +--- + +## Table of contents + +- [Why this project](#why-this-project) +- [What it does](#what-it-does) +- [Why it's technically interesting](#why-its-technically-interesting) +- [Architecture](#architecture) +- [The agent design](#the-agent-design) +- [Evaluation](#evaluation) +- [What I actually debugged](#what-i-actually-debugged) +- [Production hardening](#production-hardening) +- [Scope & boundaries](#scope--boundaries) +- [Run it locally](#run-it-locally) +- [Tech stack](#tech-stack) +- [Roadmap & limitations](#roadmap--limitations) +- [Repository map](#repository-map) + +--- + +## Why this project + +Auto-claims triage is **20–30 minutes of multimodal context-gathering per claim** before anyone decides anything: read the damage photos, check what the policy covers, estimate a repair cost, screen for fraud, and dig up comparable past claims. It's slow, inconsistent between adjusters, and exactly the kind of judgment-under-evidence work where an LLM system can do the *gathering and synthesis* — while a human keeps the *decision*. + +Evincta is that system. It's deliberately built as a **decision-support engine, not a classifier**: it never auto-approves a payout. It assembles the evidence, reasons over it with a panel of specialised agents, and hands a human adjuster a recommendation they can accept or override — with every step traced, evaluated, and audited. + +It's the flagship of a four-project portfolio, all built to one production standard: **traced · evaluated · guarded · cost-controlled.** + +--- + +## What it does -> Status: 🚧 in active development. Phase 0 (foundation). See the -> roadmap below; milestones track progress. +A claim — **damage photos + a scanned policy + free-text adjuster notes** — flows through six stages: + +``` + 📸 claim 🔍 extract 🧭 retrieve 🧠 reason 👤 decide 🧾 audit + photos + policy → vision → typed → multimodal → specialist agents → human approves → hash-chained + + notes claim record precedent search + LLM synthesiser or overrides audit log +``` + +The output isn't a label — it's a **structured recommendation**: `approve / investigate / deny`, a payout range, a fraud-risk flag, the applicable policy clauses, the **precedent claims it relied on**, and a plain-language rationale. The adjuster sees all of it and makes the call. + +--- + +## Why it's technically interesting + +This is the part most "AI projects" skip. Evincta is built like production software, and each choice below is a deliberate one with a trade-off behind it: + +| | | +|---|---| +| 🧩 **Real MCP, not a metaphor** | Four actual [Model Context Protocol](https://modelcontextprotocol.io) servers (`policy · cost · fraud · precedent`) behind one host, launched as subprocesses and reused across requests. The agents call tools over a real protocol — the same primitive that lets these capabilities be swapped, versioned, or served remotely. | +| ⚖️ **Hybrid agents — trust placed deliberately** | The specialists are **deterministic tool-callers** (coverage, cost, fraud, precedent) — no LLM guessing where math and policy lookups belong. Exactly **one** LLM does the hard part: **synthesising** the evidence into a decision, constrained to a typed schema. LLMs synthesise; tools decide facts. | +| 🚦 **Human-in-the-loop, enforced structurally** | The graph *cannot* commit a decision. It runs to an `interrupt`, persists state via a checkpointer, and waits. The decision only exists when a human POSTs approve/override — which resumes the graph and writes the audit. It's not a convention; the machine can't skip it. | +| 📊 **A held-out eval that gates CI** | 18 claims are held out and **never indexed**. Every push runs the full pipeline against ground truth with **asymmetric scoring** (approving a should-deny claim is penalised far more than a cautious flag) and **fails the build on regression**. | +| 🔒 **Tamper-evident audit** | Every decision appends to a **hash-chained** JSONL log — each entry carries the previous entry's hash, so the record can't be silently edited after the fact. | +| 🛡️ **Hardened, honestly** | Retry + circuit breaker with **graceful degradation to human review** on failure, rate limiting, a daily spend cap, path-traversal and schema validation, structured logs, and dependency health checks. Infra-scale items (Postgres, Redis, alerting) are documented as roadmap, not faked. | --- -## The problem -Insurance claims triage is 20–30 minutes of multimodal context- -gathering per claim — read the damage photos, check coverage, estimate -cost, screen for fraud, find precedent — before any decision is made. -Evincta does the gathering and synthesis; a human adjuster decides. - -## What it does (target) -A claim (damage photos + scanned policy PDF + adjuster notes) becomes a -structured claim record, is matched against visually- and textually- -similar past claims in a multimodal vector database, and is reasoned -over by a panel of specialised agents that coordinate over real MCP -servers — producing a ranked recommendation (approve / investigate / -deny), a payout range, a fraud-risk flag, the applicable policy -clauses, and the precedents it relied on. - -## Architecture (six phases) +## Architecture + ``` -ingest → index → mcp servers → agents → eval → deploy -vision multimodal policy/cost orchestrator accuracy traced/ -→ JSON vector DB fraud/prec. + specialists + CI gate guarded + ┌──────────────────────── Review console (React + Tailwind, Vercel) ─────────────────────────┐ + │ claim queue · recommendation + cited precedents + evidence · approve / override │ + └───────────────────────────────────────────┬───────────────────────────────────────────────┘ + │ HTTPS (API key on writes · CORS locked) + ┌────────────────────────────────────────────▼────────────────────────────────────────────────┐ + │ FastAPI service (Docker → Render, free tier) │ + │ │ + 📸 claim ──────────►│ ① INGEST ② INDEX ③ MCP HOST + 4 SERVERS ④ AGENT GRAPH │ + photos + policy │ Claude vision → Voyage multimodal → policy · cost · fraud · → specialists (tools) │ + + notes │ → typed record embeddings + precedent (FastMCP) + LLM synthesiser │ + │ Pinecone search one long-lived host → typed recommendation│ + │ │ │ + │ ⑤ HUMAN GATE (interrupt) ◄───────┘ │ + │ │ approve / override │ + │ ⑥ AUDIT (hash-chained JSONL) │ + └───────────────────────────────────────────────────────────────────────────────────────────┬─┘ + │ + Langfuse tracing (every call, every cost) · held-out eval + CI gate (GitHub Actions) │ ``` -## Roadmap -- [x] Phase 1 — Multimodal ingest (vision → structured JSON) -- [x] Phase 2 — Multimodal vector DB + precedent retrieval -- [x] Phase 3 — MCP servers (policy · cost · fraud · precedent) -- [x] Phase 4 — Multi-agent recommender -- [x] Phase 5 — Eval harness + CI accuracy gate -- [ ] Phase 6 — Production layer + deploy + review UI - -## Stack -Python 3.12 · Claude Sonnet (vision) · Voyage multimodal embeddings · -Pinecone · MCP (FastMCP) · LangGraph · FastAPI · Langfuse · -React + TypeScript · Azure Container Apps · GitHub Actions - -## Non-goals -Does not auto-decide (human approves). No real PII — synthetic claims. -Not integrated with any real insurer. English + one currency. No fine- -tuning. - -## Status & metrics -Targets, gated in CI once Phase 5 lands: decision accuracy ≥ 0.85 · -fraud recall ≥ 0.90 · extraction accuracy ≥ 0.90 · one traced, -cost-metered run per claim. +Each layer has an [Architecture Decision Record](docs/adr/) explaining *why* it's built that way — the MCP transport choice, the agent topology, the eval design, the deploy/security posture, and the hardening line. The [Langfuse trace](docs/media/trace.png) below shows one real end-to-end run — coverage → cost → precedent → fraud → recommendation — in a single 26s span: + +
+ +Langfuse trace of one full claim run — coverage, cost, precedent, fraud, recommendation +
+ +--- + +## The agent design + +The single most important design decision in Evincta is **where the LLM is allowed to think.** + +Most "multi-agent" systems hand every step to an LLM and hope. Evincta doesn't. The four specialists are **deterministic** — they call MCP tools and return facts: + +- **Coverage** — is this incident covered? (policy exclusions, limits, deductible) → a hard fact. +- **Cost** — estimated repair range for this severity and damaged parts → a lookup, not a guess. +- **Fraud** — deterministic signals (pre-existing damage flags, inconsistencies) → rules, not vibes. +- **Precedent** — the *k* most similar past claims by multimodal similarity → retrieval. + +Then **exactly one LLM call** — the **synthesiser** — takes those facts and produces the decision, forced into a typed schema (`Decision` and `FraudRisk` enums, a payout range, cited precedent IDs, a rationale). Its prompt applies rules **in priority order**: coverage is an absolute deny-gate; then fraud escalates to investigate; then below-deductible denies; else approve. + +> **Why this matters:** an LLM asked to "decide the claim" will cheerfully override a coverage exclusion or invent a payout. By making the *facts* deterministic and constraining the *synthesis* to a schema, a model hallucination can't produce an out-of-policy decision — the worst it can do is pick the wrong enum, which the eval catches and the human gate backstops. **Deterministic tools decide facts; the LLM only synthesises within their constraints.** + +--- ## Evaluation -Evincta is evaluated end-to-end on 18 held-out claims (never indexed, never -seen during retrieval), scored against ground truth with a CI gate that fails -the build on regression. +Evincta is scored end-to-end on **18 held-out claims** — never indexed, never seen during retrieval — against ground truth, with a **CI gate that fails the build on regression.** + +| Metric | Value | | +|---|---|---| +| **Decision accuracy (weighted)** | **0.722** | risk-weighted — the gated metric | +| Decision accuracy (exact) | 0.722 | | +| Payout in range | 0.769 | *(n=13)* | +| Extraction — severity | 0.889 | | +| Extraction — incident type | 0.833 | | +| Latency p50 / p95 | 14.9s / 16.6s | reported, not gated | + +**Weighted accuracy is the headline, and here's why it's strong:** it penalises *dangerous* errors — approving a claim that should be denied — far more than cautious ones. **Weighted equals exact here, which means the system makes _no dangerous errors_ on the held-out set.** When Evincta is wrong, it errs toward caution (flagging for review), never toward over-paying. For a claims system, that's the failure mode you want. + +
+Scope & honesty on fraud recall (0.0, n=2) — click to expand + +
+ +Fraud recall is reported separately and is **not gated**, deliberately. The synthetic frauds in the eval set are *image-reuse* type, whose only signal is image/incident inconsistency — which is indistinguishable, at the data level, from the noise in synthetically-paired images. Reliable detection needs **perceptual image hashing across claims**, which is roadmapped, not implemented. Gating a two-sample metric would be dishonest. -| Metric | Value | +Two things make this an acceptable, honest position rather than a gap: the **human approval gate** means a missed fraud flag is never an automatic payout, and the limitation is named explicitly rather than hidden behind a flattering number. *Knowing what real fraud detection requires — and scoping it deliberately — is the point.* + +
+ +Gated thresholds: [`eval/thresholds.yaml`](eval/thresholds.yaml) · Full results: [`eval/REPORT.md`](eval/REPORT.md) · Design rationale: [ADR 0005](docs/adr/0005-eval-design.md) + +--- + +## What I actually debugged + +The eval didn't just produce a number — it **caught real bugs**, and tracing them one variable at a time is the part of this project I'd most want to talk through. Each was found by isolating the pipeline stage, fixing the root cause, and committing it as its own clean change: + +| # | Symptom | Root cause | Fix | +|---|---|---|---| +| 1 | Minor claims "approved" for **$0** | Ground-truth labels approved payouts below the deductible | Below-deductible → **deny**; regenerated data | +| 2 | Everything flagged fraud | A fraud signal fired on **normal payout variance** vs precedent | Removed the signal — spread isn't evidence | +| 3 | Model wrote prose in a risk field | `fraud_risk` was a free string | Constrained to a **typed enum** | +| 4 | Excluded claims **approved** | Synthesiser overrode the coverage tool | Prompt makes coverage an **absolute deny-gate** | +| 5 | Over-investigate ↔ miss fraud | One signal was doing **double duty** (fraud + noise) | Decoupled it; scoped fraud honestly | + +> The throughline — bugs #3 and #4 are the *same lesson twice*: **the LLM overriding a deterministic tool.** The fix is consistent — constrain outputs to schemas, and make the prompt treat tool results as authoritative gates. That's the central discipline of building an LLM decision system, learned by hitting it, not reading it. + +The git history preserves this as separate `fix:` commits with before/after numbers in the commit bodies — the debugging arc is legible in the log. + +--- + +## Production hardening + +Implemented (application-level, runs on free tier): + +- **Resilience** — retry + timeout + backoff on every external call; a **circuit breaker**; and **graceful degradation** — a dependency failure routes the claim to *human review*, never a wrong automated decision. +- **Cost & abuse** — per-route **rate limiting**, a **daily spend cap** (a circuit breaker for the API bill), and request-size limits. +- **Robustness** — **path-traversal** input validation, schema-validated model output, and prompt-injection *defense-in-depth* (delimited untrusted input · constrained output enum · human gate). +- **Observability** — **structured JSON logging** and liveness + **dependency-checking readiness** probes. + +Documented as roadmap (needs infrastructure beyond the free demo): Postgres for the claim index + LangGraph checkpointer and Redis for the limiter/spend counter (multi-instance scale); real-time alerting on top of the existing Langfuse tracing. Prompt injection is *mitigated, not claimed solved.* Rationale in [ADR 0008](docs/adr/0008-hardening.md). + +--- + +## Scope & boundaries + +**Evincta is the adjudication engine, not a claimant intake portal.** Claims come from a synthetic dataset that stands in for an upstream intake pipeline — the way intake and adjudication are separate systems in a real carrier. This is deliberate: + +- The **synthetic data has ground truth**, which is what makes the held-out eval possible at all. Real user uploads have no labels to score against. +- The differentiating engineering is *after* the claim arrives — extraction, retrieval, agents, the gate. A file-upload form demonstrates none of it. +- No real PII; English + a single currency; not integrated with any real insurer; no fine-tuning. + +--- + +## Run it locally + +```bash +git clone https://github.com/AttiR/evincta.git && cd evincta + +# 1. environment +python -m venv .venv && source .venv/bin/activate +pip install -r requirements.txt +cp .env.example .env # add your keys: ANTHROPIC, VOYAGE, PINECONE (+ optional LANGFUSE) + +# 2. build the demo data + index (one-time) +python -m data.generate # synthetic claims with ground truth +python -m services.index.build # embed + upsert precedents to Pinecone + +# 3. run the API (boots one MCP host, reused across requests) +uvicorn api.main:app --reload # → http://localhost:8000 (GET /health → ok) + +# 4. run the review console +cd web && npm install && npm run dev # → http://localhost:5173 +``` + +Or the whole API in one container: + +```bash +docker build -t evincta-api . && docker run --rm -p 8000:8000 --env-file .env evincta-api +``` + +Run the eval and the gate: + +```bash +python -m eval.run_eval # scores the 18 held-out claims end-to-end +python -m eval.gate # exit 0 if every gated metric passes +``` + +--- + +## Tech stack + +| Layer | Choice | |---|---| -| Decision accuracy (weighted) | 0.722 | -| Decision accuracy (exact) | 0.722 | -| Payout in range | 0.769 (n=13) | -| Extraction — severity | 0.889 | -| Extraction — incident | 0.833 | -| Latency p50 / p95 | 14.9s / 16.6s | - -**Weighted decision accuracy** penalises dangerous errors (approving a claim -that should be denied) far more than cautious ones; weighted == exact here -means the system makes **no dangerous errors** on the eval set — when it is -wrong, it errs toward caution, not toward over-paying. - -**Scope & honesty:** Fraud recall is reported separately (0.0, n=2). The -synthetic frauds are image-reuse type; reliable detection requires perceptual -image hashing across claims — a roadmapped capability, not yet implemented. -Every recommendation is gated by human approval regardless, so a missed fraud -flag is never an automatic payout. - -Gated metrics and thresholds: [`eval/thresholds.yaml`](eval/thresholds.yaml). -Full results: [`eval/REPORT.md`](eval/REPORT.md). +| **Vision + reasoning** | Claude Sonnet (multimodal extraction + synthesis) | +| **Embeddings** | Voyage `voyage-multimodal-3` (1024-dim, image + text in one space) | +| **Vector DB** | Pinecone (serverless) | +| **Tools** | Model Context Protocol — 4 FastMCP servers + a shared host | +| **Orchestration** | LangGraph (typed state · checkpointer · `interrupt` human gate) | +| **API** | FastAPI (async · lifespan-managed MCP host) | +| **Frontend** | React + TypeScript + Tailwind | +| **Observability** | Langfuse (per-call tracing + cost) | +| **Eval / CI** | Custom harness + GitHub Actions gate | +| **Deploy** | Docker → Render (API, free) · Vercel (frontend) | +| **Language / CI** | Python 3.12 · GitHub Actions | + +--- + +## Roadmap & limitations + +- [x] **Phase 1** — Multimodal ingest (vision → typed claim record) +- [x] **Phase 2** — Multimodal vector DB + precedent retrieval +- [x] **Phase 3** — MCP servers (policy · cost · fraud · precedent) + host +- [x] **Phase 4** — Multi-agent recommender + human gate + audit +- [x] **Phase 5** — Held-out eval harness + asymmetric scoring + CI gate +- [x] **Phase 6** — FastAPI + React console · deploy · security · production hardening +- [ ] **Next** — perceptual-hash fraud detection · Postgres/Redis for horizontal scale · real-time alerting · SSO + +**Known limitations (stated plainly):** fraud recall is limited without perceptual hashing; the demo runs single-instance (SQLite + JSON index); latency is ~15s/claim (sequential vision + synthesis calls); auth is demo-grade, SSO-ready at the same seam. None of these are hidden — each has a documented path forward. + +--- + +## Repository map + +``` +evincta/ +├─ services/ +│ ├─ ingest/ # ① Claude vision → typed claim record +│ ├─ index/ # ② Voyage embeddings + Pinecone precedent search +│ ├─ mcp/ # ③ 4 FastMCP servers + the shared host +│ ├─ agents/ # ④ specialists (deterministic) + synthesiser (LLM) + graph + audit +│ └─ common/ # resilience (retry · breaker · budget) · logging +├─ api/ # FastAPI: lifecycle endpoints + human gate + auth +├─ web/ # React + Tailwind review console +├─ eval/ # ⑤ held-out eval · asymmetric metrics · thresholds · CI gate +├─ docs/adr/ # architecture decision records (0001–0008) +└─ .github/workflows/ # CI + the eval gate +``` + +
+
+ +**Built by [Atti Rehman](https://github.com/AttiR).** Traced, evaluated, guarded, cost-controlled. + +*The AI gathers and reasons. A human decides.* + +