Skip to content

Develop #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AttiR merged 3 commits into from
Jul 6, 2026
Merged

Develop #88

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6199ff9
feat(hardening): resilience, cost controls, robustness, ops
AttiR Jul 6, 2026
3a763f5
fix(concurrency): serialize shared MCP host tool calls with a lock
AttiR Jul 6, 2026
bd76dd1
Merge pull request #87 from AttiR/feat/phase-6c-concurrency
AttiR Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 84 additions & 29 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,16 @@

*Damage photos, a scanned policy, and adjuster notes go in.
An evidence-backed recommendation — with its reasoning, precedents, and audit trail — comes out.*

**The AI gathers and reasons. A human decides.**

<br>

[![Live demo](https://img.shields.io/badge/▶_Live_demo-Enter_as_demo_adjuster-7c3aed?style=for-the-badge)]([Evincta](https://evincta.vercel.app))
[![Live demo](https://img.shields.io/badge/▶_Live_demo-Enter_as_demo_adjuster-7c3aed?style=for-the-badge)](https://evincta.vercel.app)
&nbsp;
[![Eval gate](https://img.shields.io/badge/eval_gate-passing-16a34a?style=for-the-badge)](.github/workflows/eval.yml)
&nbsp;
[![Eval gate](https://img.shields.io/badge/CI-eval_gate_passing-16a34a?style=for-the-badge)](.github/workflows/eval.yml)
[![Hardening tests](https://img.shields.io/badge/hardening-tested_in_CI-16a34a?style=for-the-badge)](.github/workflows/ci.yml)

<br>

Expand All @@ -31,15 +34,25 @@ An evidence-backed recommendation — with its reasoning, precedents, and audit
---

> [!NOTE]
> **Live demo:** [https://evincta.vercel.app](https://evincta.vercel.app) → click **“Enter as demo adjuster.”**
> The API runs on a free-tier container that sleeps when idle, so the **first load may take ~30s to wake** — after that it's fast. Browsing and deciding claims costs nothing (recommendations are pre-computed and cached).
> **Live demo:** [evincta.vercel.app](https://evincta.vercel.app) → click **“Enter as demo adjuster.”**
> The API runs on a free-tier container that sleeps when idle, so the **first load may take ~30s to wake** — after that it's fast. Browsing and deciding claims costs nothing; recommendations are pre-computed and cached.

<div align="center">

<!-- Replace with a screenshot or GIF of the review console -->
<img src="docs/media/console.png" alt="Evincta review console — claim queue, recommendation, evidence, approve/override" width="820">
<img src="docs/media/console.png" alt="Evincta review console — claim queue, recommendation, evidence, approve/override" width="840">

*The adjuster's review console: queue, recommendation with cited precedents, evidence, and the approve / override gate.*
<sub><i>The adjuster's review console: queue, recommendation with cited precedents, evidence, and the approve / override gate.</i></sub>

</div>

---

<div align="center">

**A production-grade AI system, not a notebook demo.**
Vision · retrieval · tool-using agents · human-in-the-loop · held-out eval · deployed · hardened — end to end.

`traced` &nbsp;·&nbsp; `evaluated` &nbsp;·&nbsp; `guarded` &nbsp;·&nbsp; `cost-controlled`

</div>

Expand All @@ -48,13 +61,15 @@ An evidence-backed recommendation — with its reasoning, precedents, and audit
## Table of contents

- [Why this project](#why-this-project)
- [At a glance](#at-a-glance)
- [What it does](#what-it-does)
- [Why it's technically interesting](#why-its-technically-interesting)
- [Architecture](#architecture)
- [The agent design](#the-agent-design)
- [Evaluation](#evaluation)
- [What I actually debugged](#what-i-actually-debugged)
- [Production hardening](#production-hardening)
- [Engineering standard](#engineering-standard)
- [Scope &amp; boundaries](#scope--boundaries)
- [Run it locally](#run-it-locally)
- [Tech stack](#tech-stack)
Expand All @@ -69,7 +84,21 @@ Auto-claims triage is **20–30 minutes of multimodal context-gathering per clai

Evincta is that system. It's deliberately built as a **decision-support engine, not a classifier**: it never auto-approves a payout. It assembles the evidence, reasons over it with a panel of specialised agents, and hands a human adjuster a recommendation they can accept or override — with every step traced, evaluated, and audited.

It's the flagship of a four-project portfolio, all built to one production standard: **traced · evaluated · guarded · cost-controlled.**
It is the flagship of a four-project portfolio, all built to one standard: **traced · evaluated · guarded · cost-controlled.**

---

## At a glance

| | |
|---|---|
| **Domain** | Auto / vehicle-damage insurance claims |
| **Input** | Damage photos · scanned policy · free-text adjuster notes |
| **Output** | `approve / investigate / deny` · payout range · fraud-risk · policy basis · cited precedents · rationale |
| **Core pattern** | Deterministic tools decide facts; a single constrained LLM synthesises the decision |
| **Safety model** | Structural human-in-the-loop gate + hash-chained audit + graceful degradation to human review |
| **Proof** | 18 held-out claims · asymmetric scoring · **CI gate that fails the build on regression** |
| **Live** | Deployed on Vercel + Render (free tier), secured and rate-limited |

---

Expand All @@ -89,16 +118,16 @@ The output isn't a label — it's a **structured recommendation**: `approve / in

## Why it's technically interesting

This is the part most "AI projects" skip. Evincta is built like production software, and each choice below is a deliberate one with a trade-off behind it:
This is the part most "AI projects" skip. Evincta is built like production software, and each choice below is deliberate, with a trade-off behind it:

| | |
|---|---|
| 🧩 **Real MCP, not a metaphor** | Four actual [Model Context Protocol](https://modelcontextprotocol.io) servers (`policy · cost · fraud · precedent`) behind one host, launched as subprocesses and reused across requests. The agents call tools over a real protocol — the same primitive that lets these capabilities be swapped, versioned, or served remotely. |
| 🧩 **Real MCP, not a metaphor** | Four actual [Model Context Protocol](https://modelcontextprotocol.io) servers (`policy · cost · fraud · precedent`) behind one host, launched as subprocesses and reused across requests. The agents call tools over a real protocol — the primitive that lets these capabilities be swapped, versioned, or served remotely. |
| ⚖️ **Hybrid agents — trust placed deliberately** | The specialists are **deterministic tool-callers** (coverage, cost, fraud, precedent) — no LLM guessing where math and policy lookups belong. Exactly **one** LLM does the hard part: **synthesising** the evidence into a decision, constrained to a typed schema. LLMs synthesise; tools decide facts. |
| 🚦 **Human-in-the-loop, enforced structurally** | The graph *cannot* commit a decision. It runs to an `interrupt`, persists state via a checkpointer, and waits. The decision only exists when a human POSTs approve/override — which resumes the graph and writes the audit. It's not a convention; the machine can't skip it. |
| 📊 **A held-out eval that gates CI** | 18 claims are held out and **never indexed**. Every push runs the full pipeline against ground truth with **asymmetric scoring** (approving a should-deny claim is penalised far more than a cautious flag) and **fails the build on regression**. |
| 🔒 **Tamper-evident audit** | Every decision appends to a **hash-chained** JSONL log — each entry carries the previous entry's hash, so the record can't be silently edited after the fact. |
| 🛡️ **Hardened, honestly** | Retry + circuit breaker with **graceful degradation to human review** on failure, rate limiting, a daily spend cap, path-traversal and schema validation, structured logs, and dependency health checks. Infra-scale items (Postgres, Redis, alerting) are documented as roadmap, not faked. |
| 🛡️ **Hardened and tested** | Retry + circuit breaker with **graceful degradation to human review**, rate limiting, a daily spend cap, path-traversal and schema validation, structured logs, and dependency health checks — each **verified by tests that run in CI**. Infra-scale items are documented as roadmap, not faked. |

---

Expand All @@ -125,11 +154,11 @@ This is the part most "AI projects" skip. Evincta is built like production softw
Langfuse tracing (every call, every cost) · held-out eval + CI gate (GitHub Actions) │
```

Each layer has an [Architecture Decision Record](docs/adr/) explaining *why* it's built that way — the MCP transport choice, the agent topology, the eval design, the deploy/security posture, and the hardening line. The [Langfuse trace](docs/media/trace.png) below shows one real end-to-end run — coverage → cost → precedent → fraud → recommendation — in a single 26s span:
Every layer has an [Architecture Decision Record](docs/adr/) explaining *why* it's built that way — MCP transport, agent topology, eval design, deploy/security posture, and the hardening line. The [Langfuse trace](docs/media/trace.png) below shows one real end-to-end run — coverage → cost → precedent → fraud → recommendation — in a single 26s span:

<div align="center">
<!-- Replace with the Langfuse trace screenshot -->
<img src="docs/media/trace.png" alt="Langfuse trace of one full claim run — coverage, cost, precedent, fraud, recommendation" width="720">
<img src="docs/media/trace.png" alt="Langfuse trace of one full claim run — coverage, cost, precedent, fraud, recommendation" width="740">
<br><sub><i>One claim, fully traced — every model call and its cost, observable end to end.</i></sub>
</div>

---
Expand All @@ -142,7 +171,7 @@ Most "multi-agent" systems hand every step to an LLM and hope. Evincta doesn't.

- **Coverage** — is this incident covered? (policy exclusions, limits, deductible) → a hard fact.
- **Cost** — estimated repair range for this severity and damaged parts → a lookup, not a guess.
- **Fraud** — deterministic signals (pre-existing damage flags, inconsistencies) → rules, not vibes.
- **Fraud** — deterministic signals (pre-existing damage, inconsistencies) → rules, not vibes.
- **Precedent** — the *k* most similar past claims by multimodal similarity → retrieval.

Then **exactly one LLM call** — the **synthesiser** — takes those facts and produces the decision, forced into a typed schema (`Decision` and `FraudRisk` enums, a payout range, cited precedent IDs, a rationale). Its prompt applies rules **in priority order**: coverage is an absolute deny-gate; then fraud escalates to investigate; then below-deductible denies; else approve.
Expand Down Expand Up @@ -171,7 +200,7 @@ Evincta is scored end-to-end on **18 held-out claims** — never indexed, never

<br>

Fraud recall is reported separately and is **not gated**, deliberately. The synthetic frauds in the eval set are *image-reuse* type, whose only signal is image/incident inconsistency — which is indistinguishable, at the data level, from the noise in synthetically-paired images. Reliable detection needs **perceptual image hashing across claims**, which is roadmapped, not implemented. Gating a two-sample metric would be dishonest.
Fraud recall is reported separately and is **not gated**, deliberately. The synthetic frauds in the eval set are *image-reuse* type, whose only signal is image/incident inconsistency — indistinguishable, at the data level, from the noise in synthetically-paired images. Reliable detection needs **perceptual image hashing across claims**, which is roadmapped, not implemented. Gating a two-sample metric would be dishonest.

Two things make this an acceptable, honest position rather than a gap: the **human approval gate** means a missed fraud flag is never an automatic payout, and the limitation is named explicitly rather than hidden behind a flattering number. *Knowing what real fraud detection requires — and scoping it deliberately — is the point.*

Expand All @@ -195,20 +224,39 @@ The eval didn't just produce a number — it **caught real bugs**, and tracing t

> The throughline — bugs #3 and #4 are the *same lesson twice*: **the LLM overriding a deterministic tool.** The fix is consistent — constrain outputs to schemas, and make the prompt treat tool results as authoritative gates. That's the central discipline of building an LLM decision system, learned by hitting it, not reading it.

The git history preserves this as separate `fix:` commits with before/after numbers in the commit bodies — the debugging arc is legible in the log.
The git history preserves this as separate `fix:` commits with before/after numbers in the bodies — the debugging arc is legible in the log.

---

## Production hardening

Implemented (application-level, runs on free tier):
The live system is hardened across five areas. Application-level hardening is **implemented and verified by tests that run in CI**; infrastructure-scale items are documented as precise roadmap, not faked — the line between the two is itself a deliberate engineering decision ([ADR 0008](docs/adr/0008-hardening.md)).

- **Resilience** — retry + timeout + backoff on every external call; a **circuit breaker**; and **graceful degradation** — a dependency failure routes the claim to *human review*, never a wrong automated decision.
- **Cost &amp; abuse** — per-route **rate limiting**, a **daily spend cap** (a circuit breaker for the API bill), and request-size limits.
- **Robustness** — **path-traversal** input validation, schema-validated model output, and prompt-injection *defense-in-depth* (delimited untrusted input · constrained output enum · human gate).
- **Observability** — **structured JSON logging** and liveness + **dependency-checking readiness** probes.
| Area | Implemented | Verified by |
|---|---|---|
| **Resilience** | Retry + timeout + backoff on every external call; circuit breaker; **graceful degradation** — a dependency failure routes the claim to *human review*, never a wrong automated decision | Unit tests (breaker opens/half-opens; forced failure → `investigate`) |
| **Cost &amp; abuse** | Per-route **rate limiting**; a **daily spend cap** (a circuit breaker for the API bill); request-size limits | `429` on limit breach; cap refuses spend over ceiling; `413` on oversized body |
| **Robustness** | **Path-traversal** input validation; schema-validated model output; prompt-injection *defense-in-depth* (delimited untrusted input · constrained output enum · human gate) | `422` on traversal input; malformed output raises |
| **Observability** | **Structured JSON logging** (no secrets/PII); liveness + **dependency-checking readiness** probes | `/health/ready` reports per-dependency status |
| **Concurrency** | Shared-host tool calls serialised under a lock | Documented Postgres/Redis seam for multi-instance scale |

Documented as roadmap (needs infrastructure beyond the free demo): Postgres for the claim index + LangGraph checkpointer and Redis for the limiter/spend counter (multi-instance scale); real-time alerting on top of the existing Langfuse tracing. Prompt injection is *mitigated, not claimed solved.* Rationale in [ADR 0008](docs/adr/0008-hardening.md).
> **The safest failure mode for a decision system:** on any partial failure, Evincta escalates to a human rather than deciding on incomplete evidence. A crash is bad; a confident wrong decision on missing data is worse. The circuit breaker degrades *into the human gate* — reusing the safety mechanism the whole system is built around.

**Roadmap (needs infrastructure beyond the free demo):** Postgres for the claim index + LangGraph checkpointer and Redis for the limiter/spend counter (multi-instance scale); real-time alerting on top of the existing Langfuse tracing. Prompt injection is *mitigated via defense-in-depth, not claimed solved.*

---

## Engineering standard

- **Traceable** — every model call and its cost is observable in Langfuse; one full trace is shown above.
- **Evaluated** — a held-out set, asymmetric scoring aligned to real risk, and a CI gate that blocks regressions.
- **Guarded** — resilience, rate limits, spend caps, and input/output validation, each with a test.
- **Cost-controlled** — token spend is estimated per call and hard-capped; the demo is cached to cost nothing to view.
- **Decision-recorded** — eight [ADRs](docs/adr/) capture *why* each layer is built the way it is.
- **Legible history** — small, focused commits; the debugging arc is readable in the log.
- **Honest** — limitations (fraud recall, single-instance state, demo auth) are stated plainly, with a documented path forward for each.

*The through-line: production-minded AI engineering — building systems that are observable, measurable, safe to fail, and honest about their edges.*

---

Expand Down Expand Up @@ -249,11 +297,12 @@ Or the whole API in one container:
docker build -t evincta-api . && docker run --rm -p 8000:8000 --env-file .env evincta-api
```

Run the eval and the gate:
Run the eval, the gate, and the hardening tests:

```bash
python -m eval.run_eval # scores the 18 held-out claims end-to-end
python -m eval.gate # exit 0 if every gated metric passes
pytest tests/ -v # eval + hardening tests (breaker, degradation, validation)
```

---
Expand All @@ -270,7 +319,8 @@ python -m eval.gate # exit 0 if every gated metric passes
| **API** | FastAPI (async · lifespan-managed MCP host) |
| **Frontend** | React + TypeScript + Tailwind |
| **Observability** | Langfuse (per-call tracing + cost) |
| **Eval / CI** | Custom harness + GitHub Actions gate |
| **Resilience** | tenacity (retry/backoff) · custom circuit breaker · slowapi (rate limit) |
| **Eval / CI** | Custom harness · pytest · GitHub Actions gate |
| **Deploy** | Docker → Render (API, free) · Vercel (frontend) |
| **Language / CI** | Python 3.12 · GitHub Actions |

Expand All @@ -284,9 +334,9 @@ python -m eval.gate # exit 0 if every gated metric passes
- [x] **Phase 4** — Multi-agent recommender + human gate + audit
- [x] **Phase 5** — Held-out eval harness + asymmetric scoring + CI gate
- [x] **Phase 6** — FastAPI + React console · deploy · security · production hardening
- [x] **Next** — perceptual-hash fraud detection · Postgres/Redis for horizontal scale · real-time alerting · SSO
- [ ] **Next** — perceptual-hash fraud detection · Postgres + Redis for horizontal scale · real-time alerting · SSO

**Known limitations (stated plainly):** fraud recall is limited without perceptual hashing; the demo runs single-instance (SQLite + JSON index); latency is ~15s/claim (sequential vision + synthesis calls); auth is demo-grade, SSO-ready at the same seam.
**Known limitations (stated plainly):** fraud recall is limited without perceptual hashing; the demo runs single-instance (SQLite + JSON index); latency is ~15s/claim (sequential vision + synthesis calls); auth is demo-grade, SSO-ready at the same seam. None are hidden — each has a documented path forward.

---

Expand All @@ -303,14 +353,19 @@ evincta/
├─ api/ # FastAPI: lifecycle endpoints + human gate + auth
├─ web/ # React + Tailwind review console
├─ eval/ # ⑤ held-out eval · asymmetric metrics · thresholds · CI gate
├─ tests/ # eval + hardening tests (run in CI)
├─ docs/adr/ # architecture decision records (0001–0008)
└─ .github/workflows/ # CI + the eval gate
└─ .github/workflows/ # CI · eval gate · hardening tests
```

---

<div align="center">
<br>

**Built by [Atti Rehman](https://github.com/AttiR).** Traced, evaluated, guarded, cost-controlled.
**Built by [Atti Rehman](https://github.com/AttiR)**

Traced · evaluated · guarded · cost-controlled.

*The AI gathers and reasons. A human decides.*

Expand Down
Loading
Loading