Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,9 +284,9 @@ python -m eval.gate # exit 0 if every gated metric passes
- [x] **Phase 4** — Multi-agent recommender + human gate + audit
- [x] **Phase 5** — Held-out eval harness + asymmetric scoring + CI gate
- [x] **Phase 6** — FastAPI + React console · deploy · security · production hardening
- [ ] **Next** — perceptual-hash fraud detection · Postgres/Redis for horizontal scale · real-time alerting · SSO
- [x] **Next** — perceptual-hash fraud detection · Postgres/Redis for horizontal scale · real-time alerting · SSO

**Known limitations (stated plainly):** fraud recall is limited without perceptual hashing; the demo runs single-instance (SQLite + JSON index); latency is ~15s/claim (sequential vision + synthesis calls); auth is demo-grade, SSO-ready at the same seam. None of these are hidden — each has a documented path forward.
**Known limitations (stated plainly):** fraud recall is limited without perceptual hashing; the demo runs single-instance (SQLite + JSON index); latency is ~15s/claim (sequential vision + synthesis calls); auth is demo-grade, SSO-ready at the same seam.

---

Expand Down
31 changes: 31 additions & 0 deletions docs/adr/0005-eval-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# ADR 0005 — Evaluation design and CI gate

## Status
Accepted

## Context
Evincta produces claim recommendations; we need a defensible measure of
quality and a guard against regression.

## Decision
- Score the 18 eval-split claims (held out, never indexed) end-to-end
through the full agent graph — not the synthesiser alone.
- Quality metrics: decision accuracy (asymmetric/weighted — approving a
deny-claim penalised far more than a cautious investigate), fraud recall,
payout-in-range, extraction accuracy.
- Operational metrics (cost, p50/p95 latency, escalation) reported, not gated.
- An integrity gate refuses to run on unsound data (leakage, $0-approves).
- thresholds.yaml gates the build on risk-weighted accuracy; CI fails on
regression. Thresholds calibrated from a baseline run, set below observed.

## Consequences
+ The number reflects what ships (whole pipeline), and a regression blocks merge.
+ Asymmetric scoring aligns the metric with real claims risk.
+ Pure-function metrics are unit-tested free; the paid run produces results once.
- Small eval set (n=18; 2 frauds): thresholds are floors, not precision targets.
- Fraud recall not gated — image-reuse detection needs perceptual hashing
(roadmapped); the human approval gate backstops missed flags.

## Alternatives rejected
- Synthesiser-only eval: cheaper but doesn't test the pipeline that ships.
- Gating fraud recall: a 2-sample metric is too noisy to gate honestly.
56 changes: 56 additions & 0 deletions docs/adr/0006-api-and-ui.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# ADR 0006 — API and review console

## Status
Accepted

## Context
The decision engine (Phases 1–5) runs from scripts. It needs an HTTP
surface and a human interface so a claim can be submitted, reviewed, and
decided by an adjuster — driving the Phase-4 human gate over the network.

## Decision

**A thin API over the graph.** FastAPI exposes the claim lifecycle:
`POST /claims` (submit → run to the interrupt → return the
recommendation), `GET /claims` and `GET /claims/{id}` (queue + detail),
and `POST /claims/{id}/decision` (approve/override → resume the graph →
write the audit). The API imports the existing `services.agents.run`
functions; no decision logic is reimplemented in the API layer.

**One long-lived MCP host.** The app boots a single MCP host in FastAPI's
`lifespan` and reuses it across all requests, replacing the per-call host
spin-up that dominated latency. Sync graph code runs off the event loop
via `run_in_threadpool` to avoid nested event loops.

**A thin claim index.** `api/store.py` maps claim → thread_id → status
for the queue/detail views. Authoritative state (graph checkpoints, audit
log) already lives in durable stores from Phase 4, so the index is a
lookup table only (JSON for the demo; a Postgres seam noted for scale).

**List vs detail payloads** are shaped separately: compact rows for the
queue, the full object for the review screen.

**Review console.** React + Tailwind, deliberately restrained (a
system-of-record aesthetic): a claim queue, a detail pane showing the
recommendation with cited precedents and the evidence it reasoned from,
and an approve / override action. Color encodes only the three decision
states. Cited precedents are surfaced so the adjuster can verify the
reasoning rather than trust a black box.

## Consequences
+ The human gate becomes an HTTP contract; the graph still cannot commit a
decision without the decision POST. Override lets the human reject the
AI's call, recorded in the audit.
+ Reusing the host makes the API fast; the env-var API base lets the same
frontend hit localhost or the deployed API by config alone.
+ The API is a thin skin over tested graph code — low duplication, low risk.
- The JSON index is single-writer (fine for one instance); Postgres is the
documented path for concurrency (ADR 0008).
- The console is review-only by design; claimant intake is out of scope
(see the README scope section).

## Alternatives rejected
- Reimplementing decision logic in the API: duplicates tested code.
- Per-request MCP host: the slow pattern; the lifespan host replaces it.
- An in-UI claim-submit/upload control: conflates the adjuster role with
claimant intake, which is deliberately out of scope.
Loading