Conversation
Build the data half of Phase 1: a reproducible generator that turns real severity-labelled vehicle images into complete multimodal claims for development and evaluation. - make_manifest.py: flattens the train/val dataset into one severity-balanced pool (40 minor / 40 moderate / 40 severe) and writes data/raw/manifest.csv - policies.py: synthetic auto policies rendered to PDF (the document the extractor reads), with deductibles, limits, and exclusions - notes.py: seeded, deliberately messy adjuster notes (abbreviations, missing fields) — deterministic so the dataset is reproducible - labels.py: deterministic ground-truth decisions derived from severity + policy (exclusion -> deny, fraud -> investigate, total-loss ratio -> capped payout); three-severity model, no 'total' - build.py: assembles 80 claims, plants a 15% fraud subset (image_reuse / incident_mismatch / pre_existing), writes the index/eval split, seed=42 Decision distribution across 80 claims: 57 approve / 12 deny / 11 investigate — all three paths represented for a meaningful eval. Bulk images and generated claims are gitignored; two sample claims committed under data/samples/ for tests.
Feat/phase 1 data generator
- split multi-import and semicolon/colon one-liners (E401/E701/E702) - remove unused imports: random (labels), os (embed) - drop duplicate/unused Langfuse import (extractor)
- add pyproject.toml: pythonpath=["."] so `services` imports, and skip integration tests by default via the `integration` marker - add requirements.txt with runtime deps - install requirements in CI before running ruff/pytest
- add requirements.txt with runtime deps - add pyproject.toml: pythonpath=["."] and skip integration tests by default - CI installs requirements before ruff/pytest - resolve ruff E401/E701/E702/F401/F811 across data generator and services
feat(index): multimodal embeddings + pinecone store
Embed each index-split claim as image + text vectors (voyage-multimodal-3, 1024-d) and upsert to Pinecone. Eval-split claims are excluded as a leakage guard. Prints claim/vector counts and index stats.
feat(index): build Pinecone index from generated claims
Query image+text vectors, fuse by best score per claim, return top-k with outcome metadata + matched_on. Leakage guard verified: eval claims never indexed, never retrieve themselves. Embedding calls traced. Closes #N, #N
Constructing the Voyage/Langfuse clients at import time made `from services.index.embed import DIM` require VOYAGE_API_KEY, breaking pytest collection in CI. Build clients lazily on first embed instead. Also split multi-import in test_index.py (ruff E401).
feat(index): dual-vector precedent retrieval + leakage tests
Wrap multimodal precedent retrieval as a FastMCP stdio tool (find_precedents_tool) returning JSON-safe decision/payout metadata. Bootstrap repo root onto sys.path and declare deps so runs the standalone file; strip tool inputs for robustness. Lazy-init the Pinecone client so the module imports without credentials.
Three FastMCP stdio tools for claim decisioning: - policy.check_coverage: coverage decision from exclusions + terms - cost.estimate_repair: repair-cost range from severity and parts - fraud.fraud_signals: risk score from extraction signals + precedent spread
feat(mcp): add policy, cost, and fraud MCP servers
Four FastMCP servers over stdio (precedent/policy/cost/fraud). Host launches + discovers tools at runtime, exposes them to Claude, routes calls, traces the turn. precedent-server wraps Phase-2 retrieval. Closes #20
Add free tests for policy/cost/fraud tools and pin mcp==1.28.1 so CI can import the servers. Also split a semicolon statement in demo (ruff E702).
feat(mcp): host discovers 4 servers, Claude-driven tool calls
- state.py: ClaimState TypedDict shared across the graph (inputs, per-specialist outputs, recommendation, human gate) - tools.py: call_tool bridges a node to the MCP host for one tool call - specialists.py: coverage/cost/precedent/fraud nodes that each fill their slice of state
feat(agents): claim state, MCP tool bridge, and specialist nodes
Claude reasons over coverage/cost/fraud/precedent findings via a forced tool schema and returns a Pydantic-validated Recommendation (decision, payout range, fraud_risk, confidence, rationale, cited precedent claim_ids, policy_basis). Traced as agents.synthesise; loads .env so the clients authenticate. Adds scripts/try_synthesiser.py to exercise it.
feat(agents): synthesiser node producing a validated Recommendation
Specialist tool-callers (coverage/cost/fraud/precedent) over the MCP host feed an LLM synthesiser that returns a typed Recommendation. Graph pauses at a human-approval interrupt; only after approval is a tamper-evident audit entry written. Synthesiser traced; cost = vision+embeddings+1 call. Closes #27
feat(agents): LangGraph recommender + human gate + audit
Minor claims produced $0 'approvals' when repair fell below the deductible — incoherent ground truth that would skew the Phase-5 eval. Fix: payout <= 0 now denies (reason 'below_deductible'); deductibles lowered to [100,250,500]. Regenerated data + rebuilt index. Distribution after fix: 56 approve / 13 deny / 11 investigate; minor approvals now $100-$1,205.
fix(data): deny below-deductible claims instead of approving for $0
- eval/dataset.py: load split=='eval' claims and assert_integrity() (all three decisions present, no $0 approvals, and no eval claim leaked into the index) - stop tracking data/evincta.sqlite (runtime checkpoint store) and ignore it
feat(eval): eval-set loader with integrity guards; ignore runtime db
Cost-aware decision accuracy (exact + weighted, penalising dangerous approve-when-deny/investigate errors most), fraud recall, payout-in-range for approvals, and Phase-1 extraction accuracy on severity/incident.
feat(eval): scoring metrics for the claims pipeline
The eval surfaced the synthesiser over-investigating (17/18 claims). Root causes found by tracing the held-out set: 1. fraud_risk was a free 'str' field — the model wrote prose instead of the tool's categorical risk. Constrained to a FraudRisk enum. 2. The synthesiser approved excluded claims, ignoring the coverage tool. The prompt now treats coverage.covered=false as an absolute deny gate, applied in priority order (coverage > fraud > deductible > approve). 3. fraud_server flagged 'payout_outlier_vs_precedent' on normal payout variance (and on $0 deny precedents). Removed; not a fraud signal. 4. image_inconsistency was decoupled from fraud risk. It's a data/extraction signal (flows to extraction_confidence/needs_human), not fraud evidence. Result on the held-out evaluation: weighted decision accuracy 0.722 with zero dangerous (approve-when-deny) errors; payout-in-range 0.769. feat(eval): add held-out evaluation, asymmetric scoring, and CI gate Run the 18 eval-split claims through the full Phase-4 graph and score: - Decision (asymmetric/weighted) - Fraud recall - Payout-in-range - Extraction accuracy The integrity gate rejects unsound data (leakage or $0 approvals). Operational metrics (cost, p50/p95 latency, escalation rate) are reported. thresholds.yaml now gates the build, and CI fails on regression. Closes #35
- fix ruff E401/E701/E702/F401 across eval gate/report/run_eval - pin langgraph + langgraph-checkpoint-sqlite + pyyaml in requirements - eval workflow gates the committed eval/results.json against thresholds instead of running the full (data- and API-dependent) eval in CI
fix(agents): correct synthesiser decision logic
- api/main.py: FastAPI app whose lifespan boots one MCP host (reused across requests) and a /health endpoint; dev CORS for the UI - api/store.py: simple lock-guarded JSON claim store (upsert/get/all) - tools.py: reuse a shared MCP host when set by the API, else fall back to a throwaway host for scripts/eval
feat(api): FastAPI app with shared MCP host + claim store
- claims.py: POST /claims (extract → graph → recommendation),
GET /claims, GET /claims/{id}, POST /claims/{id}/decision (human gate)
- main.py: mount claims router; register shared MCP host with its event
loop so threadpool tool calls run on the host's loop (fixes deadlock)
- tools.py: drive shared-host calls via run_coroutine_threadsafe
- ignore runtime artifacts (claims store + audit log)
feat(api): claim lifecycle endpoints + shared-host fix
FastAPI service holds one long-lived MCP host (fixes per-call spin-up); endpoints expose the claim lifecycle — submit, list, detail, decide. The decision endpoint resumes the graph at the human gate (approve or override) and writes the audit. React+Tailwind console: claim queue, review detail (recommendation + evidence + cited precedents), and the approve/override action. Container-ready for 6b deploy. Closes #N, #N, #N
feat(api,ui): claim review console over the decision graph
One image that boots the app + its MCP servers
feat: Containerize the API
A dependency that rejects un-keyed write requests. Adds demo login for now
feat: Protect the expensive routes
Pre-compute and cache recommendation claims during demo setup so recruiter browsing never triggers live LLM calls. - Pre-seed recommendation claims in the claim index - Serve cached recommendations instead of invoking the model - Eliminate per-view AI costs during the demo - Prevent API limit issues during recruiter browsing - Add reset flow to refresh demo state for the next visitor
Remove duplicate json import, unused Path import, and split semicolon-separated statements for E702.
feat(demo): pre-seed cached recommendations for zero-cost browsing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.