extraction-pipeline

A benchmarked invoice/document extraction pipeline: the part most demos skip. Anyone can prompt a vision model over five clean PDFs. This repo is about the other 4,995: rotated scans, European decimal commas, OCR that turns 0 into O, three different words for "total", and a customer's phone number you must not ship to an LLM. It ingests PDFs / images / photos, redacts PII before any model call, does layout-aware extraction into a strict Pydantic schema, scores every field, runs validation rules (line items sum to subtotal, tax math, currency/date normalization), routes low-confidence fields to a human review queue, and feeds corrections back as few-shot exemplars. It ships a benchmark harness that scores invoice extraction accuracy field-by-field against a naive single-prompt baseline on public datasets, plus a synthetic dataset so the whole thing (including the numbers below) runs offline with zero API keys.

Why it converts

The model backend is pluggable: an anthropic backend for production, and a deterministic mock backend that reads the layout the same way an LLM would, so the entire system (and its benchmark) runs with no network. The benchmark's job is to show the delta the engineering buys you: same reading ability, but the engineered pipeline (layout association, validation, normalization, confidence routing) versus a naive "flatten the text and grab the first number after total" baseline.

Architecture

flowchart TD
    A[PDF / image / photo / .json sidecar] --> B[ingest<br/>pdfplumber · EXIF · deskew · optional Tesseract OCR]
    B --> C[PII redaction<br/>emails · phones · cards Luhn · SSN/SIN · IBAN]
    C -->|redacted tokens| D[layout-aware extraction<br/>mock backend or Anthropic backend]
    D --> E[restore PII placeholders]
    E --> F[normalize<br/>currency · dates · money]
    F --> G[validate + repair<br/>line-items sum · tax math · date sanity]
    G --> H[field-level confidence scoring]
    H -->|high confidence| I[auto-accepted record]
    H -->|low confidence| J[review queue - local web UI]
    J --> K[human correction]
    K --> L[few-shot exemplar store]
    L -.re-injected into future prompts.-> D
    subgraph Benchmark
      M[synthetic / SROIE / CORD / DocILE] --> N[pipeline vs naive baseline]
      N --> O[per-field P/R/F1]
      O --> P[eval dashboard · failure gallery]
    end

Loading

Everything is a small, testable unit: ingest returns a Document of tokens-with-boxes; redaction is a pure text/token transform with a restore map; the backend is an interface (extract(document, exemplars) -> InvoiceExtraction); normalization/validation/confidence operate on plain records. Swapping mock for anthropic changes one flag.

Results: SYNTHETIC dataset, MOCK-MODE backend

Read the label. These numbers are produced entirely offline by a deterministic synthetic dataset scored by the deterministic mock backend. They prove the harness and the pipeline-vs-baseline delta are real and reproducible. They are not a claim of real-world accuracy. The real-dataset + real-model run is the human TODO below. Full artifacts: results/synthetic_mock/ (metrics.md, dashboard.html, gallery.html, results.json).

60 documents, OCR-style token noise at 0.05/token, seed=7:

Field	Pipeline P	Pipeline R	Pipeline F1	Naive baseline F1	ΔF1
vendor_name	0.883	0.883	0.883	0.017	+0.867
invoice_number	0.932	0.917	0.924	0.050	+0.874
invoice_date	0.906	0.800	0.850	0.267	+0.583
due_date	0.965	0.917	0.940	0.276	+0.664
currency	1.000	1.000	1.000	0.717	+0.283
subtotal	0.883	0.883	0.883	0.632	+0.252
tax	0.932	0.917	0.924	0.000	+0.924
total	0.883	0.883	0.883	0.033	+0.850
line_items	0.839	0.823	0.831	0.000	+0.831
macro			0.902	0.221	+0.681

Where the naive baseline bleeds, and why the pipeline doesn't:

• tax sits at 0.000 because the baseline grabs the first number after the word "tax", which is the 13% rate, not the amount. The pipeline associates the label with the value on the same row.
• total comes in at 0.033: "Subtotal" contains the substring "total", so the baseline's keyword search returns the subtotal. The pipeline reads the labelled total (and cross-checks subtotal + tax = total).
• The baseline never parses the table at all, which is why line_items scores 0.000. The pipeline detects the header row, infers columns, and reconstructs items.
• For dates and currency, the baseline keeps raw strings and misreads C$ as USD. The pipeline normalizes to ISO-8601 and ISO-4217.

Reproduce exactly (pinned deps, deterministic):

extract-pipeline generate  --out data/synthetic --count 60 --seed 7 --noise 0.05
extract-pipeline benchmark --dataset synthetic --data data/synthetic --backend mock \
                           --out results/synthetic_mock

Quickstart

uv venv --python 3.12
uv pip install -e .

# 1. Make a synthetic dataset (images + OCR token sidecars + ground truth)
extract-pipeline generate --out data/synthetic --count 60

# 2. Extract one document (mock backend, fully offline)
extract-pipeline extract examples/sidecars/synthetic-0000.json
extract-pipeline extract examples/sidecars/synthetic-0000.json --json

# 3. Benchmark pipeline vs naive baseline; writes results.json + dashboard.html + gallery.html
extract-pipeline benchmark --dataset synthetic --data data/synthetic --backend mock --out benchmark_out

# 4. Drag-drop demo (open http://127.0.0.1:8001, drop a PDF / image / .json sidecar)
extract-pipeline demo

# 5. Human-in-the-loop review loop
extract-pipeline enqueue examples/sidecars/synthetic-0001.json --queue review_queue.jsonl
extract-pipeline review --queue review_queue.jsonl --exemplars exemplars.jsonl   # http://127.0.0.1:8000

Sample invoices for the demo live in examples/.

Running the tests

uv pip install -e ".[dev]"
pytest

The suite (69 tests, runs in ~1s) is fully offline: no network, no API keys, no Tesseract. It fakes nothing about the pipeline: the mock backend is a real heuristic extractor, and the benchmark test asserts the pipeline beats the baseline.

OCR is an optional extra that degrades gracefully

This machine has no Tesseract, so OCR is opt-in:

uv pip install -e ".[ocr]"   # then install the binary, e.g. `brew install tesseract`

Without it, PDFs with a text layer and images that carry a token sidecar (foo.png + foo.json) still work end to end. A raw photo with no sidecar and no Tesseract raises a clear OcrUnavailableError telling you how to enable OCR. It never silently returns empty results. The synthetic dataset ships sidecars precisely so the benchmark needs no OCR engine.

Run it for real (human TODO)

The mock-mode numbers above are the floor, not the pitch. To produce real-dataset, real-model metrics:

1. Set your key (test/prod is your call; the pipeline redacts PII before every call regardless):

   export ANTHROPIC_API_KEY=sk-ant-...
   uv pip install -e ".[anthropic]"

2. Fetch a public dataset (adapters included; scripts land data in data/<name>/):

   bash scripts/download_sroie.sh     # receipts: company/date/address/total
   bash scripts/download_cord.sh      # receipts: menu line items + totals
   bash scripts/download_docile.sh    # licence-gated invoice sample (see script)

3. Benchmark against a live model:

   extract-pipeline benchmark --dataset sroie --data data/sroie/ICDAR-2019-SROIE/data \
                              --backend anthropic --out results/sroie_real

   The default model is claude-opus-4-8. Real datasets label a subset of fields (e.g. SROIE has no line items; CORD has no vendor). Only labelled fields are scored, and the dashboard states which.

Legal & privacy guardrails

• PII redaction happens before the model call, not after. Emails, phone numbers, Luhn-valid card numbers, SSN/SIN, and IBANs are replaced with placeholders in the tokens the backend sees; a redaction map restores them locally afterward. The layout/text backend is the path that can be redacted: a reason to prefer it over dropping raw images at a vision model.
• Redact-minimally, retain-minimally: no PII is written to the review store or exemplars beyond the fields being corrected.
• The anthropic backend is off by default and client-deployed with the client's own key; this repo never bundles or requires one.
• Metrics in this README were produced by code in this repo (extract-pipeline benchmark) and are reproducible with the commands shown. No number here is hand-written.

Hire me

I make AI-era and money-critical code production-safe: extraction pipelines that hold up past the demo, with redaction, validation, and confidence built in. Custom extraction & benchmarking → https://amin-ale.github.io/portfolio-site/offer-extraction.html · amin.ale.business@gmail.com

Further reading

Why these numbers matter more than the demo: Why your extraction demo fails at 5,000 documents, a data-backed walk through this pipeline's failure taxonomy, validation rules, and confidence design, using the synthetic mock-mode benchmark above.

License

MIT. See LICENSE.