Skip to content

docs: correct corpus and CI claims with evidence #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ChunkyTortoise merged 1 commit into from
Sep 22, 2026
Merged

docs: correct corpus and CI claims with evidence #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 14 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
# DocExtract AI

Fixture-backed document intelligence with two-pass extraction, agentic retrieval, and an eval gate.

<p align="center">
<img src="./docs/assets/eval-proof.svg" width="720" alt="DocExtract deterministic evaluation proof: 95.5% field-level score from a 28-fixture replay, two extraction passes, an eval gate, and a separately labeled 202-case authoring corpus." />
</p>
Fixture-backed document intelligence with two-pass extraction, agentic retrieval, and an offline CI evaluation check.

## Deterministic eval replay

Expand All @@ -13,15 +9,15 @@ Fixture-backed document intelligence with two-pass extraction, agentic retrieval
| Evidence | What it is | What it is not |
|----------|------------|----------------|
| **28 fixtures** | Deterministic replay behind the 95.5% field-level score (`scripts/eval_offline_replay.py`, `autoresearch/baseline.json`) | Not the authoring-corpus size |
| **202 cases** | Authoring corpus (151 golden + 51 adversarial) | Not the replay fixture total and not the score population |
| **200 cases** | Authoring corpus: 150 golden + 50 adversarial cases, stored as 202 JSONL lines including two metadata rows | Not the replay fixture total and not the score population |

How the 28-fixture replay is scored, and how it relates to the 202-case authoring corpus: [docs/eval-boundary.md](docs/eval-boundary.md). Held-out live eval (protocol only; performance unmeasured): [docs/held-out-live-eval-protocol.md](docs/held-out-live-eval-protocol.md).
The verified replay scored 28 committed prediction fixtures against 72 lookup cases, with 44 fixtures pending. Its weighted field-level accuracy is 0.9555 (95.5% rounded), not F1 or live-model performance. Retrieval recall, support and abstention remain unmeasured. See [retrieval and extraction evidence](docs/retrieval-extraction-evidence.md) for the score, populations and limitations. Held-out live eval (protocol only; performance unmeasured): [docs/held-out-live-eval-protocol.md](docs/held-out-live-eval-protocol.md).

### Reviewer path

1. Replay the committed fixtures: `python scripts/eval_offline_replay.py --floor 0.85`
2. Compare the replayed field-level score to **95.5%**
3. Inspect the two extraction passes and the deterministic eval gate on the proof card
3. Inspect the [two extraction passes](docs/adr/0003-two-pass-extraction.md) and the [offline CI evidence](docs/retrieval-extraction-evidence.md)
4. Continue to retrieval, architecture, and the honest scope notes below
5. Run the local fixture-backed UI: `DEMO_MODE=true streamlit run frontend/app.py` — [DEMO.md](DEMO.md)

Expand All @@ -33,7 +29,7 @@ The hosted Streamlit URL is intentionally omitted until anonymous access is veri

## Eval gate {#eval-gate}

Prompts are code. DocExtract treats extraction quality as a **merge-blocking CI signal**, not a post-hoc dashboard number.
DocExtract reports extraction quality through passing or failing CI checks. Successful checks do not establish enforced merge protection. The recorded 2026-09-19 repository audit returned `Branch not protected` and an empty branch-rules list; merge blocking was not enforced in that observation. See [CI and merge enforcement](docs/retrieval-extraction-evidence.md#ci-and-merge-enforcement).

| Signal | What runs | When |
|--------|-----------|------|
Expand All @@ -43,28 +39,28 @@ Prompts are code. DocExtract treats extraction quality as a **merge-blocking CI
| **Held-out live protocol** | Public or synthetic docs, untouched test partition, `score_extraction` | Unmeasured until a funded run is logged ([protocol](docs/held-out-live-eval-protocol.md)) |
| **Drift cron** | Golden set vs production prompt version | Daily 13:23 UTC |

**Eval gate proof (red blocked PR):** [#32 — intentional regression (keep open / expect red)](https://github.com/ChunkyTortoise/docextract/pull/32). Executed vs replayed stages: [docs/eval-gate-proof.md](docs/eval-gate-proof.md). See also [docs/eval-methodology.md](docs/eval-methodology.md).
**Failing CI check demonstration:** [#32, intentional regression (keep open / expect red)](https://github.com/ChunkyTortoise/docextract/pull/32). Executed vs replayed stages: [docs/eval-gate-proof.md](docs/eval-gate-proof.md). See also [docs/eval-methodology.md](docs/eval-methodology.md).

| Metric | Value | Basis |
|--------|-------|-------|
| Extraction accuracy (field-level, critical fields weighted 2×) | **95.5%** | Always-on CI offline replay of **28** deterministic fixtures (`scripts/eval_offline_replay.py`); not a paid live grade |
| Test suite | **80% CI coverage gate** | `--cov-fail-under=80`; the changing collected-test total is intentionally omitted ([portfolio-metrics.yaml](docs/portfolio-metrics.yaml)) |
| Authoring corpus | **202 cases** (151 golden + 51 adversarial) | `evals/golden_set.jsonl` + `evals/adversarial_set.jsonl` (line counts); separate from the 28-fixture offline replay |
| Authoring corpus | **200 cases** (150 golden + 50 adversarial) | `evals/golden_set.jsonl` + `evals/adversarial_set.jsonl`: 202 lines including two metadata rows; separate from the 28-fixture offline replay |
| Cost / latency | See [cost-model.md](docs/cost-model.md) | Modeled only until a funded `scripts/benchmark.py` run is committed |

<details>
<summary>CI-replayed eval breakdown by document type (from committed <code>autoresearch/baseline.json</code>)</summary>
<summary>Verified offline replay by document type (RA11, 2026-09-19; weighted field-level accuracy)</summary>

| Document type | Score | Cases |
|---|---|---|
| invoice | 0.9731 | 13 |
| receipt | 0.9107 | 4 |
| purchase_order | 0.9762 | 3 |
| bank_statement | 0.9581 | 4 |
| invoice | 0.9669 | 13 |
| receipt | 0.9091 | 4 |
| purchase_order | 0.9745 | 3 |
| bank_statement | 0.9613 | 4 |
| medical_record | 0.9923 | 3 |
| identity_document | 0.8139 | 1 |

Overall: 0.955 across 28 cases, replayed on every eval-gated PR at zero API cost.
Overall: 0.9555 across 28 committed prediction fixtures, with 44 of 72 lookup cases pending. The historical baseline comparison score is 0.95546. This replay uses no API calls.

</details>

Expand All @@ -84,7 +80,7 @@ Upload → ARQ worker → classify → extract → validate → embed → search

## Why this is interesting (engineering)

- **Eval-gated CI**: `eval-gate.yml` offline job replays 28-case deterministic baseline at zero API cost; PRs touching prompts or extraction services must pass before merge
- **Offline evaluation in CI**: `eval-gate.yml` replays 28 committed prediction fixtures at zero API cost and reports check status; the recorded repository audit did not show enforced merge protection
- **FastAPI & Strict Type Safety**: End-to-end Pydantic V2 validation contracts, typed error domains, and deterministic schema enforcement preventing malformed extraction persistence
- **PostgreSQL (pgvector) & ARQ Queue**: Document chunk embeddings indexed via pgvector HNSW vectors, decoupled background document processing via Redis and ARQ worker queue
- **Agentic RAG**: ReAct Think → Act → Observe over hybrid retrieval tools; primary search story in API and Streamlit ([`agentic_rag.py`](app/services/agentic_rag.py), [`agent_trace.py`](frontend/pages/agent_trace.py))
Expand Down
65 changes: 65 additions & 0 deletions docs/retrieval-extraction-evidence.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Retrieval and Extraction Evidence

This note keeps extraction replay evidence separate from retrieval evidence. It follows the boundaries documented in [Eval Boundary](eval-boundary.md) and [Eval Gate Proof](eval-gate-proof.md).

## Extraction replay

The offline extraction replay was run from detached commit `7b1a515303cff036a8a16670aad39dc10b5b7b7a` with this exact command:

```sh
cd /Users/cave/Projects/clients/.worktrees/docextract-hf-ra11 || exit 1
env -i PATH=/usr/bin:/bin PYTHONDONTWRITEBYTECODE=1 /Users/cave/Projects/clients/docextract/.venv/bin/python -S scripts/eval_offline_replay.py --floor 0.85 --out /tmp/hero-hf-ra11-offline-replay.json
```

The command exited with status 0. It replayed 28 of 72 lookup cases and reported:

| Result | Value |
|---|---:|
| Combined weighted field-level extraction score | 0.9555 |
| Golden fixture score | 0.9264 |
| Adversarial fixture score | 1.0 |
| Baseline score | 0.95546 |
| Floor | 0.85 |
| Pending fixtures | 44 |

The JSON output artifact is `/tmp/hero-hf-ra11-offline-replay.json`, SHA-256 `16c267869038d8aaa90febb474d34ab5a78d0eca94fe7514103e523dfa8f39ed`, 958 bytes. The committed comparison artifact is [autoresearch/baseline.json](../autoresearch/baseline.json), with `case_count` 28 and `overall_score` 0.95546.

The extraction score is weighted field-level accuracy over committed prediction fixtures. Legacy output field names contain `f1`, but this result is not F1. It is also not a live model result.

## Three separate denominators

These populations have different roles and must not be summed or substituted for one another.

| Population | Count | Role |
|---|---:|---|
| Committed prediction fixtures | 28 | Predictions scored by the offline extraction replay, matching `baseline.json` case count 28 |
| Lookup cases in [autoresearch/eval_dataset_72.json](../autoresearch/eval_dataset_72.json) | 72 | Expected outputs and lookup records, split into 51 golden and 21 adversarial cases; 44 currently have no committed prediction fixture |
| Authoring lines in the JSONL corpus | 202 | 151 lines in [evals/golden_set.jsonl](../evals/golden_set.jsonl) and 51 lines in [evals/adversarial_set.jsonl](../evals/adversarial_set.jsonl) |

Each JSONL authoring file opens with one `_meta` row at version `2.0.0`. Removing those two metadata rows leaves 150 golden cases and 50 adversarial cases, or 200 non-metadata authoring cases. The 202 authoring lines are not the extraction replay denominator.

## Retrieval measures are separate

Retrieval recall is the fraction of required ground-truth information present in the retrieved contexts for a retrieval query. Its population must be a specified set of queries with ground truth and captured retrieved contexts. Retrieval recall is unmeasured in the committed tree because there is no committed enabled judge run containing query-level ground truth, retrieved contexts, scores, and an aggregate over a declared population.

Support, called faithfulness in the evaluator, is the fraction of answer claims supported by retrieved contexts. Its population must be answers paired with the contexts used to produce them. Support is unmeasured in the committed tree because there is no committed enabled judge run containing claim decisions, source contexts, scores, and an aggregate over a declared answer population.

Abstention is whether the system declines to answer when the available evidence is insufficient. Its population must include labeled answerable and unanswerable queries, recorded system decisions, and an explicit abstention metric such as recall on unanswerable queries plus false-abstention rate on answerable queries. Abstention is unmeasured in the committed tree because no such labeled retrieval population and result artifact are committed.

The 0.9555 extraction replay score establishes none of retrieval recall, support, or abstention. It compares committed parsed extraction fields with expected document fields. It does not execute retrieval, inspect retrieved contexts, score answer claims, or test abstention decisions.

## Optional LLM judge implementation

[app/services/ragas_evaluator.py](../app/services/ragas_evaluator.py) defines `context_recall`, `faithfulness`, and `answer_relevancy` as Claude LLM-as-judge measures. Their configured weights are 0.35, 0.40, and 0.25. Evaluation is gated by `RAGAS_ENABLED`, which defaults off. When the flag is off, the evaluator returns `None` for all three measures and for the overall score. Therefore, the implementation does not provide a measured retrieval score in the committed tree.

## Demo fixture provenance trap

[frontend/demo_data/eval_sample.json](../frontend/demo_data/eval_sample.json) contains `context_recall` 0.91, `faithfulness` 0.93, `answer_relevancy` 0.88, `overall` 0.91, and `fixtures_evaluated` 16. Its `run_id` is `demo-eval-run-001` and its timestamp is `2024-11-20T09:00:00Z`.

Those values are demo fixture data, not a measurement. They do not establish retrieval recall, support, abstention, or a production evaluation result.

## CI and merge enforcement

[The Eval Gate workflow](../.github/workflows/eval-gate.yml) can execute the offline replay and report a passing or failing job. CI execution is distinct from enforced merge protection. The repository branch-protection query returned `Branch not protected`, and the branch-rules query returned an empty list. A passing job must not be described as an enforced merge gate under that repository state.

This note does not claim that README wording is correct. Correcting README gate and corpus wording is the later mandatory RA13 task and is outside this change.
Loading