Skip to content

Benchmark current parser against Docling and MinerU #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
EstaraAtopos merged 1 commit into from
Aug 17, 2026
Merged

Benchmark current parser against Docling and MinerU #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
241 changes: 241 additions & 0 deletions docs/evaluations/14B-controlled-open-source-parser-benchmark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,241 @@
# Task 14B Controlled Open-source Parser Benchmark

## Status

- Technical status: `CONTROLLED_OPEN_SOURCE_PARSER_BENCHMARK_CLOSED`
- Quality status: `CONDITIONAL_PARSER_INTEGRATION_CANDIDATE_ESTABLISHED`
- Baseline: `6cca8cb0881226c6031f3adafff0d89c1146d584`
- Selected integration candidate: `docling@2.117.0`
- Selected role: `conditional_complex_document_parser_candidate`
- Production parser changed: false
- Production adapter authorized: false

## Decision

Docling is the best eligible candidate in this controlled comparison, but it
did not pass every document-class gate and must not replace the current parser
globally. It is recommended for the next adapter task only for:

- scanned English and Chinese PDFs;
- simple table PDFs with a preserved row/column shape.

The following classes remain excluded from Docling routing until their
individual gates are repaired and rerun:

- multi-column PDFs: reading-order accuracy was `0.8214`;
- mixed-layout table PDFs: the table was not retained;
- formula PDFs without composition with the existing `FormulaRegion` path:
formula structure retention was `0.0`.

The current native parser remains the simple-document fallback. The next task
must introduce per-document quality routing through the existing parser and
KnowledgeChunk contracts; it must not create a second ingestion or retrieval
system.

MinerU is not eligible. Its installed `3.4.4` metadata reports the non-standard
`LicenseRef-MinerU-Open-Source-License`, its multi-column score was lower, and
its Chinese headings were commonly merged into paragraph bodies. That reduced
the retrieval Chinese corpus to two chunks and prevented a downstream gain.

## Why This Is Not Merely A Strict Threshold Problem

The rejected cases have observable structural failures:

- Docling and MinerU interleaved content across columns in the synthetic
two-column PDF.
- Both candidates detected the simple table, but neither retained the table in
the mixed-layout blocker.
- Neither candidate emitted a governed `formula` block for the raster formula
fixture.
- MinerU merged nine Chinese term headings with their definitions as ordinary
paragraphs, so the existing heading-aware KnowledgeChunk builder produced
two broad chunks rather than eleven concept-preserving chunks.

These are content and provenance defects, not arbitrary policy failures. A
global replacement would therefore make some real documents worse despite the
higher aggregate score.

## Controlled Corpus

The corpus contains eleven repository-independent, runtime-generated synthetic
PDFs (fifteen pages total):

- single-column born-digital English;
- two-column born-digital English;
- scanned English;
- scanned Chinese;
- mixed-layout blocker;
- simple table;
- raster formula;
- negative/no-term document;
- three-page repeated header/footer document;
- two-page English retrieval concepts;
- two-page independent Chinese retrieval evidence.

The retrieval pair contains ten concepts in physics, mechanics, control
engineering, and applied mathematics, including three intentionally confusable
groups. The Chinese PDF contains no complete English gold term. Fixtures do
not contain private course material, inline bilingual answers, aliases, or
Provider output.

## Execution Contract

Each parser ran in an isolated process and normalized to a common evaluation
block contract. Candidate model artifacts and caches remained under
repository-external temporary storage and are not tracked by Git.

| Parser | Runtime | Local model policy | License gate |
|---|---|---|---|
| current native/Tesseract/FormulaRegion | project runtime | existing project configuration | pass |
| Docling 2.117.0 | isolated Conda environment | explicit pre-provisioned artifacts, offline mode | pass (MIT metadata) |
| MinerU 3.4.4 | isolated Conda environment | local pipeline, CPU, offline mode | blocked (non-standard license) |

No parser was allowed to call an external API. MinerU's local CLI used only a
loopback endpoint; the network audit observed zero external hosts and zero
external parser requests.

## Parser Metrics

| Metric | Current | Docling | MinerU |
|---|---:|---:|---:|
| Parse success | 0.8182 | 1.0000 | 1.0000 |
| Reading order (mean where defined) | 0.7778 | 0.9802 | 0.9762 |
| Heading-definition integrity | 1.0000 | 1.0000 | 1.0000 |
| Page provenance completeness | 0.8182 | 1.0000 | 1.0000 |
| Block provenance completeness | 0.8182 | 1.0000 | 1.0000 |
| BBox provenance completeness | 0.0000 | 1.0000 | 1.0000 |
| Table retention (simple + mixed) | 0.0000 | 0.5000 | 0.5000 |
| Formula structure retention | 0.0000 | 0.0000 | 0.0000 |
| Repeated header/footer filter | 0.0000 | 1.0000 | 1.0000 |
| Duplicate body blocks | 0 | 0 | 0 |
| Median runtime | 32.42 ms | 5,049.68 ms | 10,366.44 ms |
| Peak RSS | 76.83 MB | 2,446.58 MB | 1,830.62 MB |

`Parse success` requires non-empty governed content, not merely a zero process
exit code. The current parser's scanned English and Chinese runs exited cleanly
but produced no eligible content, so both correctly fail that metric.

Aggregate values do not authorize routing. The selection manifest separately
retains critical per-fixture gates:

| Critical fixture | Current | Docling | MinerU |
|---|---:|---:|---:|
| Two-column reading order | 1.0000 | 0.8214 | 0.7857 |
| Scanned English content success | false | true | true |
| Scanned Chinese content success | false | true | true |
| Simple table retention | 0.0000 | 1.0000 | 1.0000 |
| Mixed-layout table retention | 0.0000 | 0.0000 | 0.0000 |
| Formula structure retention | 0.0000 | 0.0000 | 0.0000 |
| Repeated header/footer filter | 0.0000 | 1.0000 | 1.0000 |

## Existing-pipeline Retrieval Impact

Normalized blocks were converted through
`knowledge_governance.build_knowledge_chunks_from_parse_blocks` and ranked by
the existing fixed local backend:

- model: `intfloat/multilingual-e5-small`;
- revision: `614241f622f53c4eeff9890bdc4f31cfecc418b3`;
- backend: `local_multilingual_e5_pytorch_cpu_v1`.

No new embedding abstraction, model, vector database, translation step, or
gold mapping was introduced. Gold concept IDs were used only after ranking to
calculate metrics.

| Parser | EN chunks | ZH chunks | hit@1 | hit@3 | MRR | No result |
|---|---:|---:|---:|---:|---:|---:|
| Current | 2 | 2 | 0.1000 | 0.1000 | 0.1000 | 0 |
| Docling | 11 | 11 | 0.5000 | 0.6000 | 0.5500 | 0 |
| MinerU | 11 | 2 | 0.1000 | 0.1000 | 0.1000 | 0 |

Docling materially improves concept-preserving chunk construction and the
controlled downstream retrieval baseline, but `hit@3 = 0.6000` is not a claim
of broad product quality. It establishes a conditional adapter candidate only.

## Architecture Boundary

Before this task:

```text
PDF -> current parser/OCR/layout diagnostics -> parse blocks
-> existing heading-aware KnowledgeChunk builder
-> existing multilingual retrieval
```

This task adds only an offline comparison path:

```text
synthetic PDF
-> isolated current / Docling / MinerU probe
-> neutral evaluation blocks
-> existing KnowledgeChunk builder
-> existing multilingual retrieval
-> metric and selection artifacts
```

Production remains unchanged. The ordered next task may add:

```text
production PDF
-> existing parser adapter and document-quality router
-> simple digital: current native parser
-> scanned or validated simple-table: Docling candidate
-> formula: compose existing FormulaRegion
-> failed class gate: current fallback or fail closed
-> existing parse blocks -> existing KnowledgeChunk -> existing retrieval
```

The PDF selection interface and concept learning card are deliberately not
changed here. Direct PDF text-layer selection remains an input mechanism; the
student-facing concept learning card remains the intended output and will be
implemented only after the parser adapter is closed.

## Safety

- Application external API used: false.
- External parser request count: 0.
- Real Provider requests: 0.
- Real credentials read: false.
- Private fixture used: false.
- Incident database accessed: false.
- Production parser changed: false.
- Production adapter changed: false.
- Model/cache tracked: false.
- Request/response/complete source bodies stored in artifacts: false.

## Artifacts

- `14B-controlled-parser-benchmark-results.json`
- SHA-256: `293abe451b713f9f68d354f5dc8f6472760558a06104d7aba248e426429cad5d`
- `14B-controlled-parser-benchmark-matrix.csv`
- SHA-256: `62908931e193141407e832323a5048d98e860cc4d9ed3426e6674324386ac6d0`
- `14B-parser-selection-manifest.json`
- SHA-256: `7eddca05ac7c0aa72e6b5d4c28a567960768de5a339c17404cc22e65d6ec6232`

## Verification

- New and related parser/layout/chunk tests: `66 passed, 1 skipped`.
- Full pytest in the task runtime: `1722 passed, 5 skipped`.
- `dev_check` in the canonical runtime: `1720 passed, 7 skipped`; migration
and backend API smoke passed. The two additional skips reflect optional
layout-runtime availability in that separately pinned runtime, not failures.
- Release safety: pass.
- `git diff --check`: pass.
- Cross-Corpus V2 frozen hashes remain:
- manifest: `a53a5152aa66189c954b703c42fb815a6a8c7f83933051f80f1538e78bdc3f88`;
- gold: `3379bc1e6c589256dcec384a9f7435c96277f6754b7cb44cff69f86244a9b3f0`;
- English bundle: `e84fc99a993c099628fe7d740c1e248fefe1e03cbece7b4df43d02cd7dfaddc5`;
- Chinese bundle: `cebd3274cb0802fa645948d9b641b1776c60ab99700bbf3cd4a80999bc45b5f7`.
- Accident database before/final:
- SHA-256: `9e6fb68ab13dff2763e2fff18d2aee531486360c557f058eb8a471d9209a9eaa`;
- size: `1015808`;
- mtime: `1785496597`;
- WAL/SHM: absent/absent.

## Next Ordered Step

Add a bounded Docling adapter through the existing production parser contract,
with explicit document-class routing, existing FormulaRegion composition,
fallback/fail-closed behavior, offline model provisioning, resource bounds,
and no change to KnowledgeChunk or retrieval semantics. Do not begin the PDF.js
reader or concept-card redesign until that adapter task is merged and verified.
34 changes: 34 additions & 0 deletions docs/evaluations/artifacts/14B-controlled-parser-benchmark-matrix.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
parser_id,fixture_id,parse_success,reading_order_accuracy,heading_definition_integrity,page_provenance_completeness,block_provenance_completeness,bbox_provenance_completeness,table_retention,formula_retention,header_footer_filter_rate,duplicate_block_count,runtime_ms,peak_rss_mb
baseline_native_tesseract_formula_region,single_column_born_digital,True,1.0,,1.0,1.0,0.0,,,,0,29.63,46.7
baseline_native_tesseract_formula_region,two_column_born_digital,True,1.0,,1.0,1.0,0.0,,,,0,54.58,47.38
baseline_native_tesseract_formula_region,scanned_english,False,0.0,,0.0,0.0,0.0,,,,0,85.7,76.59
baseline_native_tesseract_formula_region,scanned_chinese,False,0.0,,0.0,0.0,0.0,,,,0,85.79,76.83
baseline_native_tesseract_formula_region,mixed_layout_blocker,True,1.0,,1.0,1.0,0.0,0.0,,,0,31.0,47.06
baseline_native_tesseract_formula_region,simple_table,True,1.0,,1.0,1.0,0.0,0.0,,,0,30.43,46.83
baseline_native_tesseract_formula_region,raster_formula,True,,,1.0,1.0,0.0,,0.0,,0,80.31,52.31
baseline_native_tesseract_formula_region,negative_no_terms,True,,,1.0,1.0,0.0,,,,0,30.73,47.08
baseline_native_tesseract_formula_region,repeated_header_footer,True,1.0,1.0,1.0,1.0,0.0,,,0.0,0,31.97,47.0
baseline_native_tesseract_formula_region,retrieval_english,True,1.0,1.0,1.0,1.0,0.0,,,,0,35.97,46.97
baseline_native_tesseract_formula_region,retrieval_chinese,True,1.0,1.0,1.0,1.0,0.0,,,,0,32.42,47.28
docling,single_column_born_digital,True,1.0,,1.0,1.0,1.0,,,,0,7105.47,1469.59
docling,two_column_born_digital,True,0.8214,,1.0,1.0,1.0,,,,0,5488.17,1549.94
docling,scanned_english,True,1.0,,1.0,1.0,1.0,,,,0,5989.8,1998.62
docling,scanned_chinese,True,1.0,,1.0,1.0,1.0,,,,0,7488.16,2446.58
docling,mixed_layout_blocker,True,1.0,,1.0,1.0,1.0,0.0,,,0,4509.57,1651.75
docling,simple_table,True,1.0,,1.0,1.0,1.0,1.0,,,0,4573.45,1693.83
docling,raster_formula,True,,,1.0,1.0,1.0,,0.0,,0,6621.94,2022.55
docling,negative_no_terms,True,,,1.0,1.0,1.0,,,,0,4540.28,1640.38
docling,repeated_header_footer,True,1.0,1.0,1.0,1.0,1.0,,,1.0,0,5049.68,1505.3
docling,retrieval_english,True,1.0,1.0,1.0,1.0,1.0,,,,0,4942.29,1645.81
docling,retrieval_chinese,True,1.0,1.0,1.0,1.0,1.0,,,,0,4670.07,1644.03
mineru,single_column_born_digital,True,1.0,,1.0,1.0,1.0,,,,0,10366.44,1416.88
mineru,two_column_born_digital,True,0.7857,,1.0,1.0,1.0,,,,0,8639.2,1414.5
mineru,scanned_english,True,1.0,,1.0,1.0,1.0,,,,0,7730.35,1421.36
mineru,scanned_chinese,True,1.0,,1.0,1.0,1.0,,,,0,8838.18,1425.73
mineru,mixed_layout_blocker,True,1.0,,1.0,1.0,1.0,0.0,,,0,9145.91,1424.64
mineru,simple_table,True,1.0,,1.0,1.0,1.0,1.0,,,0,10429.65,1415.84
mineru,raster_formula,True,,,1.0,1.0,1.0,,0.0,,0,13224.48,1830.62
mineru,negative_no_terms,True,,,1.0,1.0,1.0,,,,0,9541.72,1403.34
mineru,repeated_header_footer,True,1.0,1.0,1.0,1.0,1.0,,,1.0,0,11807.61,1527.08
mineru,retrieval_english,True,1.0,1.0,1.0,1.0,1.0,,,,0,12114.65,1483.81
mineru,retrieval_chinese,True,1.0,1.0,1.0,1.0,1.0,,,,0,12112.85,1443.22
Loading
Loading