Skip to content

Add Poppler pdftotext -bbox-layout as a second ground-truth oracle #5

Description

@Fanaperana

Problem

Today benchmark/spatial.py computes IoU + centroid error against a single ground-truth oracle: raw Tesseract OCR at 150 DPI. Tesseract is a bad oracle on complex forms — on IRS 1040 it extracts "Treasiny—Intemal" for "Treasury—Internal", which tanks spdf's measured F1 to 15.5% even though its output is visually correct ("Department of the Treasury—Internal").

We need a second, independent oracle so the benchmark reports where two engines agree, not where tesseract's hallucinations live.

Proposal

Add Poppler pdftotext -bbox-layout as a second oracle. It emits per-word bounding boxes directly from the PDF content stream (no rasterisation), which is exactly the ground truth we need for digitally-born PDFs.

Design

  1. benchmark/spatial.py:
    • Keep the existing tesseract-TSV pipeline for raster-only fixtures (example-1.jpg, test-ocr.pdf).
    • For PDF fixtures, run pdftotext -bbox-layout <pdf> - and parse the <word xMin yMin xMax yMax> XHTML output.
    • Report three columns per engine: F1 vs tesseract, F1 vs pdftotext, F1 vs agreement (words where the two oracles also agree).
  2. CLI flag --oracle {tesseract,pdftotext,both} (default both on PDFs).
  3. Update README.md benchmark table: headline number becomes F1 against oracle agreement, which is a more defensible claim.

Acceptance

  • benchmark/spatial.py supports both oracles.
  • benchmark/results/spatial.{md,json} includes both columns.
  • README headline numbers updated with the methodology note.
  • IRS-1040 F1 under the agreement metric is > 60% (hypothesis: pdftotext and tesseract both agree on most prose; disagreement concentrates in form-field labels where tesseract is unreliable).

Metadata

Metadata

Assignees

No one assigned

    Labels

    benchmarkBenchmark / corpus / ground truthenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions