feat(pdf): read PDF files, including two-column papers (OXD-060)

[AlexanderSlokov]

Closes OXD-060. Re-scopes OXD-061 from XL to M.

oxidiris paper.pdf now works the same way oxidiris paper.md does. Detection is by file header
rather than extension, so cat paper.pdf | oxidiris - works and a mis-named file still reads.

The spike overturned its own premise

OXD-060 was written expecting to shell out to Poppler's pdftotext -layout. Measured on five
real two-column papers — Borg, Omega, MapReduce, Bigtable, Spanner — the pure-Rust pdf-extract
won on both axes, so there is no external dependency and no subprocess.

Reading order. There is no ground truth to diff against, so the two extractors were compared
to each other (word sequences, NFKC, de-hyphenated, matched in order by LCS ratio):

Paper	Ordered overlap
Spanner	0.959
Bigtable	0.954
Borg	0.953
MapReduce	0.925
Omega	0.699

Omega is where they disagree, and Poppler is the one that is wrong: it lifts the Figure 1
labels (Monolithic, Two-level, scheduling logic) out of the figure and drops them into the
middle of the abstract. pdf-extract keeps the body running clean. -layout was rejected
outright — it puts the two columns side by side on every line.

Speed, best of three, release build. oxidiris runs the whole pipeline (extract, tidy, parse,
tokenize, print); pdftotext only extracts:

Paper	Pages	Size	oxidiris	pdftotext
Borg	18	0.9 MB	176 ms	144 ms
Omega	14	4.3 MB	288 ms	319 ms
MapReduce	13	0.2 MB	24 ms	60 ms
Bigtable	14	0.2 MB	36 ms	71 ms
Spanner	14	0.4 MB	73 ms	89 ms

Full write-up, including why lopdf on its own was dropped before benchmarking, in
ADR 003.

Shape of the change

New crate oxidiris-pdf, holding the entire PDF dependency tree. oxidiris-core is untouched and
still compiles for wasm32-unknown-unknown, which is the whole reason for the split. Output is
plain text rather than a Document: a PDF has no reliable block structure, and the plain-text
parser already rejoins hard-wrapped lines.

pdf bytes -> oxidiris_pdf::extract -> plain text -> oxidiris_core::load -> Document

Text is tidied before the reader sees it — ligatures expanded, hyphenated line breaks welded,
isolated short numbers dropped. Every step is a pure &str -> String function, tested without a
PDF in sight.

One rule was renamed after measurement. It was written to remove running page numbers.
Measuring the corpus showed the isolated numeric lines are mostly chart axis tick labels pulled
out of figures; Bigtable produces none at all. The function and its comments now say what it
actually does.

Errors

Encrypted, damaged, and no-text-layer files each explain themselves. pdf-extract panics rather
than returning Err on some malformed input, so the panic is caught and converted (spec §4.4) —
the only place in the project that catches a panic.

$ oxidiris scan.pdf
oxidiris: cannot read scan.pdf: the PDF has no text layer (1 page scanned); it is probably a scan and needs OCR

Fixtures

testdata/make_pdf_fixtures.py generates three small PDFs with no third-party library. The real
papers are not committed — they are under publisher copyright and a fixture has to be
redistributable.

Known gaps, tracked on OXD-061

• Figure and table text still lands in the body stream. Reading Borg, the Figure 1 labels appear
  after the first-page footer. Poppler has the same defect, so there is no tool to defer to.
• A PDF produces no heading structure, so the outline panel is empty for one.
• A compound breaking at its own hyphen (over- / commitment) comes back welded shut.
  Separating those needs a dictionary.
• The encrypted path is implemented but has no test fixture; writing an RC4-encrypted PDF by
  hand was judged not worth the cost. Left unticked in the acceptance list rather than claimed.

New advisory ignore

deny.toml now ignores RUSTSEC-2026-0192 (ttf-parser unmaintained), reached through
lopdf -> pdf-extract. The advisory states no upgrade exists — skrifa is a change lopdf has to
make upstream. It is a maintenance notice, not a vulnerability. The entry names it and says to
delete it the moment lopdf switches; also recorded as carried debt on OXD-061.

Verification

• make check green: 291 tests, fmt, clippy, wasm build.
• cargo deny check: advisories, bans, licenses, sources all ok.
• All five real papers read end to end; stdin, mis-named files and truncated files all produce
  clean messages.
• make msrv not run — the 1.88 toolchain is not installed locally. The highest MSRV in the
  new dependency tree is 1.85, so this should pass, but CI is what confirms it.

🤖 Generated with Claude Code

https://claude.ai/code/session_0189gxRgTxfvzjNYezCfS2iR