pdfdecomp

Decompose PDFs into structured text, visual assets, and a canonical JSON manifest. Built on Docling for layout analysis and Tesseract for OCR.

What it does

Given a PDF, pdfdecomp produces an output package containing:

• manifest.json — canonical structured representation with every detected component (headings, paragraphs, tables, figures, etc.), bounding boxes, page mapping, and RAG chunking hints
• document.md / document.txt — full-text exports
• document.docling.json — raw Docling document model
• assets/ — cropped images of tables and figures, plus CSV exports for tables

Installation

Local (development)

pip install -e ".[all]"

Docker

Three image targets are available:

Target	Description
slim	Docling + Tesseract (default)
full	Adds OCRmyPDF + Ghostscript for scanned PDF recovery
api	FastAPI HTTP service on top of slim

# Build the slim image
docker build --target slim -t pdfdecomp:slim .

# Build the full image
docker build --target full -t pdfdecomp:full .

# Build the API image
docker build --target api -t pdfdecomp:api .

Apptainer / Singularity (HPC)

# Build SIF directly from definition file
./scripts/build_container.sh apptainer

# Or build Docker image and convert to SIF
./scripts/build_container.sh docker slim --sif

Usage

CLI

# Convert a single PDF
pdfdecomp convert input.pdf -o output/

# Batch-process a directory of PDFs
pdfdecomp batch papers/ -o output/

# Quick metadata inspection
pdfdecomp inspect input.pdf

Docker

# Convert a PDF (use absolute paths or $PWD)
docker run --rm \
  -v "$PWD/tests/fixtures":/in:ro \
  -v "$PWD/out":/out \
  pdfdecomp:slim convert /in/digital_simple.pdf -o /out/result

# Run the API server
docker run --rm -p 8000:8000 pdfdecomp:api

Apptainer (HPC)

./scripts/run_apptainer.sh /path/to/pdfdecomp.sif convert /data/in/doc.pdf -o /data/out/doc

HTCondor submit files are provided in condor/ for cluster job submission.

API

# Health check
curl http://localhost:8000/health

# Convert a PDF (returns a zip archive)
curl -X POST http://localhost:8000/convert \
  -F "file=@document.pdf" \
  -o result.zip

CLI Options

Flag	Default	Description
--mode / -m	auto	OCR mode: auto, force-ocr, ocrmypdf
--ocr-lang	eng	OCR language(s)
--tables/--no-tables	--tables	Enable table structure detection
--page-images/--no-page-images	--page-images	Render page images for asset cropping
--export-embedded-images	off	Extract raw embedded images via pypdf
--images-scale	2.0	Page render scale (effective DPI = scale * 72)
--chunking-max-tokens	512	Max tokens per RAG chunk
--max-pages	0	Limit pages to process (0 = unlimited)
--max-file-size-mb	500	Reject files larger than this
--debug	off	Write debug overlays and diagnostics

All flags can also be set via environment variables with the PDFDECOMP_ prefix (e.g., PDFDECOMP_MODE=force-ocr).

Output Structure

output/
  manifest.json              # Canonical structured manifest
  document.md                # Markdown export
  document.txt               # Plain text export
  document.docling.json      # Raw Docling document model
  assets/
    tables/                  # Table images (PNG) and CSV exports
    visuals/                 # Figure/picture crops (PNG)
    raw-images/              # (optional) Embedded images from pypdf
  debug/                     # (optional) Debug overlays

Pipeline Stages

1. Ingest — validate, fingerprint (SHA256), extract page dimensions
2. Docling conversion — layout analysis, OCR, table structure recognition
3. Export — markdown, plain text, and Docling JSON
4. Quality assessment & recovery — detect low-quality extractions, re-run with force-OCR or OCRmyPDF
5. Asset materialization — crop figures and tables, export table CSVs
6. Embedded image extraction — (optional) raw images via pypdf
7. RAG chunking — structure-aware chunk boundaries via Docling's HybridChunker
8. Manifest — assemble the canonical JSON output

Project Structure

src/pdfdecomp/
  cli.py              # Typer CLI (convert, batch, inspect)
  api.py              # FastAPI HTTP service
  config.py           # Pydantic settings
  pipeline.py         # Pipeline orchestrator
  docling_adapter.py  # Docling interface layer
  manifest.py         # Manifest builder
  assets.py           # Visual asset extraction
  chunking.py         # RAG chunking hints
  coordinates.py      # Bounding box utilities
  taxonomy.py         # Component type classification
  recovery.py         # Quality assessment & OCR recovery
  diagnostics.py      # Debug overlays & timing reports
  embedded.py         # Embedded image extraction
  exceptions.py       # Custom exceptions

containers/           # Apptainer definition file
condor/               # HTCondor submit files
scripts/              # Build and run helper scripts
tests/                # Pytest suite

Development

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run integration tests (requires Docling models)
pytest -m integration

# Lint
ruff check src/ tests/

License

MIT