collector

Agent harness that crawls PDFs, extracts with Docling, and compiles a grounded Obsidian wiki — with page images, JSON Canvas, and cross-document citations.

Inspired by the Karpathy LLM Knowledge Base pattern.

URL / query / local path
  → explore & download PDFs
  → Docling extract (markdown + images)
  → Obsidian vault with documents, page images, wiki articles
  → JSON Canvas visual map

Prerequisites

brew install qpdf ghostscript
pip install docling
npm install

Requires a Claude Code subscription (the agent runs via @anthropic-ai/claude-agent-sdk).

Optional: npm i -g apify-cli for deep site crawling.

Usage

# Crawl a site for PDFs
node collect.mjs "https://arxiv.org/list/cs.AI/recent" --name ai-papers

# Direct PDF URLs
node collect.mjs "https://arxiv.org/pdf/2401.04088 https://arxiv.org/pdf/2312.11805" --name llm-papers

# Research query (agent searches the web)
node collect.mjs "SEC 10-K filings NVIDIA 2023-2025" --name nvidia

# Local PDFs
node collect.mjs ./my-pdfs/ --name my-research

# Custom wiki topics
node collect.mjs "https://example.com/reports" --topics topics.txt

Output

collections/<name>/
  raw/          ← downloaded PDFs
  extracted/    ← Docling markdown + images per PDF
  vault/
    .obsidian/
    Index.md
    documents/  ← full extraction per PDF (frontmatter + wikilinks)
    pages/      ← extracted figures, tables, charts (PNGs)
    wiki/       ← cross-document compiled articles with citations
    canvas/     ← JSON Canvas visual map

Open vault/ in Obsidian → Cmd+G for graph view, or open canvas/overview.canvas for the visual map.

How wiki compilation works

The agent reads every extracted document file, then writes wiki articles citing specific numbers and facts with [[wikilinks]] back to source documents. Claims are grounded in the extracted text — if information isn't in the documents, the agent says so rather than hallucinating.

Example citation:

MMLU: ==90.04%== — first to exceed human experts ([[documents/gemini|Gemini]], Section 5.1.1)

Options

Flag	Default	Description
--name, -n	derived from input	Collection name
--topics, -t	built-in defaults	Topics file (one per line)
--output, -o	./collections	Output parent directory
--model, -m	SDK default	Claude model override
--verbose, -v	false	Show raw SDK events

License

MIT