Coda to Outline ETL Pipeline

One-off Python scripts that pull pages + tables from the Coda API (as Markdown/JSON) and push them into an Outline collection. The tooling is intentionally simple—run it locally, inspect the generated files, and only then hit Outline. For coding style, review expectations, and background, see AGENTS.md and CONTRIBUTING.md.

Quick Start

1. Copy .env.example to .env, then fill in CODA_API_TOKEN, OUTLINE_API_TOKEN, OUTLINE_API_URL, and the Outline OUTLINE_COLLECTION_ID you plan to target (there is no baked-in default).
2. Install uv and sync dependencies:

   uv sync

3. Activate the venv (source .venv/bin/activate) or prefix commands with uv run ....
4. Extract, inspect, and upload:

   uv run python main.py --doc-ids <DOC_ID> --skip-upload
   ls data/raw  # review markdown + manifests
   uv run python main.py --skip-extract --collection-id <OUTLINE_COLLECTION_ID> --dry-run
   uv run python main.py --skip-extract --collection-id <OUTLINE_COLLECTION_ID> --publish --replace-collection

Architecture (High Level)

Phase	What Happens	Key Artifacts
Extraction (extract_from_coda.py)	Lists docs/pages, exports each page as HTML→Markdown (with an inline SVG/IMG-friendly fallback), dumps every table as JSON.	data/raw/<doc>_<page>.md, data/raw/<doc>_table_<table>.json, data/raw/<doc>_pages_manifest.json
Upload (upload_to_outline.py)	Reads manifests to recreate hierarchies, optionally purges the destination collection, and uploads each Markdown file to Outline (or simulates in --dry-run).	Outline documents + dry-run logs

Each manifest entry looks like:

{
  "doc_id": "doc-123",
  "page_id": "page-456",
  "page_name": "Spec",
  "parent_page_id": null,
  "order": 1,
  "file_name": "doc-123_page-456.md"
}

The uploader sorts by depth → order → name so children are always created after their parents. If no manifest is present, it falls back to a flat alphabetical upload.

Running Locally

• Extract only: uv run python main.py --skip-upload
• Upload only: uv run python main.py --skip-extract --collection-id <ID> [--publish] [--replace-collection]
• Dry run: add --dry-run to print would-be uploads without touching Outline.
• Direct scripts: uv run python extract_from_coda.py --doc-ids ..., uv run python upload_to_outline.py --input-dir ./data/raw --collection-id ...

Logs live in logs/coda_to_outline.log and are replaced on every run, so capture them immediately if you need to share output.

Testing

This repo now includes lightweight smoke tests meant to validate the manifest planner and Markdown rendering helpers without hitting the real APIs. Install deps with uv sync (pytest ships with the default environment) and run:

uv run pytest

Because this project targets local one-off runs, tests rely on small fakes/mocks rather than full integration coverage.

Contributing

See CONTRIBUTING.md for the workflow, code style, and validation checklist. Pull requests should include the commands/logs you ran plus any throwaway Outline collection IDs that reviewers might need.