site-capture

site-capture is a Python CLI that captures websites into durable artifacts for agent workflows.

It is designed around a narrow contract:

1. Discover URLs from robots.txt, sitemap files, or a bounded same-origin crawl.
2. Ask a browser driver to render one URL at a time.
3. Save screenshots, Markdown, optional HTML, per-page metadata, and a crawl ledger.

The default driver is playwriter, so the tool can use a real Chrome session controlled by Playwriter. An optional Playwright driver is available for public/static sites where a separate browser profile is enough.

Install

From the repo:

python -m pip install .

For local development:

make install-local
site-capture --json doctor

Commands

site-capture --json doctor
site-capture discover https://example.com --max-pages 50
site-capture capture https://example.com --out ./captures/example-home
site-capture crawl https://example.com --out ./captures/example --max-pages 100

If --out is omitted, capture and crawl write to ./captures/<host>-<command>-<timestamp>.

Use Playwriter explicitly:

site-capture crawl https://example.com \
  --driver playwriter \
  --session auto \
  --out ./captures/example \
  --formats screenshot,markdown

Capture a mobile view:

site-capture crawl https://example.com \
  --device mobile \
  --out ./captures/example-mobile

Remove a rendered overlay from both screenshots and rendered HTML with exact selectors:

site-capture capture https://example.com \
  --formats screenshot,html \
  --remove-selector '.newsletter-modal' \
  --remove-selector '#tracking-sandbox'

Missing or invalid removal selectors are reported as warnings. The capture tool does not infer substitute selectors.

For visual baselines and rendered HTML copies, settle lazy content and scroll-triggered states across the complete document:

site-capture capture https://example.com \
  --formats screenshot,html \
  --scroll-entire-page

Use an existing Playwriter session:

playwriter session list
site-capture crawl https://example.com --session 1 --out ./captures/example

Use Playwriter direct CDP mode:

site-capture crawl https://example.com --direct --out ./captures/example
site-capture crawl https://example.com --direct ws://localhost:9222/devtools/browser/... --out ./captures/example

Use the optional Playwright driver:

python -m pip install 'site-capture[playwright]'
playwright install
site-capture crawl https://example.com \
  --driver playwright \
  --playwright-profile ~/.site-capture/chrome-profile \
  --out ./captures/example

Output Layout

captures/example/
  manifest.json
  pages.jsonl
  pages/
    home-4be0d5625f/
      meta.json
      page.png
      page.md
      links.json

manifest.json summarizes the run. pages.jsonl is the machine-readable ledger agents should consume. Each page directory contains a meta.json file with the same stable shape:

{
  "url": "https://example.com/",
  "final_url": "https://example.com/",
  "status": 200,
  "title": "Example",
  "ok": true,
  "driver": "playwriter",
  "device": "desktop",
  "viewport": {"width": 1440, "height": 1200},
  "screenshot": "page.png",
  "markdown": "page.md",
  "html": null,
  "links": ["https://example.com/about"],
  "warnings": [],
  "error": null,
  "session": "1"
}

index.md gives a human-readable table of captured pages and artifact links. In pages.jsonl, artifact paths are relative to the capture root so the folder can be moved.

On reruns, existing successful pages are reused unless --force is passed. Crawl output distinguishes page_count, fresh captured_count, and reused_count; each ledger row includes source: "captured" or source: "reused".

JSON Policy

--json writes JSON to stdout. Progress goes to stderr. Error output follows this shape:

{"ok": false, "error": "message"}

The CLI never prints tokens or cookies intentionally. Captured page artifacts may contain private page content if your browser session can access it, so do not commit capture output.

Formats

--formats accepts:

• screenshot: full-page PNG.
• markdown: Playwriter getPageMarkdown output when using the Playwriter driver; deterministic rendered-HTML conversion when using the Playwright driver.
• html: rendered HTML.

Aliases are accepted: md for markdown, png for screenshot, and all for every format.

Default:

--formats screenshot,markdown

Notes

site-capture does not enforce robots.txt exclusion rules. It reads robots.txt only to find Sitemap: entries. Use it only on sites where you have permission or a legitimate reason to capture pages.

The crawler is intentionally bounded. Use --max-pages, --delay-ms, and --allow-off-origin explicitly when you need broader coverage.

Development

make test
site-capture --help
site-capture --json doctor