Code Wiki Scraper

Preservation-first scraper for codewiki.google pages.

Goals

• Preserve Code Wiki content with minimal loss.
• Render diagrams correctly in Markdown viewers.
• Keep diagram semantics available as text for AI agents.
• Provide a small CLI that is easy to extend.

Strategy

Code Wiki diagrams are rendered as Graphviz SVG wrapped inside a data URI. Converting them to Mermaid loses structure and labels in many cases, so this project uses a two-track output:

1. Exact SVG assets are extracted to an assets directory and linked from the Markdown, so human readers see the real diagram.
2. Textual diagram semantics are extracted from the SVG graph structure and embedded into the Markdown for LLM context.

This preserves visual fidelity and gives agents node / edge information without forcing them to parse raw XML.

Quick Start

uv sync
uv run codewiki-scraper scrape https://codewiki.google/github.com/google/filament -o filament_codewiki.md

Default export flow:

uv run codewiki-scraper scrape https://codewiki.google/github.com/google/filament

This writes into output/google-filament/.

Compatibility entry point:

python codewiki_scraper.py scrape https://codewiki.google/github.com/google/filament -o filament_codewiki.md

Output

The scraper writes:

• output/<slug>/<slug>.md
• output/<slug>/<slug>.agent.md
• output/<slug>/assets/*.svg
• output/<slug>/<slug>.html
• output/<slug>/metadata.json
• output/manifest.json for batch runs

The Markdown includes:

• fenced code blocks for source snippets
• rendered image links for diagrams
• fenced text blocks with diagram semantics
• cleaned HTML snapshot for auditing extraction quality
• metadata for downstream tooling

The companion agent markdown includes:

• the same textual documentation
• code blocks
• diagram semantics
• no rendered image links, which keeps it cleaner for LLM ingestion

CLI

codewiki-scraper scrape URL -o OUTPUT.md
codewiki-scraper batch URL1 URL2
codewiki-scraper batch --input-file urls.txt

Options:

• --output-dir output
  • base directory for generated export bundles
• --diagram-format svg|mermaid
  • svg is the default and recommended mode.
  • mermaid is best-effort and may lose detail.
• --no-diagram-summary
  • keep rendered diagrams only

Development

uv sync --group dev
uv run pytest

Quality tooling:

uv run ruff check .
uv run ruff format .
uv run ty check --config-file ty.toml

Make targets:

make help
make sync
make check
make fmt
make scrape-samples