crawler.py

A recursive website crawler that discovers and lists every URL on a site by combining sitemap seeding with a concurrent HTML link crawler.

Built for people who need a complete URL inventory of their own websites — for SEO audits, migration planning, broken-link checks, sitemap generation, or feeding into other tooling.

Intended for sites you own or have explicit permission to crawl. This tool does not honor robots.txt. Only point it at infrastructure you control.

---

Table of Contents

• Features
• Requirements
• Installation
• Quick Start
• Usage
  • Basic Examples
  • Common Workflows
• Command-Line Reference
• How It Works
  • Crawl Strategy
  • URL Canonicalization
  • Filtering Rules
  • Concurrency and Politeness
• Output
• Exit Codes
• Performance Tuning
• Troubleshooting
• Limitations
• Extending the Script
• Security Notes
• License
• See Also

---

Features

• Sitemap seeding — tries a set of common sitemap locations (/sitemap.xml, /wp-sitemap.xml, /sitemap_index.xml, etc.) first, including recursive descent into <sitemapindex> files.
• Full recursive crawl — breadth-first traversal of every <a href> link on the site, with optional depth and page-count limits.
• Concurrent fetching — thread-pooled workers with a global rate limiter for polite behavior on your own servers.
• Robust URL canonicalization — deduplicates page, page/, page#frag, and trailing-slash variants.
• Redirect-aware — records final URLs after HTTP redirects.
• Content-aware — only parses responses whose Content-Type is HTML, and skips obvious non-HTML extensions (images, PDFs, archives, media, etc.).
• Subdomain support — optionally follow links across subdomains of the same registrable domain.
• Clean stdout/stderr split — URLs go to stdout, logs go to stderr, so > urls.txt works without pollution.
• Graceful interruption — Ctrl-C writes partial results instead of discarding everything.
• Zero external dependencies beyond requests and beautifulsoup4.

---

Requirements

• Python 3.8+
• requests
• beautifulsoup4

---

Installation

pip install requests beautifulsoup4

---

Quick Start

# Crawl a site, write URLs to a file
python crawler.py https://example.com -o urls.txt

# Show the first bit of a crawl to see what's happening
python crawler.py example.com -d 2 -v

---

Usage

python crawler.py <target> [options]

The <target> argument accepts either a bare hostname or a full URL. If no scheme is given, https:// is prepended automatically.

python crawler.py example.com
python crawler.py https://example.com
python crawler.py https://example.com/blog/

Basic Examples

Crawl a small site with default settings (8 workers, unlimited depth):

python crawler.py https://mysite.com

Seed from sitemap, then crawl everything recursively, output to a file:

python crawler.py mysite.com -o urls.txt

Verbose progress so you can see the frontier grow:

python crawler.py mysite.com -v

Common Workflows

Polite crawl of a large production site (throttled):

python crawler.py mysite.com --workers 4 --delay 0.3 -o urls.txt

Quick structural overview — just two levels deep:

python crawler.py mysite.com -d 2 -v

Smoke test with a hard page cap (useful before a big run):

python crawler.py mysite.com -m 100 -v

Crawl across subdomains (blog., docs., shop., etc.):

python crawler.py example.com --include-subdomains -o urls.txt

Skip sitemap lookup and start from a specific page:

python crawler.py https://example.com/section/ --no-sitemap

Self-signed cert on a staging environment:

python crawler.py https://staging.internal --insecure

---

Command-Line Reference

Argument	Type	Default	Description
target	positional	—	Target URL or hostname. Required.
-t, --timeout	float	10.0	Per-request timeout in seconds.
-d, --max-depth	int	unlimited	Maximum crawl depth from seeds. 0 = only the seed URL(s).
-m, --max-pages	int	unlimited	Hard cap on the number of pages actually fetched.
-w, --workers	int	8	Concurrent worker threads.
--delay	float	0.0	Global minimum delay between request starts, in seconds.
--no-sitemap	flag	off	Skip sitemap seeding; start crawling from target.
--include-subdomains	flag	off	Follow links to other subdomains of the same registrable domain.
-o, --output	path	stdout	Write URLs to a file instead of printing them.
--insecure	flag	off	Disable TLS certificate verification.
-v, --verbose	flag	off	Print progress and per-URL errors to stderr.
-h, --help	flag	—	Show help and exit.

Flag Details

-d, --max-depth

Depth is counted from the seed URL(s). Sitemap URLs are seeds, so -d 1 means "seed pages plus everything they link to." -d 0 fetches only the seeds themselves. When omitted, the crawler runs until it exhausts links or hits --max-pages.

--delay

This is a global rate limiter, not per-worker. With --delay 0.3 --workers 8, the crawler still only starts one request every 300ms site-wide — the workers just fill in the gaps between throttled request starts. Combine with --workers when you want controlled concurrency but bounded throughput.

-m, --max-pages

Counts fetched pages, not discovered URLs. When the limit is hit, the queue is cleared and the crawl ends gracefully. Useful for dry runs on sites with tens of thousands of pages.

--include-subdomains

Uses a rough eTLD+1 computation. This handles common two-part TLDs (co.uk, com.au, co.jp, co.nz, com.br, co.in) but is not a full Public Suffix List implementation. See Limitations.

---

How It Works

Crawl Strategy

1. Normalize the target to https://host/ if no scheme was provided.
2. Seed the frontier:
   • Always includes the target URL itself.
   • If --no-sitemap is not set, tries a list of common sitemap paths and recursively parses <sitemapindex> files. All same-host sitemap URLs are added as seeds.
3. Breadth-first crawl:
   • Pulls the current frontier out of the queue.
   • Dispatches the batch to a ThreadPoolExecutor.
   • For each successful HTML response, extracts all <a href> links.
   • Filters links against the same-domain rule, canonicalizes them, and enqueues unseen ones for the next frontier.
4. Stop when:
   • The queue is empty,
   • --max-pages is reached, or
   • The user presses Ctrl-C (partial results are still written).

URL Canonicalization

Every URL is passed through canon() before being added to any set. This:

• Strips fragments (#section),
• Lowercases the scheme and host,
• Normalizes an empty path to /,
• Drops the URL params component (rarely used, safe to discard).

This is what prevents page, page/, page#top, and PAGE from being treated as four separate URLs.

Filtering Rules

A link is skipped if:

• It starts with mailto:, tel:, javascript:, or data:.
• Its scheme is not http or https.
• Its host isn't the target host (unless --include-subdomains is set).
• Its path ends with a known non-HTML extension (images, media, archives, fonts, Office docs, executables — see SKIP_EXTENSIONS in the script).

A response is skipped (no link extraction) if:

• It's not HTTP 200.
• Its Content-Type header doesn't contain html.

Concurrency and Politeness

• Workers are thread-based (ThreadPoolExecutor). The crawler is I/O-bound, so threads are more than enough — no asyncio needed.
• A global lock enforces --delay across all workers.
• A single requests.Session is shared across workers, which enables HTTP keep-alive and connection pooling. This is a significant speedup over opening a fresh connection per request.
• KeyboardInterrupt is caught in main() and the discovered set is written out before exit.

---

Output

By default, URLs are printed to stdout, one per line, sorted lexicographically. Progress, warnings, and errors go to stderr.

This means you can safely do:

python crawler.py mysite.com > urls.txt          # works
python crawler.py mysite.com | wc -l             # count
python crawler.py mysite.com 2>/dev/null         # suppress logs

With -o FILE, the same sorted output is written to a file and a summary line goes to stderr:

[+] Fetched 1,247 page(s); 3,891 unique URL(s) discovered.

Example output:

https://example.com/
https://example.com/about
https://example.com/blog/
https://example.com/blog/hello-world
https://example.com/contact
https://example.com/products/widget

---

Exit Codes

Code	Meaning
0	Success — URLs were discovered and written.
1	No URLs discovered, or output file could not be written.
2	Invalid target / argument parsing error.

Useful in shell scripts:

python crawler.py mysite.com -o urls.txt || echo "Crawl failed"

---

Performance Tuning

The two knobs that matter most are --workers and --delay.

Scenario	Suggested Flags
Local dev server, few pages	--workers 4
Small production site (< 1k pages)	--workers 8 --delay 0.1
Large production site (10k+ pages)	--workers 8 --delay 0.3
Shared hosting, be conservative	--workers 2 --delay 1.0
Behind a CDN / Cloudflare	--workers 10 --delay 0.05

Notes:

• More workers than the server can handle will slow the crawl down (retries, timeouts) rather than speed it up.
• The delay is global, so --workers 32 --delay 1.0 still only sends ~1 request/second.
• A single requests.Session is reused, so TCP connection setup cost is paid once per host, not per request.

---

Troubleshooting

"No URLs discovered." The target may be a JavaScript-heavy SPA. This crawler only parses static HTML — it doesn't execute JS. Try viewing the page source; if there are no <a href> links in it, there's nothing for the crawler to find. Use a headless browser (Playwright, Selenium) if you need JS rendering.

"Crawl error: ..." lines in verbose mode Individual request failures are logged and skipped so one bad URL doesn't stop the crawl. Common causes:

• Connection reset — server rate-limiting you. Increase --delay.
• Timeout — increase --timeout.
• DNS errors — link points to a dead host.
• SSL errors — try --insecure (staging only) or fix the cert.

Crawl runs forever. Your site likely has unbounded query-string permutations (?page=N, calendar links, faceted search, etc.). Fixes:

• Use -m to cap total pages.
• Add specific query parameters to a denylist in canon().
• Use -d to limit depth.

Sitemap not found, but one exists. The script tries a fixed list of paths. Add your custom path to SITEMAP_PATHS at the top of the script, or pass --no-sitemap and point directly at the sitemap URL as the target.

Missing URLs from other subdomains. Pass --include-subdomains. Note that the script's registrable_host() is a simplification — see Limitations.

---

Limitations

• No JS rendering. Only static HTML <a href> links are discovered. SPAs and JS-injected navigation are invisible to this crawler.
• No robots.txt honoring. By design — this is for your own sites.
• No authentication. If pages require login, you'd need to inject cookies or auth headers into session before calling crawler.crawl(...).
• No POST/form handling. GET requests only.
• Simplified eTLD+1. registrable_host() handles the most common two-part TLDs but is not a full Public Suffix List. If you need perfect subdomain matching across exotic TLDs, replace it with tldextract or publicsuffix2.
• No rate-limit backoff. If the server returns 429 or 503, the script logs and moves on. Add exponential backoff in _fetch() if your target enforces limits.
• Query strings preserved verbatim. Sometimes this is what you want; sometimes it explodes the URL space. Filter via canon() as needed.

---

Extending the Script

Some common extensions and where to make them:

Filter out query strings entirely (aggressive deduplication):

# in canon()
return urlunparse(p._replace(path=path, params="", query="", fragment=""))

Ignore specific query parameters:

from urllib.parse import parse_qsl, urlencode

def canon(url):
    url, _ = urldefrag(url)
    p = urlparse(url)
    path = p.path or "/"
    drop = {"utm_source", "utm_medium", "utm_campaign", "session_id"}
    kept = [(k, v) for k, v in parse_qsl(p.query, keep_blank_values=True)
            if k not in drop]
    query = urlencode(kept)
    return urlunparse(p._replace(path=path, params="", query=query, fragment=""))

Add authentication cookies:

session = requests.Session()
session.cookies.set("session", "abc123", domain="mysite.com")
# then pass session to Crawler()

Retry transient failures with backoff:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(total=3, backoff_factor=0.5,
              status_forcelist=[429, 500, 502, 503, 504])
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))

Export as JSON or CSV: The discovered set is a plain set[str] in main() — pipe it through json.dump() or csv.writer instead of " ".join().

---

Security Notes

• Use only on sites you own or have written permission to crawl. The tool does not read robots.txt and will follow every link it can reach.
• TLS verification is on by default. --insecure exists for internal staging servers with self-signed certs — never use it against production.
• XML entity declarations are blocked in the sitemap parser to prevent billion-laughs / XXE-style attacks from a malicious sitemap.
• Sitemap recursion is depth-limited to prevent an adversarial sitemap index from making the crawler recurse indefinitely.
• Same-domain enforcement uses hostname comparison, not netloc, which prevents the classic https://evil.com@yoursite.com/ userinfo trick from leaking the crawl off-domain.
• No credentials are sent, and no form data is submitted. The crawler is read-only at the HTTP level.

---

License

MIT License — Copyright (c) 2026 evanesoteric

---

See Also

• requests documentation
• BeautifulSoup documentation
• Sitemaps protocol
• Public Suffix List — for a more robust subdomain matcher