Go Web Crawler

Small, single-binary crawler that stays on one domain and exports page data to CSV.

Requirements

• Go 1.25.5 (matches go.mod)

Quick start

Run directly with Go:

go run . https://example.com 5 100

Build a binary and run it:

go build -o crawler .
./crawler https://example.com 5 100

Usage

./crawler <URL> <maxConcurrency> <maxPages>

Arguments:

• URL: Starting page to crawl (must include scheme, e.g. https://)
• maxConcurrency: Maximum number of concurrent fetches
• maxPages: Maximum number of pages to crawl on the same domain

Output

The crawl writes report.csv to the repo root with these columns:

• page_url
• h1
• first_paragraph
• outgoing_link_urls (semicolon-separated)
• image_urls (semicolon-separated)

Behavior notes

• Only crawls pages on the same hostname as the start URL.
• Skips non-HTML responses.
• Stops when maxPages is reached.

Example

go run . https://golang.org 10 200

After the run finishes, check report.csv for the results.