Go Crawler

A lightweight and efficient web crawler written in Go. This tool crawls a given website, extracts internal links, and generates a report of all discovered pages.

Features

• Crawls web pages starting from a given base URL.
• Extracts and resolves internal links.
• Limits the number of concurrent crawling tasks to avoid overwhelming servers.
• Generates a detailed report of all crawled pages, sorted by the number of links pointing to each page.

Installation

1. Clone this repository:

   git clone https://github.com/MrKuros/go-crawler.git
   cd go-crawler
   

2. Build the project

   go build
   

3. Run the script

   ./crawler <baseURL> <maxConcurrentPages> <maxPages>
   

Example

./go-crawler "https://example.com" 3 25

Sample Output

=============================
REPORT for https://example.com
=============================
Found 3 internal links to https://example.com/page1
Found 2 internal links to https://example.com/page2

Dependencies

golang.org/x/net/html for parsing HTML.

Contributing

Contributions are welcome! Feel free to open issues or submit pull requests.

License

This project is licensed under the MIT License. See the LICENSE file for details.