SiteBox

A fast, concurrent website downloader for offline use. Written in Go.

Features

• Recursive crawling - Follows links up to configurable depth (default: 50)
• Domain filtering - Only downloads URLs under specified domain
• Blacklist filtering - Exclude URLs matching patterns (default: GitHub)
• File type filtering - Downloads JS, CSS, fonts by default
• Image support - Optional image downloading with deduplication
• ASP.NET support - Automatically converts .asp files to .html
• Concurrent downloads - Configurable parallelism (default: 5)
• Retry logic - Automatic retries for failed requests
• Progress bar - Visual progress indicator

Installation

From source

go install sitebox/cmd/sitebox@latest

Build from source

git clone <repository-url>
cd sitebox
make build

The binary will be in build/sitebox.

Cross-platform builds

make build-all

Builds for Linux, macOS, and Windows (amd64 and arm64).

Usage

Basic download

sitebox download -d example.com -u https://example.com/docs -o ./mysite

Download with images

sitebox download \
  -d example.com \
  -u https://example.com/docs/getting-started \
  -o ./docs \
  --images

Verbose output

sitebox download -d example.com -u https://example.com -o ./site -v

Advanced options

sitebox download \
  -d example.com \
  -u https://example.com \
  -o ./site \
  --concurrent 10 \
  --retries 5 \
  --depth 100 \
  --images

CLI Reference

Commands

Command	Description
download	Download a website for offline use
help	Help about any command
version	Print version information

Download Flags

Flag	Short	Required	Default	Description
--domain	-d	Yes	-	Domain to filter URLs
--url	-u	Yes	-	Starting URL for crawling
--output	-o	Yes	-	Output directory for downloaded files
--suffix	-	No	site	Suffix appended to output directory
--verbose	-v	No	false	Print every downloaded URL
--images	-	No	false	Download images (PNG, SVG, JPG, etc.)
--depth	-	No	50	Maximum crawling depth
--concurrent	-	No	5	Number of concurrent downloads
--retries	-	No	3	Number of retries for failed requests
--progress	-	No	true	Show progress bar

Architecture

sitebox/
├── cmd/
│   └── sitebox/
│       └── main.go              # CLI entry point
├── internal/
│   ├── cli/
│   │   └── cli.go               # Cobra CLI implementation
│   ├── downloader/
│   │   ├── config.go            # Configuration
│   │   ├── downloader.go        # Orchestration
│   │   └── errors.go            # Error definitions
│   ├── scraper/
│   │   └── scraper.go           # Colly-based scraper
│   ├── filter/
│   │   └── filter.go            # URL filtering logic
│   ├── resource/
│   │   └── manager.go           # Resource deduplication
│   └── transformer/
│       └── transformer.go       # Filename transformations
├── pkg/
│   └── downloader/
│       └── api.go               # Public API for GUI integration
├── go.mod
├── go.sum
├── Makefile
└── README.md

Design Principles

1. Separation of concerns - CLI, business logic, and scraping are separate
2. Dependency injection - Components are loosely coupled
3. Interface-based design - Easy to mock for testing
4. Thread safety - Concurrent access is handled with mutexes

GUI Integration

The pkg/downloader package provides a public API for GUI integration:

import "sitebox/pkg/downloader"

// Simple usage
err := downloader.Quick("example.com", "https://example.com/docs", "./output")

// Fluent API
client := downloader.NewClient()
err := client.
    SetDomain("example.com").
    SetURL("https://example.com/docs").
    SetOutput("./output").
    SetImages(true).
    SetConcurrent(10).
    Download()

// With progress callback (for future implementation)
client.OnProgress(func(event downloader.Event) {
    fmt.Printf("Downloaded: %s\n", event.URL)
})

Development

Prerequisites

• Go 1.21 or later
• Make (optional, for using Makefile)

Running tests

make test
# or
go test -v ./...

Running tests with coverage

make test-coverage

Formatting code

make fmt

Building

make build        # Current platform
make build-all    # All platforms
make install      # Install to $GOPATH/bin

License

ISC