feat(engine): implement async httpx range download engine and worker pool

[archittmittal]

Closes #41

Why

Right now reliadl download crashes on every URL, both in the v0.3.0 PyPI release and on master:

AttributeError: 'StateManager' object has no attribute 'initialize'

It was a placeholder, and resume only printed a message. This PR adds the parallel range-download engine and connects both commands to it.

What changed

reliadl/download_engine.py (new)

• Probe: sends a HEAD request for size, Accept-Ranges and ETag. If HEAD is refused or ranges aren't advertised, it falls back to a one-byte Range: bytes=0-0 GET and reads the total from Content-Range.
• Worker pool: fixed-size chunks go into an asyncio.Queue, and max_parallel_workers workers share one pooled httpx.AsyncClient. HTTP/2 is used when h2 is installed. Accept-Encoding: identity keeps byte offsets meaningful.
• Writes: chunks stream straight into a pre-allocated <output>.part using the existing SparseFileWriter (pwrite). Each chunk is SHA-256 hashed as it streams.
• Safety checks: chunk requests send If-Range with a strong ETag (or Last-Modified), so a file that changes mid-download is caught. The engine also checks Content-Range, rejects over-long responses, and detects truncated bodies.
• Retries: timeouts, resets, 408/429/5xx and truncation are retried with exponential backoff plus jitter, and Retry-After is respected. Errors that can't succeed on retry (4xx, 412, range ignored) stop the run straight away.
• Rate limit: optional cap using the existing TokenBucketRateLimiter.
• Checkpointing: after every chunk, the .part file is fsynced and then the state is saved atomically. The state file never claims bytes that aren't on disk yet.
• Completion: the whole file is hashed and checked against the expected hash if one was given, .part is atomically renamed to the output, and the state file and empty state directory are removed.

Graceful shutdown and resume

• DownloadEngine.request_shutdown() is hooked to SIGINT and SIGTERM. Cancelling the task (Ctrl+C on Windows) does the same. Workers are cancelled, in-flight chunks go back to PENDING, state is saved as CANCELLED, and DownloadCancelledError is raised. The CLI prints the resume command and exits with status 130.
• resume probes the server again and refuses if the size or ETag changed. It re-hashes every chunk marked complete in .part before trusting it, then fetches only what's missing.

CLI

• download gets new options: --expected-hash/--sha256, --workers/-j, --chunk-size, --limit-rate and --config. resume accepts the same engine options.
• A progress line shows on an interactive terminal.

Other

• New exceptions: RangeNotSupportedError and DownloadCancelledError.
• httpx>=0.28.1 is now a dependency, with optional extras reliadl[http2] and reliadl[socks].
• docs/CLI_GUIDE.md is updated.

Out of scope

• Single-stream fallback for servers without range support is feat(engine): implement single-stream fallback engine for non-Range servers #42. These servers now get a clear RangeNotSupportedError instead of a crash, and feat(engine): implement single-stream fallback engine for non-Range servers #42 can catch that error to fall back.
• Rich progress UI is feat(cli): implement Click CLI interface with real-time rich progress display #43, and tracing is feat(telemetry): implement OpenTelemetry distributed tracing spans for workers and chunks #38.
• --adachunk / --whittle are still accepted but don't do anything yet, same as before. The engine uses fixed-size chunks.

Testing

• tests/unit/test_download_engine.py adds 20 tests that run against a real local range server. They cover:
  • byte-identical output
  • concurrency between 2 and the configured limit
  • HEAD-rejected fallback
  • a server without range support
  • 503/429 retries
  • a 404 that abandons the run, then a successful resume
  • a shutdown checkpoint that keeps only completed chunks
  • resume fetching only the missing bytes
  • a corrupted chunk being fetched again
  • rejection when the remote ETag changed
  • task cancellation
  • the CLI download, resume and invalid-configuration paths
• Full suite: 985 tests pass locally on macOS with Python 3.11. ruff check --select=E,F is clean.
• Tried by hand against https://proof.ovh.net/files/10Mb.dat with 4 workers and 1 MB chunks. The SHA-256 matched curl | shasum. A SIGINT in the middle exited 130 with 1 of 10 chunks checkpointed, and reliadl resume finished with the same hash.
• Built a wheel from a clean clone and installed it in a fresh venv: it contains only the reliadl package, and a download worked.

🤖 Generated with Claude Code