Plug-and-play FastAPI backend for data scrapers. Spin up a scraping API in minutes with built-in async connection pooling, automatic header rotation, and rate limiting.
- Async connection pooling — aiohttp-based, configurable per-host limits, DNS caching, keep-alive
- Header rotation — randomizes User-Agent, Accept-Language, Referrer per request
- Rate limiting — token bucket per domain + global, configurable RPS and burst
- Auto-retry — exponential backoff on connection failures
- Batch scraping — concurrent multi-URL with semaphore-bounded concurrency
- Base scraper class — subclass
BaseScraper, overrideparse(), done - Python 3.13 —
asyncio.TaskGroupfriendly, modern type syntax
# Clone
git clone https://github.com/TunahanB/scraper-api
cd scraper-api
# Install
pip install -e .
# Configure
cp .env.example .env
# Run
uvicorn app.main:app --reloadAPI docs at http://localhost:8000/docs.
curl -X POST http://localhost:8000/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'Response:
{
"url": "https://example.com",
"status_code": 200,
"content_type": "text/html",
"body": "<html>...</html>",
"body_size": 1256,
"elapsed_ms": 312.5,
"scraped_at": "2026-07-08T12:00:00Z"
}curl -X POST http://localhost:8000/scrape/batch \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com", "https://httpbin.org/ip"], "max_concurrency": 3}'curl http://localhost:8000/health
curl http://localhost:8000/statsCreate scrapers/myscraper.py:
from typing import Any
import aiohttp
from app.core.scraper import BaseScraper
class MyScraper(BaseScraper):
async def parse(self, response: aiohttp.ClientResponse, body: str) -> Any:
from bs4 import BeautifulSoup
soup = BeautifulSoup(body, "lxml")
return {
"title": soup.title.string if soup.title else None,
"links": [a.get("href") for a in soup.find_all("a")],
}Register it in app/routers/scrape.py:
from scrapers.myscraper import MyScraper
SCRAPERS["default"] = MyScraper()All settings via environment variables (see .env.example):
| Variable | Default | Description |
|---|---|---|
POOL_MAX_CONNECTIONS |
100 | Max total connections |
POOL_MAX_PER_HOST |
10 | Max connections per host |
POOL_MAX_RETRIES |
3 | Retry count on failure |
RATE_RPS |
5 | Global requests per second |
RATE_BURST |
10 | Burst capacity |
DEBUG |
false | Enable hot-reload + verbose logs |
app/
├── main.py # FastAPI app, lifespan, routes
├── config.py # Dataclass config, env loading
├── core/
│ ├── pool.py # aiohttp connection pool
│ ├── headers.py # Header rotator
│ ├── ratelimit.py # Token bucket rate limiter
│ └── scraper.py # BaseScraper (override parse())
├── routers/
│ ├── scrape.py # Scraper registry
│ └── health.py # /health, /stats
└── models/
└── schemas.py # Pydantic request/response models
scrapers/
└── example.py # Example scraper (copy to start your own)
- Python 3.13+
- aiohttp
- FastAPI
- uvicorn