ScrapeFlow — Automated Web Data Collection & Monitoring

A production-style Python portfolio project for automated web data collection, monitoring, change detection, PostgreSQL storage, scheduled updates, and business-ready Excel/CSV reporting.

Demo Results

• 1,000 products collected across 50 paginated pages
• Requests + BeautifulSoup and Playwright engines verified
• PostgreSQL persistence with current state and historical snapshots
• Duplicate-safe repeated runs: 1,000 unchanged records detected correctly
• Change detection verified with new / changed / unchanged classification
• Scheduled scraping verified with APScheduler
• Webhook notification verified on detected changes
• CSV and Excel exports generated successfully
• Streamlit dashboard with monitoring KPIs, run history, filtering, and downloads
• Automated tests: 3 passed

Dashboard Preview

Monitoring Overview

Change Detection

Catalog & Export

A portfolio-ready Python project that collects structured product data from public web pages, validates and deduplicates it, stores current state and history in PostgreSQL, detects new/changed records, exports CSV/Excel reports, and exposes a Streamlit monitoring dashboard.

The demo source is Books to Scrape, a website intentionally created for scraping practice. The scraper is configurable, rate-limited, checks robots.txt, and never attempts to bypass authentication, CAPTCHAs, paywalls, or access controls.

What this project demonstrates

• Python web scraping with Requests + BeautifulSoup
• Optional browser rendering with Playwright
• Pagination and configurable crawl limits
• Data cleaning and validation with Pandas
• Deterministic deduplication
• PostgreSQL persistence via SQLAlchemy 2.x + psycopg 3
• Snapshot history and new / changed / unchanged detection
• CSV and Excel exports
• Error logging and run audit trail
• Optional webhook notification when new or changed records are detected
• Scheduled collection with APScheduler 3.x or GitHub Actions
• Streamlit dashboard with KPIs, run history, filters, change log, and downloads
• Unit tests for parsing and change detection

Architecture

Public pages
   |
   v
Scraper engine
(Requests/BS4 or Playwright)
   |
   v
Parser -> validation -> normalization -> deduplication
   |
   v
Change detector
   |----------------------|
   v                      v
PostgreSQL              Webhook
current + history       notification
   |
   |------------|
   v            v
CSV / Excel   Streamlit dashboard

Project structure

ScrapeFlow/
├─ app.py
├─ docker-compose.yml
├─ pyproject.toml
├─ requirements.txt
├─ .env.example
├─ .github/workflows/scrape.yml
├─ src/scrapeflow/
│  ├─ cli.py
│  ├─ config.py
│  ├─ db.py
│  ├─ exporters.py
│  ├─ logging_config.py
│  ├─ notifications.py
│  ├─ pipeline.py
│  ├─ scheduler.py
│  └─ scrapers/
│     ├─ parser.py
│     ├─ playwright_scraper.py
│     └─ requests_scraper.py
└─ tests/
   ├─ fixtures/books_page.html
   ├─ test_change_detection.py
   └─ test_parser.py

1. Local setup

python -m venv .venv

Windows PowerShell:

.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e .
playwright install chromium
copy .env.example .env

macOS/Linux:

source .venv/bin/activate
pip install -r requirements.txt
pip install -e .
playwright install chromium
cp .env.example .env

Playwright requires its Python package plus installed browser binaries. The official Playwright documentation uses the same two-step installation pattern.

2. Start PostgreSQL

docker compose up -d postgres

The included .env.example already points to the Docker database:

postgresql+psycopg://scrapeflow:scrapeflow@localhost:5432/scrapeflow

If DATABASE_URL is not set, the application falls back to local SQLite for quick demonstrations and tests.

3. Run the pipeline

Fast HTTP mode:

python -m scrapeflow.cli run --engine requests --max-pages 5

Browser-rendered mode:

python -m scrapeflow.cli run --engine playwright --max-pages 5

Full demo crawl:

python -m scrapeflow.cli run --engine requests --max-pages 50

4. Export current data

python -m scrapeflow.cli export

Files are written to exports/:

• scrapeflow_catalog.csv
• scrapeflow_catalog.xlsx
• scrapeflow_changes.csv

5. Launch the dashboard

streamlit run app.py

Dashboard sections:

• total products
• in-stock products
• average price
• records changed in the latest run
• collection run history
• current catalog
• latest changes
• CSV/Excel download buttons

6. Scheduled collection

Local scheduler:

python -m scrapeflow.scheduler

The interval is controlled by SCHEDULE_MINUTES in .env.

A GitHub Actions workflow is also included. For hosted runs, configure repository secrets such as DATABASE_URL and optionally WEBHOOK_URL.

7. Notifications

Set WEBHOOK_URL in .env. After a successful run, ScrapeFlow sends a small notification only when new or changed records are found. If no webhook is configured, the event is written to the application log instead.

8. Tests

pytest -q

Tests use local HTML fixtures and do not require a live website.

Data model

scrape_runs

One row per collection attempt: timestamps, scraper engine, page/item counts, new/changed counts, and error message.

items

Latest known state of each product. A unique key on (source, external_id) prevents duplicate current records.

item_history

Append-only snapshots for newly discovered or changed products. The change_type field records new or changed.

Portfolio talking points

One-line description

Automated Python web-data pipeline with Requests/BeautifulSoup, Playwright, PostgreSQL, change detection, Excel export, scheduled collection, and a Streamlit monitoring dashboard.

Cover-letter sentence

I built a Python pipeline that collects paginated public web data, validates and deduplicates it, stores current and historical records in PostgreSQL, detects changes, and exports structured Excel/CSV reports.

Responsible scraping design

ScrapeFlow deliberately does not include CAPTCHA solving, login bypasses, proxy rotation for evading blocks, fingerprint spoofing, or other techniques intended to defeat a site's access controls. Before adapting this template to another domain, check the website's terms, robots.txt, applicable rate limits, and whether an official API is a better option.