European Energy Data Pipeline

CLI tool for downloading, storing, aligning, and visualising European power market data from ENTSO-E and Energidataservice.dk.

Setup

# Create and activate a virtual environment
uv venv .venv
source .venv/bin/activate    # Linux/macOS
# .venv\Scripts\activate     # Windows

# Install dependencies
uv pip install -r requirements.txt

API key

Get an ENTSO-E API key at https://transparency.entsoe.eu/ (register, then go to "My Account Settings" to generate a token).

Create a .env file in the project root (this file is gitignored):

ENTSOE_API_KEY=your-key-here

This works identically on Linux, macOS, and Windows — no shell config needed.

Usage

1. First-time fetch

Fetch all historical data (this takes a while — 10 zones, 17 border pairs, 7 EDS datasets):

python cli.py fetch --start 2020-01-01

Or start with a shorter range to verify everything works:

python cli.py fetch --start 2024-01-01

2. Align + derive

Creates hourly and 15-min aligned files, plus price averages, flow summaries, and generation mix percentages:

python cli.py align

3. Export CSVs

Export to a OneDrive/SharePoint-synced folder for colleagues:

python cli.py export --output-dir "/path/to/OneDrive/Shared Energy Data/"

Or just export locally:

python cli.py export

4. Monthly update routine

python cli.py update          # fetches last 35 days, merges with existing
python cli.py align
python cli.py export --output-dir "/path/to/OneDrive/Shared Energy Data/"

5. Browse datasets

python cli.py list-datasets
python cli.py list-datasets --category balancing

6. Plots

python cli.py plot --price-timeseries DK1 DK2 DE-LU --start 2024-01-01 --show
python cli.py plot --price-comparison DK1 DK2 --start 2024-01-01 --show
python cli.py plot --flow DK1-DE-LU --start 2024-01-01 --show
python cli.py plot --genmix DK1 --start 2024-06-01 --show
python cli.py plot --price-duration DK1 --start 2024-01-01 --show
python cli.py plot --price-heatmap DK1 --start 2024-01-01 --show

Plots are saved as PNG to plots/. Add --show to also display interactively.

Filtering

All commands support --source, --dataset, and --area flags to narrow scope:

python cli.py fetch --dataset DayAheadPrices --area DK1 --start 2024-01-01
python cli.py export --dataset DayAheadPrices --resolution hourly

Verbose output

Add -v before the subcommand for debug logging:

python cli.py -v fetch --start 2024-01-01

Configuration

Edit config.yaml to change:

• areas: Bidding zones to fetch (default: DK1, DK2, DE-LU, SE3, SE4, NO2, NL, PL, FR, BE)
• cross_border_pairs: Which border flow pairs to fetch
• default_start_date: Start date for initial historical fetch
• update_lookback_days: Days to look back on update (default: 35)
• export_dir: Path to OneDrive/SharePoint folder for CSV exports
• display_timezone: Timezone for CSV exports (default: Europe/Copenhagen)

Dataset registry

All known datasets are defined in datasets.yaml. To add a new ENTSO-E dataset, add a YAML block — no code changes needed.

Data flow

Sources (API) → data/raw/ (Parquet)
                    ↓
              data/aligned/hourly/    (resampled Parquet)
              data/aligned/quarterly/ (resampled Parquet)
                    ↓
              data/derived/           (aggregations Parquet)
                    ↓
              CSV export              (year-split CSVs for Excel)
              plots/                  (PNG charts)