PV-Augmented NILM Datasets (ACM e-Energy 2026)

A comprehensive toolkit for synthesizing photovoltaic (PV) energy injection into public NILM datasets. This project enables researchers to create realistic scenarios of residential solar energy integration for evaluating NILM algorithms under renewable energy conditions.

If you use this toolkit in your research, please cite:

@misc{wang2025energy,
      title={Energy Injection Identification Enabled Disaggregation with Deep Multi-Task Learning}, 
      author={Xudong Wang and Guoming Tang and Junyu Xue and Srinivasan Keshav and Tongxin Li and Chris Ding},
      year={2025},
      eprint={2508.14600},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2508.14600}, 
}

Overview

This toolkit was developed to support the research presented in "[ACM e-Energy 2026] Energy Injection Identification Enabled Disaggregation with Deep Multi-Task Learning". It provides methods to:

• Fetch real-world solar irradiance data from NREL's National Solar Radiation Database
• Simulate realistic PV system output with temperature effects
• Integrate PV injection with all existing NILM datasets in NILMTK-compatible H5 formats (e.g. REDD, UK-DALE, etc.)
• Export synthesized datasets in multiple formats (CSV, PKL, H5)

Features

• Real Weather Data Integration: Fetches historical solar irradiance data from NREL API
• Realistic PV Simulation: Models temperature effects, inverter efficiency, and system losses
• Multi-Dataset Support: Compatible with REDD and UK-DALE datasets via NILMTK
• Flexible Export Options: Supports CSV, Pickle, and NILMTK-compatible H5 formats
• Reactive Power Simulation: Includes reactive power modeling for comprehensive analysis
• Batch Processing: Efficiently process multiple houses and time periods
• Advanced ML/DL Compatibility: Easily integrate with machine learning/deep learning analysis workflows like nilmtk, scikit-learn, hmmlearn, PyTorch, etc

Installation

Prerequisites

• Python 3.8 or higher
• NREL API key (free registration at https://developer.nrel.gov/signup/)

Install from Source

git clone https://github.com/MathAdventurer/PV-NILM-Synthesis.git
cd PV-NILM-Synthesis
pip install -r requirements.txt
python setup.py install

Dependencies

pip install nilmtk==0.4.3
pip install pandas numpy scipy matplotlib
pip install pyyaml tqdm scikit-learn
pip install requests

Test the Python Env and Installation

Run the provided test script to verify the installation:

python tests/test_pv_simulator.py
python tests/test_data_processor.py

Quick Start (Refer the tutorial notebook quickstart.ipynb)

1. Configure NREL API

Register for a free API key at NREL Developer Network

Create a configuration file:

# config/settings.py
NREL_API_KEY = "your_api_key_here"

2. Download NILM Datasets

REDD Dataset

• Download from: http://redd.csail.mit.edu/
• Convert to NILMTK format following: https://github.com/nilmtk/nilmtk/blob/master/docs/manual/user_guide/data.md

UK-DALE Dataset

• Download from: https://jack-kelly.com/data/
• Official repository: https://github.com/jack-kelly/ukdale

3. Basic Usage

from src.data_processor import process_dataset
from src.weather_api import fetch_nrel_data
from nilmtk import DataSet

# Load weather data
weather_data = fetch_nrel_data(
    lat=42.3601,  # Boston, MA for REDD
    lon=-71.0589,
    year=2011
)

# Load NILM dataset
redd = DataSet('path/to/redd.h5')

# Process with PV injection
data, train, test = process_dataset(
    dataset_name='REDD',
    dataset=redd,
    building_number=1,
    appliances=['microwave', 'fridge', 'dish washer', 'washing machine'],
    train_start_str='2011-04-19',
    train_end_str='2011-05-03',
    test_start_str='2011-05-04',
    test_end_str='2011-05-11',
    weather_data=weather_data,
    pv_capacity=2000  # 2kW system
)

Synthesized Dataset Statistics In Our Paper

Our pre-generated datasets (Our paper used) include PV injection for the following configurations:

Dataset Configuration Table

Dataset	House	Period	Appliances	PV Capacity	Location
REDD					Boston, MA (42.36°N, 71.06°W)
	House 1	2011-04-19 to 2011-05-11	Microwave, Fridge, Dishwasher, Washing Machine	2kW
	House 2	2011-04-19 to 2011-04-29	Microwave, Fridge, Dishwasher	2kW
	House 3	2011-04-19 to 2011-04-29	Microwave, Fridge, Dishwasher, Washing Machine	2kW
UK-DALE					London, UK (51.51°N, 0.13°W)
	House 1	2013-04-01 to 2013-04-21	Kettle, Microwave, Fridge, Dishwasher, Washing Machine	2kW
	House 2	2013-06-01 to 2013-06-21	Kettle, Microwave, Fridge, Dishwasher, Washing Machine	2kW

Key Statistics (Simulated 2kW PV System)

REDD House 1

• Aggregate Power: Max 5036.50W, Mean 120.82W, Std 374.56W
• Micro-inverter: Max 1330.46W, Mean 28.21W, On-time 38.23%
• Training Period: 2011-04-19 to 2011-05-03 (3-day splits)
• Testing Period: 2011-05-04 to 2011-05-11 (1-week)

UK-DALE House 1

• Aggregate Power: Max 5432.00W, Mean 118.77W, Std 391.46W
• Micro-inverter: Max 1289.01W, Mean 33.00W, On-time 18.92%
• Training Period: 2013-04-01 to 2013-04-14 (2-week)
• Testing Period: 2013-04-15 to 2013-04-21 (1-week)

Download Pre-generated Datasets

Due to file size limitations of Github, the synthesized datasets used in our paper are available at:

• Google Drive: Link
• Hugging Face: Dataset Card Link

The download includes:

• US_Weather.pkl: Pre-fetched NREL weather data (2010-2020)
• Synthesized datasets for all houses which used in our paper in CSV and PKL formats.

Advanced Usage (Refer the tutorial notebook advanced_usage.ipynb)

Custom PV System Parameters

from src.pv_simulator import simulate_pv_output_advanced

pv_output = simulate_pv_output_advanced(
    weather_data,
    system_capacity=3000,        # 3kW system
    temperature_coefficient=-0.004,  # Custom temperature coefficient
    inverter_efficiency=0.97,     # High-efficiency inverter
    tilt_angle=35,               # Panel tilt angle
    azimuth=180,                 # South-facing
    system_losses=0.10           # 10% system losses
)

Batch Processing Multiple Houses

from scripts.batch_synthesis import BatchProcessor

processor = BatchProcessor()
processor.process_all_redd_houses(
    weather_data=weather_data,
    pv_capacities=[1000, 2000, 3000],  # Multiple PV sizes
    output_dir='data/processed/'
)

Export to NILMTK H5 Format

from src.export_utils import to_nilmtk_h5 as export_to_nilmtk_h5

export_to_nilmtk_h5(
    data=synthesized_data,
    output_path='synthesized_redd_pv.h5',
    dataset_name='REDD_PV',
    building_id=1,
    timezone='US/Eastern'
)

API Reference

Weather Data Fetching

fetch_nrel_data(lat, lon, year, api_key=None, attributes=None)

Fetches solar irradiance data from NREL NSRDB.

Parameters:

• lat (float): Latitude of location
• lon (float): Longitude of location
• year (int): Year of data to fetch
• api_key (str): NREL API key
• attributes (list): Solar attributes to fetch (default: GHI, DNI, DHI, Temperature, Wind Speed)

PV Simulation

simulate_pv_output(weather_data, system_capacity=120, temperature_coefficient=-0.005)

Simulates PV system output based on weather conditions.

Parameters:

• weather_data (DataFrame): Weather data with GHI and Temperature
• system_capacity (float): PV system capacity in Watts
• temperature_coefficient (float): Temperature coefficient (per °C)

Data Processing

process_dataset(dataset_name, dataset, building_number, appliances, ...)

Processes NILM dataset with PV injection.

Parameters:

• dataset_name (str): 'REDD' or 'UKDALE'
• dataset (nilmtk.DataSet): NILMTK dataset object
• building_number (int): Building/house number
• appliances (list): List of appliances to include
• train_start_str (str): Training period start date
• train_end_str (str): Training period end date
• test_start_str (str): Testing period start date
• test_end_str (str): Testing period end date
• weather_data (DataFrame): Weather data for PV simulation
• pv_capacity (float): PV system capacity in Watts

License

This project is licensed under the MIT License, see the LICENSE file for details.

Contact

• Email: (xudongwang@link.cuhk.edu.cn)

Acknowledgments

• National Renewable Energy Laboratory (NREL) for providing the NSRDB API
• NILMTK team for the excellent toolkit and dataset support
• REDD and UK-DALE dataset creators for making their data publicly available