Benchmarking deep reinforcement learning algorithms for autonomous energy management in off-grid solar + battery + diesel hybrid microgrids across sub-Saharan Africa. Using real climate data from NASA POWER, we train and evaluate six RL frameworks to find which policies best minimize diesel consumption and blackouts while maximizing renewable utilization under realistic African conditions.
An accompanying IEEE paper is available in paper/.
Many rural communities in sub-Saharan Africa rely on isolated microgrids — small solar + battery + diesel hybrid power systems not connected to a national grid. Operating these systems well requires deciding every 30 minutes: how much to charge or discharge the battery, and whether (and how hard) to run the diesel generator. Bad decisions waste fuel, cause blackouts, or wear out batteries early.
This project replaces hand-crafted rule-based controllers with reinforcement learning agents trained on five years of real hourly irradiance and temperature data from NASA POWER. We run a 150-run comparative study (6 algorithms × 5 locations × 5 seeds) to determine which RL framework performs best across a range of African climates — from the arid Sahel to equatorial West Africa to the Ethiopian highlands.
| Algorithm | Family | Mean Reward | Unmet Energy (kWh/yr) | Diesel (L/yr) | Battery EFC/yr |
|---|---|---|---|---|---|
| DDPG | Off-policy | −134,854 ± 23,029 | 7.5 ± 24.1 | 20,007 ± 3,241 | 2,335 ± 3,916 |
| SAC | Off-policy | −176,342 ± 3,080 | 0.0 ± 0.0 | 26,018 ± 1 | 3,905 ± 892 |
| TQC | Off-policy | −177,736 ± 3,527 | 8.2 ± 15.3 | 26,110 ± 312 | 3,407 ± 463 |
| RPPO | On-policy (RNN) | −195,672 ± 87,707 | 3,358 ± 5,001 | 13,394 ± 12,417 | 25 ± 3 |
| A2C | On-policy | −342,185 ± 104,438 | 6,489 ± 4,862 | 18,503 ± 24,671 | 19 ± 6 |
| PPO | On-policy | −354,415 ± 12,275 | 11,285 ± 612 | 165 ± 366 | 25 ± 0 |
Results averaged across 25 runs (5 locations × 5 seeds) per algorithm. Evaluation on held-out 2024 data.
Main findings:
- All three off-policy algorithms (DDPG, SAC, TQC) achieve near-zero unmet energy; no on-policy method does so reliably
- DDPG achieves 23% lower diesel consumption than SAC/TQC at equal reliability through an adaptive dispatch strategy
- SAC and TQC converge to a fixed-throttle satisficing strategy invariant across all five climate zones
- PPO collapses to a degenerate do-nothing attractor at 4/5 locations (11,285 kWh/yr unserved)
- A2C exhibits diesel reward hacking in one seed (68,328 L/yr) and high variance overall
- RPPO's LSTM memory genuinely helps when training converges, but ~50% collapse rate limits aggregate performance
- The replay buffer is the decisive structural advantage enabling off-policy methods to learn long-horizon credit assignment across 17,520-step annual episodes
Five sites spanning the climatic diversity of sub-Saharan Africa:
| City | Country | Latitude | Longitude | Climate |
|---|---|---|---|---|
| Niamey | Niger | 13.5116 | 2.1254 | Semi-arid Sahel |
| Dakar | Senegal | 14.7167 | −17.4677 | Sahel / Atlantic coast |
| Kumasi | Ghana | 6.7130 | −1.3470 | Tropical rainforest |
| Libreville | Gabon | 0.3901 | 9.4536 | Equatorial humid |
| Addis Ababa | Ethiopia | 9.0300 | 38.7400 | Highland tropical |
| Algorithm | Library | Notes |
|---|---|---|
| SAC | stable-baselines3 | Soft Actor-Critic (off-policy, entropy-regularized) |
| DDPG | stable-baselines3 | Deep Deterministic Policy Gradient (off-policy, deterministic) |
| TQC | sb3-contrib | Truncated Quantile Critics (off-policy, distributional) |
| PPO | stable-baselines3 | Proximal Policy Optimization (on-policy) |
| A2C | stable-baselines3 | Advantage Actor-Critic (on-policy) |
| RPPO | sb3-contrib | Recurrent PPO with LSTM memory (on-policy) |
- Provider: NASA POWER (Prediction of Worldwide Energy Resources)
- Variables:
ALLSKY_SFC_SW_DWN(global horizontal irradiance),T2M(2m temperature) - Resolution: Hourly from NASA, resampled to 30-minute intervals via linear interpolation
- Training window: 2019-01-01 to 2023-12-31 (5-year window)
- Evaluation window: 2024-01-01 to 2024-12-31 (held-out year, zero data leakage)
- Timestep: 30 minutes (
step_hours: 0.5) - Episode length: 365 days = 17,520 steps per year
- Training timesteps per run: 750,000 (~43 full episode-equivalents)
- Seeds: 0–4 (5 seeds per algo/location pair)
- Total runs: 150 (6 algos × 5 locations × 5 seeds)
Solar
- Panel area: 50 m²
- Efficiency: 18%
- Temperature coefficient: −0.35% per °C
- Soiling derate: 5%
Battery
- Capacity: 80 kWh
- Max charge/discharge: 20 kW
- Round-trip efficiency: 92%
- SOC operating band: 15% – 95%
- Initial SOC: 50%
Diesel Generator
- Rated power: 30 kW
- Fuel cost: $1.50/liter
- CO₂ emission factor: 2.68 kg/liter
- Startup penalty: 0.5 liters per start
- Minimum on-time: 2 hours; minimum off-time: 1 hour
- Specific fuel consumption curve: 0.33 L/kWh at 30% load → 0.26 L/kWh at 100% load
Load Profile
- Base consumption: 1.6 kWh/hour
- Evening peak multiplier: 2.0×
- Weekend multiplier: 1.15×
- Noise: ±20% Gaussian
Reward Function
- Blackout penalty: $30/kWh unserved
- Diesel cost weight: 1.0
- Battery cycle wear weight: 0.005
- Curtailment weight: 1.5
- SOC in-band bonus: 0.02
- Generator start penalty: 0.5/start
- Generator operation penalty: 0.05/hour
envs/microgrid_env.py— Gymnasium-compatibleMicrogridEnvwith continuous 2D action space (battery dispatch + diesel throttle), 6-dimensional observation (SOC, fuel, GHI, temperature, hour, load), diesel startup penalties, minimum on/off time constraints, battery rainflow cycle counting, inverter efficiency curve, and auto-refuel logic.
scripts/train_sac.py— SACscripts/train_ddpg.py— DDPGscripts/train_ppo.py— PPOscripts/train_a2c.py— A2Cscripts/train_rppo.py— Recurrent PPOscripts/train_tqc.py— TQC
scripts/eval/algo_registry.py— Algorithm registry, scenario parsing, climate inferencescripts/eval/collect_runs.py— Discovers completed run IDs underoutputs/metrics/scripts/eval/run_eval.py— Rolls out trained checkpoints on the held-out 2024 evaluation year, writes per-episode metrics toresults/raw/{run_id}/scripts/eval/build_tidy.py— Aggregates all raw results intoresults/tidy/master.parquet(+ CSV)scripts/validate/run_all.py— Runs 4 data-quality validation checks; exits non-zero on hard failuresscripts/stats/ci.py— Bootstrap and standard-error confidence interval utilitiesscripts/stats/export_tables.py— Exports per-metric statistics tables toreports/tables/scripts/plot/make_all.py— Generates all figure types (see below)scripts/run_all.py— Full pipeline orchestrator: eval → tidy → validate → stats → figures
| Figure ID | Description |
|---|---|
metric_comparison |
Bar chart comparing algorithms across all metrics |
learning_curves |
Training reward curves with smoothing |
seed_variability |
Variance across seeds per algo/location |
pareto |
Diesel cost vs. blackout Pareto frontier |
radar |
Multi-metric radar chart per algorithm |
scenario_sensitivity |
Performance sensitivity across locations |
cross_climate |
Cross-climate generalization heatmap |
structural_grouped |
Performance grouped by algorithm family |
time_series |
Episode time-series rollout (requires --rollout-logs) |
diesel_stability |
Diesel dispatch stability analysis (requires --rollout-logs) |
scripts/launch_study.py— Batch training launcher; iterates all 150 runs, skips completed checkpoints, writesstudy_manifest.jsonscripts/prefetch_locations.py— Pre-warms the NASA POWER cache for all 5 study locations before trainingMakefile— Convenience targets:all_figures,plots,validate,tidy
- Python 3.11.8
- Dependencies listed in
requirements.txt
# 1. Clone and create virtual environment
git clone <repo-url>
cd Microgrid-RL
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# 2. Install dependencies
pip install --upgrade pip
pip install -r requirements.txt
# 3. Pre-warm the NASA POWER data cache for all 5 study locations
# (downloads ~5 years of hourly data per location — do this before training)
python scripts/prefetch_locations.pypython scripts/launch_study.py --seeds 0 --total-timesteps 10000 --dry-run
# Remove --dry-run to actually train
python scripts/launch_study.py --seeds 0 --total-timesteps 10000# Launch all 150 runs (6 algos × 5 locations × 5 seeds)
# Skips runs that already have saved checkpoints — safe to re-run after interruption
python scripts/launch_study.py --total-timesteps 750000 --jobs 4# Evaluate trained checkpoints + build tidy dataset + validate + stats + all figures
python -m scripts.run_all --algorithms sac,ddpg,rppo,tqc,ppo,a2c --seeds 0,1,2,3,4 --eval-episodes 6
# Or just regenerate figures from an existing tidy dataset
make all_figures# Example: DDPG at Niamey, Niger
python -m scripts.train_ddpg \
--lat 13.5116 --lon 2.1254 \
--days 365 --seed 0 \
--total_timesteps 750000 \
--start 2019-01-01 --end 2023-12-31outputs/
metrics/ # Per-episode training metrics (JSON, one directory per run)
results/
raw/ # Per-run evaluation rollouts
tidy/
master.parquet # Aggregated tidy dataset (primary analysis input)
master.csv
reports/
figures/ # {figure_id}.png and .pdf
tables/ # {metric}_stats.csv
run_report.md # Summary report from last pipeline run
study_manifest.json # Status of all 150 training runs
Note: Trained model checkpoints (
outputs/models/*.zip) are excluded from this repository due to size (511 MB). Re-runlaunch_study.pyto reproduce them, or useresults/tidy/master.parquetdirectly for analysis.
The full research paper is available at paper/Benchmarking Deep RL for Off-Grid Hybrid Microgrids in Sub-Saharan Africa.pdf.
Leis, J., Nikookar, Y., Zhang, R., Sujay, S., Kisob, D., Bauerkoper, J., Zhou, C., Yu, J., Waseem, B., & Subzwari, T. (2026). Benchmarking Deep Reinforcement Learning for Autonomous Energy Management in Off-Grid Hybrid Microgrids Across Sub-Saharan Africa. University of Waterloo.
- Jordan Leis
- Yalda Nikookar
- Rachel Zhang
- Siri Sujay
- Devon Kisob
- Julian Bauerkoper
- Cilo Zhou
- Jennifer Yu
- Behzad Waseem
- Taha Subzwari