PPO Hyperparameter Robustness under Environment Shifts

This project investigates whether PPO hyperparameter configurations with similar performance in the nominal training environment can behave differently when the environment dynamics change.

The project was developed as part of the Reinforcement Learning course at Leibniz Universität Hannover.

Research Question

Do PPO hyperparameter configurations with similar nominal performance differ in robustness to unseen dynamics shifts?

The experiments use PPO on Pendulum-v1. Policies are trained only in the nominal environment and are then evaluated without retraining under different gravity values.

The focus is therefore on zero-shot policy robustness conditioned on the chosen hyperparameters.

---

Experimental Setup

Environment

• Gymnasium Pendulum-v1
• Nominal gravity: g = 10
• Low-gravity shift: g = 7
• High-gravity shift: g = 13
• Maximum torque: 2.0

Only gravity is changed during the robustness evaluation.

Algorithm

PPO from Stable-Baselines3 is used with an MLP policy.

Four hyperparameters are varied:

Hyperparameter	Search Space
Learning rate	Log-uniform [1e-5, 3e-3]
Clip range	Uniform [0.1, 0.3]
Entropy coefficient	Log-uniform [1e-5, 1e-2]
Number of rollout steps	{256, 512, 1024, 2048}

All other PPO parameters remain fixed.

Random Search

• 40 hyperparameter configurations
• 3 training seeds per configuration
• Approximately 100,000 training timesteps per run
• 120 trained policies in total
• Random-search seed: 42

Each policy is evaluated on the nominal environment after training.

---

Configuration Selection

The configurations are ranked according to their mean nominal evaluation return across the three training seeds.

The best configuration achieved a mean return of approximately:

Config 2: -286.80

A configuration was considered similarly high-performing when its mean nominal return was within 100 return points of the best configuration.

This selected:

Config 2
Config 33

The selection was performed using only nominal performance, before evaluating robustness under gravity shifts.

---

Robustness Evaluation

The six selected policies

2 configurations × 3 training seeds

are evaluated without additional training under:

g = 7   Low gravity
g = 10  Nominal gravity
g = 13  High gravity

Each policy is evaluated for 20 episodes using deterministic policy actions.

The same evaluation seed is used across policies and environments to make the evaluation conditions comparable.

Main Result

The selected configurations show different and direction-dependent responses to the gravity shifts.

Config	g = 7	g = 10	g = 13
Config 2	-268.79 ± 46.36	-286.80 ± 31.11	-904.62 ± 111.93
Config 33	-554.22 ± 176.42	-342.66 ± 82.95	-737.37 ± 290.82

Values are mean return ± standard deviation across the three training seeds.

Although the configurations have relatively similar nominal performance, their robustness profiles differ depending on the direction of the environment shift.

The results therefore suggest that, in this experiment, nominal performance alone does not fully characterize how a PPO policy behaves under unseen dynamics changes.

Because only three training seeds and one environment parameter are evaluated, the results should be interpreted descriptively rather than as a general statistical claim about PPO robustness.

---

Additional Analyses

Learning Curves

Evaluation checkpoints recorded during training are used to compare learning behavior and sample efficiency.

Both selected configurations show broadly similar learning behavior, with substantial improvement occurring late in training.

Hyperparameter Importance

An exploratory fANOVA analysis is performed on the 40 nominal configurations.

Estimated relative importance for nominal performance:

Hyperparameter	Importance
Learning rate	0.667
n_steps	0.189
Clip range	0.128
Entropy coefficient	0.016

These values describe importance only within the sampled search space and for nominal Pendulum performance. They are not estimates of hyperparameter importance for robustness.

---

Project Structure

.
├── README.md
├── requirements.txt
├── requirements-lock.txt
│
├── models/
│   └── main_search/
│       └── trained PPO policies
│
├── results/
│   └── main_search/
│       ├── analysis/
│       │   ├── config_summary.csv
│       │   └── figures/
│       │
│       ├── evaluations/
│       │   └── learning-curve evaluation data
│       │
│       ├── hyperparameter_importance/
│       │   └── fANOVA results
│       │
│       ├── learning_curves/
│       │   └── learning-curve analysis and figures
│       │
│       ├── robustness/
│       │   ├── analysis/
│       │   └── figures/
│       │
│       ├── runs/
│       │   └── results of all 120 training runs
│       │
│       ├── random_search_metadata.json
│       └── search_configs.json
│
└── src/
    ├── envs.py
    ├── run_experiment.py
    ├── random_search.py
    ├── analyze_search.py
    ├── evaluate_shifts.py
    ├── analyze_robustness.py
    ├── analyze_learning_curves.py
    ├── analyze_hyperparameter_importance.py
    ├── plot_nominal_ranking.py
    ├── plot_robustness_styled.py
    ├── plot_robustness_heatmap.py
    └── plot_learning_curves_ribbon.py

---

Installation

Create and activate a virtual environment:

python3 -m venv .venv
source .venv/bin/activate

Install the required packages:

.venv/bin/python -m pip install -r requirements.txt

---

Running the Experiments

All commands should be executed from the project root.

1. Run the random search

.venv/bin/python src/random_search.py

This trains the 40 configurations with three training seeds each and stores the resulting models and evaluation data.

2. Aggregate nominal performance

.venv/bin/python src/analyze_search.py

3. Evaluate selected policies under gravity shifts

.venv/bin/python src/evaluate_shifts.py

4. Analyze robustness

.venv/bin/python src/analyze_robustness.py

5. Analyze learning curves

.venv/bin/python src/analyze_learning_curves.py

6. Estimate hyperparameter importance

.venv/bin/python src/analyze_hyperparameter_importance.py

---

Figures

Figures can be reproduced with:

.venv/bin/python src/plot_nominal_ranking.py
.venv/bin/python src/plot_robustness_styled.py
.venv/bin/python src/plot_robustness_heatmap.py
.venv/bin/python src/plot_learning_curves_ribbon.py

The main visualizations include:

• nominal performance ranking of all 40 configurations
• robustness under gravity shifts
• change from nominal performance
• learning curves of the selected configurations
• fANOVA hyperparameter importance

---

Reproducibility

The repository stores:

• all sampled hyperparameter configurations
• random-search metadata
• training seeds
• individual run results
• trained policies
• evaluation checkpoints
• robustness evaluation results
• aggregated analysis tables
• figures used for the final analysis

For exact reproduction of the environment used for the final experiments:

.venv/bin/python -m pip install -r requirements-lock.txt

This allows the experiment and the reported results to be traced back to the corresponding configurations and training seeds.

---

Main Takeaway

Similar nominal PPO performance did not imply identical robustness.

In the tested Pendulum setting, two high-performing hyperparameter configurations responded differently to lower and higher gravity. This highlights the value of evaluating trained RL policies under relevant environment shifts rather than relying only on nominal performance.