Deep RL Project (EE-568, EPFL)

Re-implementation and comparison of DQN, PPO, TD3, and SAC on classic-control Gymnasium environments. From-scratch implementations using only gymnasium, torch, numpy, and the project's common/ utilities (no stable-baselines3, rllib, tianshou, etc.).

The deliverable writeup is report/report.md; the engineering diary (bugs, hparam sensitivity, env quirks) is report/diary.md.

Results (5 seeds per setting)

Algo	Env	Threshold	Mean final	Passed
DQN	CartPole-v1	>475	500.0	yes
DQN	Acrobot-v1	>-100	-82.0	yes
PPO	CartPole-v1	>475	500.0	yes
PPO	Pendulum-v1	>-200	-145.6	yes
TD3	Pendulum-v1	>-200	-174.6	yes
SAC	Pendulum-v1	>-200	-167.6	yes

See report/results_table.md for per-seed values and last-5 metrics. Per-algorithm curves and cross-algorithm comparisons are in report/figures/.

Setup

python -m venv .venv
source .venv/bin/activate    # or .venv\Scripts\activate on Windows
pip install -r requirements.txt

Reproduce everything

make all     # 5 seeds for every canonical config, plus ablations and plots

Or run pieces individually: make dqn, make ppo, make td3, make sac, make ablations, make report. On Windows without GNU Make, run the equivalent python -m scripts.run_seeds ... commands directly.

Run a single seed

python -m scripts.train --config configs/dqn_cartpole.yaml --seed 0

Logs land in results/<algo>_<env>_seed<N>_<timestamp>/ as train_log.csv, eval_log.csv, and a snapshot of the config.

Run multiple seeds

python -m scripts.run_seeds --config configs/dqn_cartpole.yaml --seeds 0 1 2 3 4

Plotting

# Regenerate every figure and the results table from the current results/.
python -m scripts.make_report
python -m scripts.ablation_plots

# Or for a one-off comparison plot:
python -m scripts.plot_results \
    --runs "results/ppo_pendulumv1_*:PPO" "results/sac_pendulumv1_*:SAC" \
    --output report/figures/custom.pdf

Algorithm / environment matrix

	CartPole-v1	Acrobot-v1	Pendulum-v1
DQN	tested	tested	not applicable (continuous)
PPO	tested	(works)	tested (n_envs=4)
TD3	not applicable (discrete)	not applicable	tested
SAC	not applicable (discrete)	not applicable	tested

PPO is the only algorithm that bridges discrete and continuous; the report uses CartPole-v1 (discrete bridge) and Pendulum-v1 (continuous bridge) for the 4-algorithm comparison.

Project layout

deep_rl_project/
├── algorithms/      # DQN, PPO, SAC, TD3 implementations
├── common/          # Shared utilities (buffers, networks, logger, eval)
├── configs/         # YAML configs per (algo, env) + ablation variants
├── scripts/         # train, run_seeds, plot_results, make_report, ablation_plots
├── results/         # Run logs (gitignored)
└── report/          # report.md, diary.md, results_table.md, figures/