Pac-Man DQN

A Deep Q-Network agent that learns to play Pac-Man from scratch via reinforcement learning. No hand-coded rules, no hardcoded strategies — just a neural network, a reward signal, and a lot of episodes.

---

Table of Contents

• What is this?
• Training & Results
• Model Architecture
• Model Parameters
• How to Use
• Repository Structure
• Training Video
• License

---

What is this?

A custom Pac-Man environment built on top of Gymnasium and trained with a Deep Q-Network from Stable Baselines3. The agent sees the game as a stack of binary feature planes — walls, pellets, ghosts, Pac-Man itself, and a decaying visit-count heatmap — and outputs one of four actions (up, down, left, right).

The game runs on a 15×15 toroidal grid (wrapping edges). Pellets spawn randomly, ghosts chase Pac-Man using a simple greedy heuristic, and the agent has 3 lives per episode. The whole thing is rendered in the terminal with Rich — no external game engine needed.

---

Training & Results

The model was trained for 1,000,000 timesteps with checkpoints saved every 50,000 steps. Here's how performance evolved.

First, it started moving randomly. After that, it learned to avoid the ghosts, but didn't start chasing pellets until step 250K more or less. Around step 400K, it learned loop avoidance, triggering a radical increment in the score, going from 5 to 30 eaten pellets in a fixed time.

The agent went from random flailing to genuinely chasing pellets and avoiding ghosts over the course of training. The biggest jumps happen between 350K and 400K steps — that's where the spatial reasoning clicks.

---

Model Architecture

The policy uses a custom CNN feature extractor fed into the standard DQN MLP head:

Input: (5, 15, 15) — 5 binary/continuous feature planes
  │
  ├─ Conv2d(5 → 32, kernel=3×3, stride=1, padding=1) + ReLU
  ├─ Conv2d(32 → 64, kernel=3×3, stride=1, padding=1) + ReLU
  ├─ Flatten
  ├─ Linear(64 × 15 × 15 → 256) + ReLU
  │
  └─ DQN MLP head (256 → 256 → 4)
        │
        Output: Q-values for [Up, Down, Left, Right]

The CNN preserves spatial resolution throughout (no pooling/striding), which matters on a 15×15 grid where every cell counts. The visit-count heatmap on channel 4 gives the network a sense of where it's already been, discouraging loops.

---

Model Parameters

Parameter	Value
Algorithm	DQN (Stable Baselines3)
Policy	CnnPolicy with custom feature extractor
Input shape	(5, 15, 15) — float32
CNN layers	2 (32 filters → 64 filters, 3×3 kernels)
Feature dimension	256
Action space	Discrete(4): Up, Down, Left, Right
Max episode steps	500
Total training timesteps	1,000,000
Checkpoint frequency	Every 50,000 steps
Replay buffer	Standard DQN replay (latest 2 kept on disk)
Observation channels	Empty, Pac-Man, Ghost, Pellet, Visit heatmap (decaying)

Reward Function

Event	Reward
Each step	−0.01 (time penalty)
Pellet eaten	+1.0
Life lost	−2.0
Moving closer to pellet	+0.05
Moving away from pellet	−0.06
Revisiting recent cell	−0.5 (loop avoidance)

---

How to Use

Setup

# Clone the repo
git clone https://github.com/maariogutierrez/pacman.git
cd pacman

# Create a virtual environment
python -m venv .venv
source .venv/bin/activate      # Linux/macOS
# .venv\Scripts\activate       # Windows

# Install dependencies
pip install -r requirements.txt

Watch the trained agent play

python main.py dqn_pacman.zip

dqn_pacman.zip model is included in the releases section of this repo.

Train from scratch

python train.py

This runs 1M timesteps and saves checkpoints to ./checkpoints/. To resume training, uncomment the DQN.load(...) line in train.py and point it at your latest checkpoint.

Evaluate a trained model

The training script automatically evaluates the final model over 10 episodes and prints mean reward ± std. You can also load and evaluate manually:

from stable_baselines3 import DQN
from pacman_env import PacManEnv

env = PacManEnv()
model = DQN.load("checkpoints/dqn_pacman_1000000_steps.zip", env=env)
mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=10)
print(f"Mean reward: {mean_reward:.2f} ± {std_reward:.2f}")

---

Repository Structure

pacman/
├── main.py              # Run the trained agent in the terminal
├── pacman.py            # Game logic: entities, arena, rendering
├── pacman_env.py        # Gymnasium environment wrapper + reward function
├── train.py             # DQN training script with CNN policy
├── requirements.txt     # Python dependencies
├── LICENSE              # MIT
└── README.md            # This file

---

Training Video

Watch the full training progression from random movement to competent gameplay:

The video shows checkpoints at 50K, 150K, 250K, 500K, 600K, 900K, and 1M steps, with score summaries for each.

---

License

This project is licensed under the MIT License — see the LICENSE file for details.