Cartpole using GA

This project implements a genetic algorithm (GA) to solve the Cartpole problem. The Cartpole problem is a classic control task where the objective is to balance a pole on a moving cart. The genetic algorithm is used to evolve a neural network that controls the movement of the cart to keep the pole upright as long as possible.

Project Overview

The Cartpole problem is a classic control task in reinforcement learning, where the goal is to balance a pole on a moving cart. The environment is described as follows:

• The state space consists of four dimensions representing the cart's position, velocity, pole angle, and pole angular velocity.
• The action space consists of two discrete actions: move the cart left or right.
• The agent receives a reward of +1 for each time step the pole remains upright.
• The episode ends if the pole angle is greater than ±12 degrees or the cart moves outside the screen's bounds.

The goal is to design an agent (controller) that learns to balance the pole on the cart for as long as possible.

In this project, a genetic algorithm is used to optimize a neural network that interacts with the Cartpole environment from Farama Foundation's Gymnasium. The goal is to evolve the weights and biases of the neural network using selection, crossover, and mutation strategies to improve its ability to balance the pole. The program runs over multiple generations, and early stopping is applied once a predefined fitness target is reached.

Key Features

• Genetic Algorithm:

  • Population-based search with selection, crossover, and mutation operations.
  • Fitness evaluation based on performance in the Cartpole environment.

• Neural Network:

  • A feedforward neural network with 4 input, 16 hidden and 1 output neurons trained to control the Cartpole system.

Installation

Prerequisites

Ensure you have the following installed:

• Python 3.7+
• NumPy
• Gymnasium

Setup

1. Clone the repository to your local machine:

git clone https://github.com/yourusername/Cartpole-using-GA.git
cd Cartpole-using-GA

2. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`

3. Install the required dependencies:

pip install python
pip install numpy
pip install gymnasium

Usage

Training the Genetic Algorithm

To train the genetic algorithm, run the following command:

main.py: This will initiate the training process. The algorithm will run for several generations, evolving the neural network's weights. It will stop early once a predefined fitness target (e.g., 500) is reached.

Testing the Best Network

After training, you can test the best evolved neural network by running:

test_best_solution.py: This will evaluate the performance of the best network over 5 episodes and output the average reward score.

Results

The training process reports the following metrics for each generation:

-Generation: The current generation number. -Average Fitness: The average fitness of all individuals in the current generation. -Best Fitness: The best fitness score achieved by any individual in the current generation. Early stopping is triggered once the fitness score reaches the target value (e.g., 500).

Example output:

python main.py
Generation 1 | Average Fitness: 13.41 | Best Fitness: 34.20
Generation 2 | Average Fitness: 20.70 | Best Fitness: 69.20
Generation 3 | Average Fitness: 57.58 | Best Fitness: 398.00
Generation 4 | Average Fitness: 89.61 | Best Fitness: 416.80
Generation 5 | Average Fitness: 105.59 | Best Fitness: 416.80
Generation 6 | Average Fitness: 194.62 | Best Fitness: 460.80
Generation 7 | Average Fitness: 276.86 | Best Fitness: 460.80
Generation 8 | Average Fitness: 299.86 | Best Fitness: 500.00
Target fitness of 500 reached in generation 8. Stopping early.

python test_best_solution.py
Average reward over 5 episodes: 487.60