Skip to content

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

Hydronaut

Hydronaut is a 2D submarine survival and treasure-collection game engine featuring an integrated Deep Q-Network (DQN) Reinforcement Learning AI agent built with C++17, SFML 2.6.1, and LibTorch.

Language Graphics Framework Build

Overview

Hydronaut serves a dual purpose as an interactive 2D arcade game and a high-performance simulation environment designed for Deep Reinforcement Learning (DRL).

Traditional Reinforcement Learning environments often interface Python training scripts with visual game engines via Inter-Process Communication (IPC) or custom socket protocols. This architecture introduces serialisation overhead, IPC latency, and frame-rate bottlenecks during experience collection. Hydronaut solves this efficiency problem by embedding PyTorch's native C++ library (LibTorch) directly inside the C++ simulation loop.

The system supports both fast headless execution (bypassing window rendering for rapid gradient updates) and real-time visual inference. Hydronaut is designed for software engineers, machine learning researchers, and developers evaluating embedded AI architectures and clean software design patterns in C++.

Key Features

  • 2D Submarine Engine & Multi-Level Physics: Features screen-relative spatial scaling and distinct mathematical obstacle trajectory types:
    • Level 1 (Obstacles Unleashed): Pure survival against rotating convex polygon swarms.
    • Level 2 (Arc of Chaos): Treasure collection combined with Sine-wave and Parabolic trajectory tracking.
    • Level 3 (Waves of Danger): Advanced treasure hunting against Secant and Exponential-Sine mathematical curve dynamics.
  • Embedded Deep Q-Network (DQN) Agent: Fully native C++ Deep Q-Learning implementation built on LibTorch. Includes target network soft-updates ($\tau = 0.005$), $\epsilon$-greedy exploration with exponential decay, and uniform experience replay (50,000 transition capacity).
  • 55-Dimensional State Space & Expanded Action Mapping: Encodes normalized player positions, velocities, dedicated treasure coordinates, step counters, and up to 8 threat slots into a 55-element float vector. Maps state inputs to 15 discrete action outputs (5 movement directions $\times$ 3 ability execution states: Normal, Hydro-Dash invulnerability, Sonar slowdown).
  • Dual Execution Pipelines:
    • Headless Training Target (train_hydronaut): Executes rapid physics iterations without visual rendering overhead.
    • Inference Visualizer Target (play_hydronaut): Renders real-time visual simulation driven by loaded network weights ($\epsilon = 0.0$).
  • Curriculum Learning & Action Holding: Multi-phase training pipeline transitioning difficulty across levels at configured episode milestones (Episodes 600 and 1200). Implements randomized action holding (4–8 frames) to simulate realistic decision latencies.
  • Decoupled Architecture (GoF Patterns):
    • Memento Pattern: SimulationMemento snapshotting enables instant episodic resets without tearing down or reallocating the physics object graph.
    • Strategy Pattern: IRewardStrategy decouples score and reward calculations from core simulation logic.
    • Observer & Command Patterns: EventBus publish-subscribe routing handles internal event streams, feeding a CommandQueue for telemetry serialization (telemetry.log).
    • State Pattern: Submarine transitions between discrete entity states (NormalState, DashState).
    • Singleton Pattern: AssetManager manages texture, font, and audio resource lifecycles.

Architecture

Hydronaut is architected around strict separation of concerns, isolating rendering, simulation physics, event distribution, and neural network updates into dedicated systems.

The core SimulationEnvironment base class manages physics time-steps, HUD updates, screen-shake dynamics, and sonar calculations. Game events (collisions, reward triggers, input actions) publish to an EventBus, which notifies active observers (RewardObserver, ScoreObserver, TelemetryObserver). For RL training, SimulationEnvironment exposes non-rendering step interfaces that feed states directly to the DQNAgent.

graph TD
    subgraph Execution Entry Points
        Main[main.cpp - Human Play]
        Train[train.cpp - Headless RL Training]
        Play[play.cpp - Inference Visualizer]
    end

    subgraph Simulation Core
        GameApp[GameApp / Menu]
        SimEnv[SimulationEnvironment Base]
        L1[Level 1: Swarm Survival]
        L2[Level 2: Sine/Parabolic]
        L3[Level 3: Sec/ExpSine]
        Submarine[Submarine Entity]
    end

    subgraph Design Patterns & Event Pipeline
        State[EntityState Pattern]
        Strategy[IRewardStrategy Pattern]
        Memento[SimulationMemento]
        EventBus[EventBus Pub/Sub]
        CmdQueue[CommandQueue Telemetry Log]
        AssetMgr[AssetManager Singleton]
    end

    subgraph Reinforcement Learning Engine
        Agent[DQNAgent - LibTorch MLP]
        Buffer[ReplayBuffer - 50k Capacity]
        Config[RLConfig Centralized Hyperparameters]
    end

    Main --> GameApp
    GameApp --> SimEnv
    Train --> SimEnv
    Play --> SimEnv

    SimEnv <|-- L1
    SimEnv <|-- L2
    SimEnv <|-- L3

    SimEnv --> Submarine
    Submarine --> State
    SimEnv --> Strategy
    SimEnv --> Memento
    SimEnv --> EventBus
    EventBus --> CmdQueue
    SimEnv --> AssetMgr

    Train --> Agent
    Play --> Agent
    Agent --> Buffer
    Agent --> Config
    SimEnv <--> Agent
Loading

Component Responsibilities

  • SimulationEnvironment: Base game engine managing object updates, collision detection, screen scaling, state vector extraction, and RL step loops.
  • Submarine & EntityState: Manages player physics, velocity vector calculations, dash state transitions, and ability cooldown timers.
  • IRewardStrategy: Evaluates level-specific reward metrics (survival time, treasure proximity delta, idleness penalties, collision penalties).
  • DQNAgent: Implements the Multi-Layer Perceptron (DQNNetImpl: 55 $\rightarrow$ 128 $\rightarrow$ 128 $\rightarrow$ 15), $\epsilon$-greedy action selection, and gradient updates via PyTorch C++.
  • SimulationMemento: Captures internal state snapshots (BaseSnapshot, entity metrics, obstacle positions) for zero-allocation state restores.
  • EventBus & CommandQueue: Dispatches system events and streams serialized command telemetry to telemetry.log.

Project Structure

hydronaut/
├── assets/                  Media assets (sprites, textures, fonts, audio)
├── include/                 Header definitions and interfaces
│   ├── AssetManager.hpp     Singleton asset loader and cache
│   ├── Command.hpp          Input/Action command representations
│   ├── CommandQueue.hpp     Thread-safe telemetry command queue
│   ├── Constants.hpp        Game engine global constants and scaling factors
│   ├── DQNAgent.hpp         LibTorch Deep Q-Network agent implementation
│   ├── EntityState.hpp      State pattern interface for player entities
│   ├── EventBus.hpp         Publish-subscribe event bus
│   ├── GameObservers.hpp    Observer implementations for score, reward, telemetry
│   ├── IInterfaces.hpp      Abstract engine and renderer interfaces
│   ├── Level1.hpp           Level 1 survival environment definition
│   ├── Level2.hpp           Level 2 sine/parabolic environment definition
│   ├── Level3.hpp           Level 3 secant/exponential environment definition
│   ├── Obstacle.hpp         Base obstacle class and collision abstractions
│   ├── ReplayBuffer.hpp     Experience replay storage buffer
│   ├── RewardStrategy.hpp   Strategy pattern implementations for level rewards
│   ├── RLConfig.hpp         Centralized hyperparameter and model configuration
│   ├── SFMLAdapters.hpp     SFML wrapper implementations
│   ├── SimulationEnvironment.hpp Abstract simulation engine base class
│   ├── SimulationMemento.hpp Memento pattern snapshot definition
│   └── Submarine.hpp        Player entity definition
├── src/                     Engine and RL implementation source files
│   ├── main.cpp             Human playable game entry point
│   ├── train.cpp            Headless RL curriculum training entry point
│   ├── play.cpp             Visual AI playback entry point
│   ├── mock_memento.cpp     Verification test for Memento state restoration
│   └── [Systems].cpp        Physics, level logic, and reward implementations
├── libtorch/                PyTorch C++ CPU/CUDA prebuilt library binaries
├── CMakeLists.txt           CMake build configuration
├── run.sh                   Build and run execution script
└── hydronaut_dqn.pt         Serialized LibTorch neural network weights

Technology Stack

Category Technology Purpose
Language C++17 Core simulation engine, physics, and RL agent implementation
Graphics & Audio SFML 2.6.1 Window creation, 2D sprite rendering, input handling, audio management
AI / Machine Learning LibTorch (PyTorch C++) Neural network architecture, tensor computations, Adam optimizer, model serialization
Build System CMake 3.18+ Multi-target compilation orchestration and dependency management
Design Patterns GoF Architecture State, Strategy, Memento, Observer, Command, and Singleton patterns
Telemetry Binary/Text Event Stream Command logging to telemetry.log via thread-safe queue

How It Works

Primary Simulation & Training Workflow

  1. Environment Initialization (reset): The training loop initializes a virtual resolution (randomized between $640 \times 480$ and $1920 \times 1080$ per episode to ensure resolution-agnostic spatial learning).
  2. State Vector Construction (getState): The environment extracts a 55-dimensional floating-point array:
    • s[0..9]: Level ID, viewport dimensions, player normalized coordinates $(x, y)$, velocity $(vx, vy)$, timeout counter, and last step reward.
    • s[10..14]: Dedicated treasure block (type, relative position $x, y$, relative velocity $vx, vy$).
    • s[15..54]: Threat slots (8 slots $\times$ 5 attributes: type, normalized position $x, y$, velocity $vx, vy$).
  3. Action Selection (choose_action): The DQNAgent evaluates the state vector. During exploration, an action is sampled uniformly; during exploitation, the state tensor passes through the online network (55 → 128 → 128 → 15), selecting the action with the highest Q-value ($\text{argmax}(Q)$).
  4. Physics Step & Action Holding (step): The selected discrete action (0–14) is applied. The simulation steps forward at $dt = 1/60\text{s}$ over an action hold duration (4–8 frames in training).
  5. Reward Calculation (IRewardStrategy): The active strategy computes a scalar reward:
    • Level 1: Survival reward (+1.0/frame), collision penalty (-1.0).
    • Levels 2 & 3: Survival baseline (+0.15/frame), treasure approach distance delta ($\Delta d \times 0.50$), idleness penalty (-0.50 if speed $&lt; 0.5$), collection spike (+200.0), collision penalty (-1.0).
  6. Experience Replay & Batch Learning (learn): Transitions $(s, a, r, s', \text{done})$ are appended to ReplayBuffer. Every 4 frames, a mini-batch of 64 transitions is sampled. The loss is computed using MSE against Bellman targets: $$Q_{\text{target}} = r + \gamma \cdot \max_{a'} Q_{\text{target}}(s', a') \cdot (1 - \text{done})$$ Gradients are clipped at norm 10.0, the online network is updated via Adam ($LR = 10^{-3}$), and target weights update softly ($\tau = 0.005$).
sequenceDiagram
    autonumber
    participant Trainer as train.cpp / Main Loop
    participant Env as SimulationEnvironment
    participant Agent as DQNAgent (LibTorch)
    participant Buffer as ReplayBuffer
    participant Target as Target Network

    Trainer->>Env: reset(windowSize)
    Env-->>Trainer: Initial State Vector (55 floats)

    loop Episode Step Loop (Max 8000 steps)
        Trainer->>Agent: choose_action(state)
        Agent-->>Trainer: Action integer (0-14)
        Trainer->>Env: step(action, hold_frames)
        Env->>Env: Advance physics & Evaluate IRewardStrategy
        Env-->>Trainer: Next State (s'), Reward (r), Done flag
        Trainer->>Buffer: push(s, a, r, s', done)
        
        alt Every LEARN_EVERY (4) steps & Buffer > WARMUP_EXP (64)
            Trainer->>Agent: learn(batch_size=64)
            Buffer-->>Agent: Sample 64 Experiences
            Agent->>Target: Evaluate Bellman Target Q-values
            Agent->>Agent: Compute MSE Loss & Clip Gradients
            Agent->>Agent: Adam Optimizer Step & Soft Target Update (tau=0.005)
        end
    end
Loading

Installation

Prerequisites

  • C++ Compiler: g++ or clang++ supporting C++17 (-std=c++17).
  • Build System: CMake version 3.18 or higher.
  • SFML Libraries: SFML 2.5+ development libraries (libsfml-graphics, libsfml-window, libsfml-system, libsfml-audio).
  • LibTorch: PyTorch C++ API library.

Installing Dependencies on Ubuntu/Debian

sudo apt-get update
sudo apt-get install -y build-essential cmake libsfml-dev

Clone & Build Setup

  1. Clone the Repository:

    git clone https://github.com/sudo-muneeb/hydronaut-game.git
cd hydronaut-game

  2. Verify LibTorch Directory: Ensure libtorch/ is present in the project root directory. If missing or installing custom PyTorch C++ binaries, place LibTorch at ./libtorch.

  3. Build via Quick Script:

    bash run.sh

  4. Manual CMake Compilation:

    mkdir -p build && cd build
cmake -DCMAKE_PREFIX_PATH=../libtorch ..
cmake --build . --target hydronaut train_hydronaut play_hydronaut mock_memento -j$(nproc)

Configuration

All Reinforcement Learning hyperparameters, neural network layout dimensions, and training loop parameters are consolidated in include/RLConfig.hpp.

Configuration Parameters Reference

Parameter Type Default Value Description
STATE_DIM int 55 State vector input dimension
HIDDEN_DIM int 128 Neurons per hidden layer (2 hidden layers)
ACTION_DIM int 15 Output discrete action space dimension
GAMMA float 0.99f Discount factor for future rewards
LR float 1e-3f Learning rate for Adam optimizer
TAU float 0.005f Polyak soft update rate for target network
EPS_START float 1.0f Initial exploration rate for $\epsilon$-greedy policy
EPS_END float 0.08f Minimum exploration rate floor
EPS_DECAY float 0.995f Multiplicative epsilon decay factor per episode
BATCH_SIZE int 64 Mini-batch sample size for experience replay
BUFFER_CAPACITY int 50000 Maximum capacity of transitions in ReplayBuffer
EPISODES int 2000 Total curriculum training episodes
MAX_STEPS int 8000 Maximum steps per episode before termination
HOLD_MIN / HOLD_MAX int 4 / 8 Frame range for holding a selected action during training
PHASE1_END_EP int 600 Curriculum threshold to transition from Level 1 to Level 2
PHASE2_END_EP int 1200 Curriculum threshold to transition from Level 2 to Level 3
MODEL_PATH const char* "hydronaut_dqn.pt" File system output path for serialized weights

To apply configuration changes, modify include/RLConfig.hpp and rebuild the targets using CMake.

Usage

1. Human Playable Game Mode

Launch the full interactive SFML game for manual play:

./run.sh
# OR manually after build:
./build/hydronaut

Controls (Human Mode)

Key Action
↑ ↓ ← → Move Submarine (Up, Down, Left, Right)
Space Activate Hydro-Dash (Invulnerability + Speed Boost)
Shift Activate Sonar Pulse (Slow down all active obstacles)
P Pause / Resume game
Escape Return to main menu / Quit
Enter Select menu items

2. Headless RL Training (train_hydronaut)

Run the high-speed headless training pipeline (no visual window rendering overhead):

./build/train_hydronaut [episodes]

Example:

./build/train_hydronaut 2000

This trains the agent across 2,000 curriculum episodes and exports model weights to hydronaut_dqn.pt.

3. Visual Inference Playback (play_hydronaut)

Run the visual SFML renderer driving the submarine with a trained model ($\epsilon = 0.0$ exploitation):

./build/play_hydronaut [level_number]

Examples:

./build/play_hydronaut 1   # Play Level 1 (Swarm Survival)
./build/play_hydronaut 2   # Play Level 2 (Sine / Parabolic)
./build/play_hydronaut 3   # Play Level 3 (Secant / Exp-Sine Curves)

API Reference

The core simulation components expose a pure C++ Environment API designed for RL integration and state snapshotting.

SimulationEnvironment Class API

Method Signature Return Type Description
reset(sf::Vector2u windowSize) std::vector<float> Resets the simulation episode, clears scores, randomizes obstacle positions, and returns the initial 55-dimensional state vector.
step(int action, float& reward, bool& isDone) std::vector<float> Applies discrete action (0–14), executes one physics time-step ($dt = 1/60\text{s}$), evaluates IRewardStrategy, updates done flags, and returns the new state vector.
getState() std::vector<float> Computes and returns the current 55-dimensional normalized state vector.
create_memento() std::unique_ptr<SimulationMemento> Captures a deep snapshot of the current state (BaseSnapshot, player state, active obstacles) for state restoration.
restore_memento(const SimulationMemento& memento) void Restores the simulation state instantly to the provided memento snapshot without memory reallocation.
setVirtualSize(sf::Vector2u sz) void Overrides the internal virtual physics resolution for multi-resolution training without altering OS window properties.
setTrainingMode(bool val) void Toggles training mode flag, disabling interactive SFML event polling loops.

API Request & Response Example (C++ Code)

#include "Level1.hpp"
#include <SFML/Graphics.hpp>
#include <iostream>

// Initialize SFML headless window handle
sf::RenderWindow window(sf::VideoMode(1920, 1080), "Headless", sf::Style::None);
Level1 env(window);
env.setTrainingMode(true);

// 1. Reset Environment
std::vector<float> state = env.reset({1920, 1080});
std::cout << "Initial state dimension: " << state.size() << std::endl; // Returns 55

// 2. Step Environment (Action 0 = Move Up)
float reward = 0.0f;
bool isDone = false;
std::vector<float> nextState = env.step(0, reward, isDone);

std::cout << "Step reward: " << reward << ", Done: " << (isDone ? "true" : "false") << std::endl;

Testing

The project includes built-in verification targets to ensure architectural invariants and state restoration safety.

Memento Pattern Verification Test (mock_memento)

The mock_memento executable tests state snapshotting fidelity. It steps the environment, takes a memento snapshot, executes divergent actions, restores the state snapshot, and verifies that spatial positions and internal flags match the pre-divergence state byte-for-byte.

Running the Test

# Build test executable
cd build && cmake --build . --target mock_memento -j$(nproc)

# Run verification test
./mock_memento

Expected Test Output

[Mock] Init...
--- Taking 10 initial steps ---
Step 10 Pos: (400, 300)
--- Creating Memento ---
--- Taking 10 divergent steps ---
Step 20 Pos: (400, 420)
--- Restoring Memento ---
Restored Pos: (400, 300)
SUCCESS: exact state match!

Development

Developers modifying or extending Hydronaut should follow established project abstractions:

  1. Adding Custom Obstacles:

    • Inherit from Obstacle in include/Obstacle.hpp.
    • Implement update(float dt), draw(sf::RenderTarget& target), and getBounds().
    • Register the new obstacle type inside level obstacle generation loops.

  2. Custom Reward Strategies:

    • Implement the IRewardStrategy interface (include/RewardStrategy.hpp).
    • Define calculateReward(const EnvironmentContext& ctx).
    • Inject the strategy using setRewardStrategy(...) inside the target level constructor.

  3. Incremental Target Compilation:

    cd build
cmake --build . --target train_hydronaut -j$(nproc)

Security

Hydronaut incorporates defensive software engineering practices:

  • Exception Safety: All level execution loops and texture loadings are wrapped in double-exception catching guards to prevent unhandled runtime aborts.
  • Resource Lifecycle Management (RAII): All game entities, levels, and neural network weights use std::unique_ptr and standard C++ RAII containers to eliminate memory leaks and dangling pointer access.
  • Asset Protection: The AssetManager singleton guards texture and font loading against double-free errors and mid-frame reallocation.
  • Bounds Validation & Clamping: Viewport coordinates and velocity inputs are strictly validated and clamped against screen boundaries to avoid out-of-bounds float anomalies.

Performance

Hydronaut includes several performance optimizations for simulation scaling:

  • Headless Execution Throughput: By bypassing SFML rendering buffers during training, train_hydronaut achieves thousands of simulation steps per second on standard CPU cores.
  • Zero-Allocation Episodic Resets: The Memento design pattern resets episodic state graphs via memory copy rather than heap destruction and object re-instantiation.
  • Direct Memory Tensor Injection: State vectors convert to PyTorch tensors using torch::from_blob to eliminate unnecessary memory copying during batch learning.
  • RPATH Embedded Binaries: CMake build scripts embed LibTorch shared library paths (BUILD_RPATH) directly into compiled binaries, eliminating runtime LD_LIBRARY_PATH lookup overhead.

Limitations

  • Discrete Action Space: The action space is limited to 15 discrete action combinations rather than continuous vector forces.
  • Fixed Threat Encoded Slots: The state vector accommodates up to 8 active threat slots (MAX_OBJ_SLOTS = 8). Objects beyond slot capacity are omitted from neural observation.
  • LibTorch C++ ABI Coupling: Compiling requires ABI matching between the installed host compiler and the LibTorch binary distribution (cxx11 ABI vs pre-cxx11 ABI).
  • CPU Execution Default: Default training scripts target host CPU tensor operations. GPU CUDA acceleration requires a CUDA-enabled LibTorch build and explicit tensor device placement.

Future Improvements

  • Continuous Control Support: Implementing Proximal Policy Optimization (PPO) or Deep Deterministic Policy Gradient (DDPG) for smooth directional thrusting.
  • Attention-Based State Encoding: Utilizing Transformer/Attention layers to encode variable-length obstacle lists without fixed slot constraints.
  • CUDA Acceleration Integration: Adding explicit GPU device dispatch in DQNAgent for high-batch training parallelization.
  • OpenTelemetry / TensorBoard Export: Extending CommandQueue to stream metrics directly to standard visualization frameworks.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages