Counter-Strike Reinforcement Learning Agent

An autonomous Counter-Strike agent trained using Reinforcement Learning (RL), leveraging offline pre-training on human demonstrations followed by live in-game fine-tuning.

Agent gameplay in action. The left panel shows live in-game footage, while the right panel visualizes the agent's real-time perception and policy decisions.

Methodology

Overview

The end-to-end architecture of the RL system, from multi-channel state observation and real-time input handling to hybrid offline/online RL updates:

At a high level, the system operates across three main interconnected modules:

• Environment Loop: The State Observer gathers game state s via Game State Integration (GSI), developer console telemetry, and a YOLO object detector. Stochastically sampled actions a ~ pi_phi(a|s) are sent to the Action Handler to execute simulated keyboard and mouse inputs.
• Data Pipeline: Environment transitions (s, a, r, s', d) stream into the online replay buffer D_on. During training, transitions are dynamically mixed with human demonstrations D_off and filtered through stratified sampling to tackle reward sparsity.
• Agent Optimization: The agent fits V_psi and Q_theta, computes the advantage A = Q_theta(s, a) - V_psi(s), and uses Advantage-Weighted Regression (AWR) to update the policy pi_phi.

RL Formulation

• State Space: Combines player context (health, ammo, velocity, position, aim_angle), primary enemy bounding box (x, y, w, h) detected via YOLO, and a 2D map occupancy grid.
• Action Space: Combines discrete actions (shooting, reloading, directional movement) and continuous horizontal mouse rotation (yaw_delta).
• Reward Function: Encourages accurate target engagement while penalizing off-target firing, empty-magazine shots, and health loss.

Training Strategy

Training strictly from scratch via online interaction is challenging due to reward sparsity and single-instance environment bottlenecks. To address this, we use Implicit Q-Learning (IQL):

1. Offline Pre-training: Pre-trains the agent on human demonstration data (D_off) to bootstrap a baseline policy. Stratified sampling is applied to ensure positive and negative reward transitions are adequately sampled.
2. Online Fine-tuning: Fine-tunes the policy through direct in-game interaction. An online replay buffer (D_on) collects recent experiences, while dynamic mixing balances D_off and D_on to maintain policy stability.

Evaluation

Training Performance

• Offline Training: Trained for 4,000 policy evaluation epochs (Q & V networks) and 1,000 policy extraction epochs (~1–2 hours total).

• Online Fine-Tuning: Fine-tuned over 70 epochs (~30 mins/epoch, 1–2 days total) with 10 episodes/epoch. Online experience ratio linearly scales over a 20-epoch warm-up to 70%. Average episode returns improved significantly from -2500 to -800 at peak.

Policy Analysis

• Firing Behavior: The offline baseline frequently misfired off-target or with empty magazines due to sparse negative samples in human demonstrations. Online fine-tuning successfully taught the agent to suppress firing unless aligned with an enemy bounding box and holding ammo.

• Spatial Navigation: Trajectory vector fields on fy_snow show coherent movement pathways, especially when centered around combat zones, with the online agent demonstrating higher visitation density in key engagement areas.

• Obstacle Avoidance: Grad-CAM activations on the 2D occupancy grid shows that the network detects obstacle boundaries and suppresses forward movement when facing obstacles directly ahead.

Setup & Usage

CS2 Game Setup

First, add -condebug to Counter-Strike 2 launch options in Steam to enable console logging, and bind the required control keys in the developer console:

bind "p" "getpos"
bind "0" "kill"
bind "-" "mp_restartgame 1"
bind "=" "buy ak47"

Next, set up GSI configuration in your CSGO config folder (e.g., D:\SteamLibrary\steamapps\common\Counter-Strike Global Offensive\game\csgo\cfg) in gamestate_integration_stats.cfg:

"GSI Configuration"
{
    "uri" "http://localhost:3000"
    "timeout" "5.0"
    "buffer"  "0.05"
    "throttle" "0.05"
    "heartbeat" "0.05"
    "data"
    {
        "provider"            "1"
        "map"                 "1"
        "round"               "1"
        "player_id"           "1"
        "player_state"        "1"
        "player_weapons"      "1"
    }
}

Finally, launch a Deathmatch game on your target map (e.g., fy_snow) and execute the match configuration commands in the console:

sv_cheats true; host_timescale 0.5; mp_roundtime 60; bot_quota 10; mp_buytime 60000; bot_difficulty 0;

Configuration Setup

Before running data collection or training scripts, update the following parameters in config/config.py:

• console_log_path: Set the absolute path to your CS2 console.log file:

  console_log_path = r"C:\Program Files (x86)\Steam\steamapps\common\Counter-Strike Global Offensive\game\csgo\console.log"

• screen_w & screen_h: Match your in-game screen resolution (e.g., 1920 x 1080).
• experiences: List the .parquet file names in ./artifacts/experiences/ to load for offline training.
• load_qv_dir / load_dir: Specify checkpoint directory paths when resuming policy extraction or loading pre-trained weights.

Collect Demonstration Data

Record human gameplay experiences (D_off) to bootstrap the offline policy:

python collect_experience.py

Outputs parquet datasets into ./artifacts/experiences/.

Offline Pre-Training

Pre-train value networks (V, Q) and extract the baseline policy (pi) from collected demonstration data:

python offline_train.py

Saves checkpoints and loss curves to ./artifacts/<timestamp>/.

Online Fine-Tuning

Fine-tune the policy through live interaction with the CS2 game environment:

python online_train.py

Simulate & Evaluate Agent

Run live inference with the trained agent policy in the CS2 environment:

python simulate.py --load_path ./artifacts/<checkpoint_folder>

Add --explore to enable stochastic exploration during evaluation.

Pre-trained Models & Datasets

Pre-trained offline and online model weights are located in the artifacts/ directory. If you would like to train an offline model without recording gameplay demonstrations yourself, you can download my demonstration dataset.

Conclusion

The agent actually doesn't perform all that well, but I believe with a bit more hyperparameter tuning (fine-tuning reward weights, expanding network capacity, and training for more epochs), its performance could improve significantly.

From start to finish, this project took about 3–4 months to complete, with a 1–2 month break in between due to other commitments (~1 month spent brainstorming and building the offline/online training pipelines, and another month running a crazy number of ablations and experiments). I'm glad that it's finally over, and I've definitely learned a lot from it.

I hope this is educational and interesting to you. If you'd like to give it a shot (and beat my agent), please go ahead! The source code and model artifacts are all here ;).

Acknowledgment & References

• Implicit Q-Learning (IQL): Offline Reinforcement Learning via Implicit Q-Learning (Kostrikov et al., 2021).
• Yolov5ForCSGO: YOLOv5 detection model for Counter-Strike.

License

This project is licensed under the MIT License.