PixelVisionAI

PixelVisionAI is a game-agnostic, vision-based embodied AI framework that learns to play arbitrary PC games by observing screen pixels and acting through OS-level mouse/keyboard inputs only. The core engine contains no game-specific logic; all game-specific behavior lives in pluggable adapters.

Hard rules:

• No game memory reading
• No mods
• No internal APIs
• Only screen pixels + OS-level input
• Linux (X11, NixOS-friendly)
• NVIDIA CUDA support (PyTorch)

Architecture Overview

Core Engine (game-agnostic)

Located in src/pixelvisionai/core/ and includes:

• Vision pipeline (frame capture, preprocessing, stacking)
• Action execution (mouse/keyboard primitives)
• Agent loop (observe → decide → act)
• Planner interface (rule-based baseline)
• RL interface (Gym-compatible environment)
• Overlay system (terminal UI)
• Logging & checkpointing
• Safety systems (emergency stop, deadman switch)

The core engine intentionally does not include:

• game names
• game-specific UI templates
• game-specific rewards
• game-specific actions

Game Adapter Interface

Adapters implement the GameAdapter interface to connect any game to the engine:

• detect_game_state(observation)
• get_available_actions(state)
• compute_reward(prev_state, current_state)
• is_terminal(state)
• get_overlay_text(state)
• reset_procedure()
• calibration_procedure()

Game Plugins

Game plugins live under src/pixelvisionai/games/ and encapsulate:

• UI template matching
• OCR
• reward shaping
• macro action libraries
• calibration helpers
• victory condition definitions

Example plugin: Captain of Industry

This repository ships with Captain of Industry as an example plugin located at: src/pixelvisionai/games/captain_of_industry/.

Minimal example plugin: generic_sandbox

A very minimal adapter used to prove the core is game-agnostic.

Repository Structure

PixelVisionAI/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│   └── pixelvisionai/
│       ├── core/
│       │   ├── engine.py
│       │   ├── vision.py
│       │   ├── input.py
│       │   ├── overlay.py
│       │   ├── safety.py
│       │   ├── logging.py
│       │   └── config.py
│       ├── rl/
│       │   ├── env.py
│       │   └── train.py
│       ├── planner/
│       │   └── rule_planner.py
│       └── games/
│           ├── captain_of_industry/
│           └── generic_sandbox/
├── scripts/
│   ├── calibrate_region.py
│   ├── overlay_test.py
│   └── emergency_stop_test.py
├── configs/
│   ├── core.yaml
│   ├── captain_of_industry.yaml
│   └── generic.yaml
└── run.py

Running

Install dependencies:

pip install -r requirements.txt

Run the example adapters (adapters are dynamically loaded from pixelvisionai.games.<game_name>.adapter):

python run.py --game generic_sandbox --config configs/generic.yaml
python run.py --game captain_of_industry --config configs/captain_of_industry.yaml

Adding a New Game Adapter

1. Create a new plugin directory: src/pixelvisionai/games/<your_game>/
2. Implement a GameAdapter class with the required interface.
3. Add any game-specific modules (vision, rewards, actions, configs) inside the plugin.
4. Expose the adapter as GameAdapterImpl in pixelvisionai.games.<your_game>.adapter.
5. Add a config file under configs/ if needed.

The core engine remains unchanged while game-specific logic stays in the plugin.