AI Agents for Every Virtual Environment

🚀 Overview

Welcome to MisterCar, your go-to solution for building versatile AI agents capable of operating in any virtual environment. Build AI assistants that can perform any task a human can do on a computer - whether you're designing bots for video games or automating routine tasks, MisterCar gives you the tools you need.

🎯 Key Use Cases

MisterCar is designed to help you in two primary domains:

🎮 Video Games

• Game Bots: Create bots to play games, search for bugs and generate reports
• Computer Vision Research: Gather and analyze artificial data from games
• Autonomous Driving Research: Develop and test autonomous driving algorithms in simulated environments

💻 General Computer Use

• GUI Automation: Interact with software applications using computer vision, OCR, mouse, and keyboard
• Process Automation: Build workflows that operate across multiple applications
• Code Intelligence: Create agents that can write, test, and execute code
• Voice Integration: Enable speech recognition and synthesis capabilities

🤖 Sense-Plan-Act Architecture

MisterCar is built around the fundamental robotics paradigm of Sense-Plan-Act (SPA), providing a structured approach to developing AI agents:

graph LR
    A[Sense] -->|Perceive Environment| B[Plan] -->|Make Decisions| C[Act]
    C -->|Change Environment| A
    style A fill:#93c47d
    style B fill:#6fa8dc
    style C fill:#e06666

Loading

To implement this paradigm, MisterCar provides four types of composable components:

graph LR
    A[Sensors] --> B[Transforms] --> C[Planners] --> D[Executors]
    style A rx:10,ry:10
    style B rx:10,ry:10
    style C rx:10,ry:10
    style D rx:10,ry:10

Loading

• Sensors capture the virtual environment through various input sources:

  • Screen capture
  • System audio and microphone input
  • Standard input devices (keyboard, mouse, gamepad)
  • External hardware (steering wheel, pedals, custom controllers)
  • System data (memory regions, pipes, process information)

• Transforms process raw inputs into meaningful representations:

  • Image processing and feature extraction
  • Audio signal processing
  • Text and numerical transformations
  • Multi-modal data fusion

• Planners determine actions using AI models and algorithms:

  • Neural networks for perception and decision making
  • Large Language Models for complex reasoning
  • Classical planning algorithms

• Executors implement decisions in the virtual environment:

  • Keyboard, mouse, and gamepad control
  • Dynamic code execution
  • System interaction (memory writing, pipe communication)

These components can be flexibly combined and reused across different stages of your AI agent's lifecycle - from initial data collection through training to deployment.

🧠 Deep Learning Workflow

MisterCar provides comprehensive support for deep learning (PyTorch/Lightning and TensorFlow backends):

1️⃣ Data Collection

• Record multimodal data from various sources:
  • Screen capture and computer vision
  • System audio and microphone input
  • Keyboard, mouse, and gamepad interactions
  • Steering wheel and external devices

2️⃣ Model Development

• Build and train custom models efficiently:
  • Use template datasets and data modules for common scenarios
  • Compose models from simple building blocks, supporting multi-modal, multi-input, multi-output architectures
  • Access pre-built neural network architectures (PyTorch modules)
  • Utilize task-specific training implementations (Lightning modules) for classification, regression, and others
  • Access and fine-tune models from HuggingFace, PyTorch Hub, and TensorFlow Hub

3️⃣ Training

• Comprehensive training utilities:
  • Monitor and visualize training progress
  • Optimize model performance
  • Handle imbalanced data
  • Extensive reinforcement learning support:
    • Multiple RL algorithms (DQN, DDPG, A2C, etc.)
    • Customizable environments, policies, and reward functions

4️⃣ Inference

• Run trained models efficiently:
  • Use dedicated prediction and evaluation runners
  • Support both batch and real-time inference
  • Integrate models within agent systems

📦 Installation

Setting up MisterCar requires careful environment configuration using Conda and Pip. This ensures compatibility, especially with deep learning libraries.

Prerequisites:

• Miniconda or Anaconda: Install from https://www.anaconda.com/download.
• Git: For cloning the repository.
• (Optional but recommended) NVIDIA GPU: For GPU acceleration. Ensure compatible NVIDIA drivers are installed system-wide.

Choosing a deep learning backend:

MisterCar is primarily developed and tested with PyTorch (via Lightning), which is the recommended backend for the best experience and support.

However, support for TensorFlow is available as an optional, advanced alternative. If you choose TensorFlow, please be aware that:

• You will need to carefully follow official TensorFlow installation guides.
• GPU setup for TensorFlow can be complex and requires specific driver, CUDA, and cuDNN versions that you must manage.
• Some examples or features might be less tested with TensorFlow compared to PyTorch.

Installation steps:

1. Clone the repository:

   git clone https://github.com/PypayaTech/MisterCar
   cd MisterCar

2. Create conda environment: Creates the base environment using environment.yml.

   conda env create -f environment.yml

   • This creates mister-car-env with Python 3.11 by default. Adjust environment.yml beforehand if needed.

3. Activate environment:

   conda activate mister-car-env

   • (Remember to activate the environment in every new terminal session.)

4. Install deep learning backend (Choose ONE):

   • (Recommended) PyTorch installation:

     • Install PyTorch, torchvision, and torchaudio using Pip by following the official instructions at https://pytorch.org/get-started/locally/.
     • Select your OS, Package=Pip, Compute Platform (CUDA version or CPU).
     • Run the generated pip install command in your activated environment.
     • Example (Linux/Win, CUDA 11.8):

       # Ensure 'mister-car-env' is active!
       pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

     • Example (CPU):

       # Ensure 'mister-car-env' is active!
       pip install torch torchvision torchaudio

   • (Optional) TensorFlow installation:

     • CPU: For a CPU-only installation, usually the following command works:

       # Ensure 'mister-car-env' is active!
       pip install tensorflow

     • GPU: TensorFlow GPU installation requires specific prerequisites (NVIDIA drivers, CUDA Toolkit, cuDNN). Follow the official TensorFlow documentation meticulously.
       • 1. Official TensorFlow pip install guide: https://www.tensorflow.org/install/pip (Start here)
       • 2. Additional context (Optional): For a walkthrough discussing potential complexities and steps involved in setting up TensorFlow with GPU support based on past experiences, you might find this article helpful.
     • Note: You are responsible for ensuring your system meets the current TF GPU requirements and correctly installing the necessary NVIDIA libraries before installing the tensorflow pip package. Do not install PyTorch if you choose TensorFlow.

5. Install remaining Pip packages: After successfully installing either PyTorch or TensorFlow in the previous step, install all other dependencies from requirements.txt.

   # Ensure 'mister-car-env' is active AND you have installed EITHER PyTorch OR TensorFlow
   pip install -r requirements.txt

   • Note: This command installs supporting libraries, including PyTorch Lightning (lightning) and others listed in the file. If you installed TensorFlow, the PyTorch-specific libraries will still be installed but likely won't be functional. This is done to keep the file structure simple. Only the backend you installed correctly in Step 4 will be usable.

6. (Optional) Verify installation:

   • If you installed PyTorch:

     python -c "import torch; print(f'PyTorch version: {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}'); print(f'Device count: {torch.cuda.device_count()}')"

   • If you installed TensorFlow:

     python -c "import tensorflow as tf; print(f'TensorFlow version: {tf.__version__}'); print(f'Num GPUs Available: {len(tf.config.list_physical_devices('GPU'))}')"

   • Run tests if available (pytest tests/).

You should now have a functional environment configured with your chosen deep learning backend. Remember to always activate the mister-car-env environment before working on the project.