SimForge

Photo-to-MuJoCo simulation pipeline orchestrated by a Claude multi-agent system.

Upload a photo, 5 AI agents analyze it and generate a physics simulation, and the result is displayed in a 3-panel UI with an interactive MJCF viewer.

How It Works

1. Upload a photo of a tabletop or workspace scene
2. 5 AI agents process the image in sequence — analyzing objects, selecting assets, generating physics XML, validating stability, and defining manipulation tasks
3. Results are displayed in a three-panel UI: uploaded image, agent pipeline status, and simulation output with scene analysis, MJCF XML, and validation screenshots

Prerequisites

• Python 3.12+
• Node.js 18+
• pnpm (npm install -g pnpm)
• MuJoCo (>= 3.5.0, installed via pip)
• MuJoCo Menagerie robot models (see Asset Preparation)

Setup

Backend

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Clone robot models (see Asset Preparation below)
git clone --depth 1 https://github.com/google-deepmind/mujoco_menagerie.git vendor/menagerie

# Verify MuJoCo installation
python3 scripts/verify_mujoco.py
python3 scripts/verify_menagerie.py

# Copy and configure environment variables
cp .env.example .env
# Edit .env and add your ANTHROPIC_API_KEY

# Start the dev server
uvicorn backend.main:app --reload --port 8000

Frontend

cd frontend
pnpm install
pnpm dev        # Starts on http://localhost:3000

Environment Variables

Copy .env.example to .env and configure:

Variable	Required	Description
ANTHROPIC_API_KEY	Yes	Claude API key for the agent pipeline
CORS_ORIGINS	No	Allowed frontend origin (defaults to http://localhost:3000)
ENV	No	Environment name (development, production)

The frontend reads NEXT_PUBLIC_API_URL from frontend/.env.local (defaults to http://localhost:8000/api).

Asset Preparation

MuJoCo Menagerie provides pre-built robot models (Franka Panda, UR5e, Sawyer, etc.). Clone it into the gitignored vendor/ directory:

git clone --depth 1 https://github.com/google-deepmind/mujoco_menagerie.git vendor/menagerie

This is ~200 MB. Robot models load via MjModel.from_xml_path() using each model's scene.xml.

Pipeline Agents

#	Agent	Role
1	Scene Architect	Analyzes the uploaded photo for scene layout, objects, materials, and spatial relations
2	Asset Curator	Selects meshes and textures from the asset library, resolves paths, sets physics properties
3	Sim Engineer	Generates MJCF XML for the MuJoCo simulation from the scene spec and asset manifest
4	Physics Validator	Validates simulation stability, checks collisions, and proposes fixes
5	Task Designer	Defines manipulation tasks, goal positions, reward sites, and success criteria

Project Structure

.
├── backend/
│   ├── main.py                 # FastAPI app with CORS and pipeline endpoint
│   ├── agents/                 # Individual agent implementations
│   │   ├── scene_architect.py
│   │   ├── asset_curator.py
│   │   ├── sim_engineer.py
│   │   ├── physics_validator.py
│   │   └── task_designer.py
│   ├── pipeline/
│   │   └── orchestrator.py     # Coordinates the 5-agent pipeline
│   ├── schemas/                # Pydantic models for data contracts
│   └── services/               # Anthropic client, image processing
├── frontend/
│   ├── app/
│   │   ├── layout.tsx          # Root layout with Toaster provider
│   │   ├── page.tsx            # Three-panel resizable layout
│   │   └── error.tsx           # Error boundary for crash recovery
│   ├── components/
│   │   ├── upload-panel.tsx    # Image upload with drag-and-drop
│   │   ├── pipeline-panel.tsx  # Agent status cards with token usage
│   │   ├── output-panel.tsx    # Tabs: Simulation, Scene Analysis, MJCF
│   │   ├── agent-card.tsx      # Individual agent status display
│   │   └── ui/                 # shadcn/ui primitives
│   └── lib/
│       ├── api.ts              # Backend API client
│       ├── use-pipeline.ts     # Pipeline state management hook
│       └── types.ts            # TypeScript type definitions
├── fixtures/                   # Test input data and example responses
├── prompts/                    # Agent system prompts
├── scripts/                    # Verification scripts
├── vendor/                     # MuJoCo Menagerie (gitignored)
├── requirements.txt
└── CLAUDE.md

Development Commands

# Backend
source .venv/bin/activate
uvicorn backend.main:app --reload --port 8000

# Frontend
cd frontend
pnpm dev          # Dev server with Turbo on :3000
pnpm build        # Production build
pnpm lint         # ESLint