🧠 XERO

AI-Powered Multimodal UI Design Generator
Text + Screenshots + Hand-Drawn Sketches + Figma Links → Production-Ready Code

Quick Start • Architecture • Libraries • Training • Figma • Deployment • Evaluation

XERO is an open-source, multimodal vision-language model (VLM) fine-tuned specifically to convert UI design inputs—natural language descriptions, pixel screenshots, hand-drawn wireframe sketches, and live Figma links—into clean, production-ready frontend code (React JSX, Tailwind CSS, StyleX, Emotion).

Unlike standard code LLMs that invent non-existent component names or style classes, XERO operates with Library-Aware RAG (Retrieval-Augmented Generation) over real component design systems (including shadcn/ui, Astryx by Meta, and Material UI v6). It retrieves exact component import statements, prop schemas, and code templates at inference time.

Base Model: HuggingFaceTB/SmolVLM2-2.2B-Instruct (or Qwen2-VL-7B-Instruct)
Fine-tuning: QLoRA 4-bit (NF4) · Training Hardware: Free Google Colab T4 / HF AutoTrain / Colab Pro
Supported Inputs: Text + Screenshots + Wireframes + Figma REST API
Supported Libraries: shadcn/ui, Astryx (Meta), Material UI v6 (73 indexed components)

✨ Key Features

Feature	Description
🖼️ Multimodal Inputs	Combines text descriptions, high-res UI screenshots, wireframes, and Figma links simultaneously
📚 Library-Aware RAG	Dynamically retrieves real component APIs (props, imports, JSX patterns) from ChromaDB vector index
🎨 Figma REST Parser	Extracts colors (HEX/RGBA), typography, spacing scales, border radii, component trees, and frame images
⚡ Zero-RAM Streaming	Training scripts use Hugging Face dataset streaming (`streaming=True`) to run fine-tuning within 200MB RAM
🛡️ Session Survival Auto-Save	Training automatically pushes LoRA checkpoints to HF Hub / Google Drive every N steps to survive disconnects
🚀 HF Spaces & ZeroGPU Deploy	Ships as a Gradio 5 web app optimized for Hugging Face Spaces ZeroGPU inference
🧪 Automated Eval Suite	Evaluates generated code for compilation, render success, import correctness, and placeholder-free execution

🏗️ System Architecture

graph TB
    subgraph INPUTS["🎯 Multimodal User Inputs"]
        TEXT["📝 Natural Language Prompt"]
        IMAGE["🖼️ Screenshot / Wireframe Sketch"]
        FIGMA["🔗 Figma File URL"]
    end

    subgraph PROCESSING["⚙️ Context & Processing Engine"]
        TE["Text Tokenizer"]
        VE["Vision Encoder<br/>(SigLIP — Frozen)"]
        FE["Figma Extractor<br/>(REST API → Design Tokens)"]
        FUSE["Multimodal Fusion Layer"]
    end

    subgraph RAG["📚 RAG Vector Database (ChromaDB)"]
        VS[("ChromaDB Vector Store<br/>73 Component Embeddings")]
        SHADCN["shadcn/ui<br/>24 components"]
        ASTRYX["Astryx (Meta)<br/>24 components"]
        MUI["Material UI v6<br/>25 components"]
    end

    subgraph MODEL["🧠 XERO VLM Backbone"]
        BASE["SmolVLM2-2.2B / Qwen2-VL-7B<br/>+ QLoRA 4-bit Adapters<br/>(adapter_model.safetensors)"]
    end

    subgraph OUTPUT["📦 Production Deliverables"]
        CODE["React JSX / HTML / CSS Code"]
        PREVIEW["Gradio Live Web Preview"]
    end

    TEXT --> TE
    IMAGE --> VE
    FIGMA --> FE
    TE --> FUSE
    VE --> FUSE
    FE --> FUSE
    FUSE --> BASE
    FUSE -.->|"Semantic Search Query"| VS
    VS -.->|"Top-K Component Docs"| BASE
    SHADCN --> VS
    ASTRYX --> VS
    MUI --> VS
    BASE --> CODE
    BASE --> PREVIEW

End-to-End Execution Sequence

sequenceDiagram
    actor User
    participant App as Gradio Web UI
    participant RAG as RAG Engine (ChromaDB)
    participant Figma as Figma REST API
    participant VLM as XERO Model (SmolVLM2 + LoRA)

    User->>App: Submits Prompt + Image + Figma Link + Selected Library
    
    par Parallel Extraction
        App->>RAG: Query component vectors for query string
        RAG-->>App: Return top-5 component schemas & imports
    and
        App->>Figma: Query Figma REST API with file key
        Figma-->>App: Return colors, typography, spacing & layout tree
    end
    
    App->>VLM: Feed fused multimodal prompt (Images + Tokens + RAG Context)
    VLM-->>App: Generate production code (streaming output)
    App-->>User: Display formatted JSX/HTML code + Live Preview

📚 Supported Component Libraries

XERO comes pre-configured with 73 component manifests across three industry-standard component libraries:

Library	Version	Components	Framework	Styling Engine	Manifest Location
shadcn/ui	v2.x	24	React	Tailwind CSS	`libraries/shadcn.json`
Astryx (Meta)	v1.x	24	React	StyleX	`libraries/astryx.json`
Material UI	v6.x	25	React	Emotion / Pigment CSS	`libraries/material-ui.json`

Manifest Structure & Adding New Libraries

To add support for a new library (e.g. Ant Design, Chakra UI, Radix), create a JSON manifest in libraries/your-library.json:

{
  "name": "your-library",
  "version": "1.0.0",
  "framework": "react",
  "styling": "tailwind",
  "components": [
    {
      "name": "Button",
      "import_statement": "import { Button } from 'your-library'",
      "description": "Interactive button supporting primary, secondary, and loading states.",
      "props": {
        "variant": ["primary", "secondary", "outline"],
        "size": ["sm", "md", "lg"],
        "loading": ["true", "false"]
      },
      "code_template": "<Button variant=\"primary\" size=\"md\">Click me</Button>",
      "usage_example": "<Button variant=\"secondary\" loading={isLoading}>Save Changes</Button>"
    }
  ]
}

Then index it into the vector database:

python -m xero.rag.index --library your-library

📁 Project Structure

d:\Projects\Xero\
├── src/xero/                       # Core Python Source Package
│   ├── config.py                   # Central settings (Model IDs, RAG params, hyperparameters)
│   │
│   ├── rag/                        # 📚 RAG Component Library Engine
│   │   ├── engine.py               #   ChromaDB vector database search engine
│   │   └── index.py                #   CLI tool to index and query library manifests
│   │
│   ├── data/                       # 📊 Data Preparation Pipeline
│   │   └── prepare.py              #   Download, normalize, and upload datasets to HF Hub
│   │
│   ├── figma/                      # 🎨 Figma REST API Integration
│   │   └── extractor.py            #   Parser for design tokens, typography, colors, layout
│   │
│   ├── training/                   # 🏋️ Model Training & Fine-Tuning
│   │   └── trainer.py              #   QLoRA 4-bit trainer with session survival callbacks
│   │
│   └── app/                        # 🚀 Deployment Web App
│       └── main.py                 #   Gradio 5 interface for HF Spaces / local run
│
├── libraries/                      # 📦 Component Library JSON Manifests
│   ├── shadcn.json                 #   24 shadcn/ui components
│   ├── astryx.json                 #   24 Astryx (Meta) components
│   └── material-ui.json            #   25 Material UI v6 components
│
├── notebooks/                      # 📓 Notebooks & Cloud Scripts
│   ├── colab_train.py              #   Streaming QLoRA training script
│   └── XERO_Training_Colab.ipynb   #   Interactive Colab training notebook
│
├── tests/                          # ✅ Verification & Quality Suite
│   └── evaluate.py                 #   Code quality metrics & automated tester
│
├── scratch/                        # 🛠️ Development scratch scripts
│   └── generate_notebook.py        #   Generator for validated Colab notebooks
│
├── pyproject.toml                  # Python package definition & dependency groups
├── .gitignore                      # Git exclusion rules
└── README.md                       # Comprehensive documentation

🚀 Quick Start

1. Installation

# Clone the repository
git clone https://github.com/your-username/xero.git
cd xero

# Install in editable mode with development & training extras
pip install -e ".[dev,training]"

2. Index Component Libraries

Build the ChromaDB vector database from indexed library manifests:

# Index all component manifests (shadcn, Astryx, Material UI)
python -m xero.rag.index

# Query the vector index to test retrieval
python -m xero.rag.index --query "pricing table with annual toggle" --library shadcn

3. Launch Local Web Interface

python -m xero.app.main
# Open http://localhost:7860 in your browser

🏋️ Training Pipeline

3-Stage Progressive Training

XERO is trained in 3 progressive stages to avoid catastrophic forgetting and align vision tokens with code generation:

graph LR
    subgraph STAGE1["Stage 1 · Vision Alignment"]
        S1["Train Vision Projector<br/>Freeze LLM Backbone<br/>Dataset: WebSight (50k)"]
    end
    
    subgraph STAGE2["Stage 2 · Code SFT"]
        S2["QLoRA on LLM Backbone<br/>Target: q_proj, v_proj, gate_proj<br/>Dataset: VISION2UI (20k)"]
    end
    
    subgraph STAGE3["Stage 3 · Library-Aware SFT"]
        S3["Inject RAG Component Context<br/>Target: shadcn / Astryx / MUI<br/>Dataset: Curated UI Samples"]
    end
    
    STAGE1 --> STAGE2 --> STAGE3

VRAM Budget & Hardware Math

SmolVLM2-2.2B-Instruct (4-bit QLoRA) — Fits on 16GB T4 GPU

SmolVLM2-2.2B in 4-bit NF4 Quantization:
├── Base Model Weights (4-bit)         ~4.1 GB
├── LoRA Adapter Weights (r=16)        ~0.2 GB
├── 8-bit Paged AdamW Optimizer        ~0.4 GB
├── Gradient Checkpointing             ~2.8 GB
├── Activation Overhead (batch=1)      ~1.5 GB
└── CUDA Overhead & Allocator          ~2.0 GB
                               TOTAL: ~11.0 GB  ✅ (Fits inside 16GB T4 VRAM)

Qwen2-VL-7B-Instruct (4-bit QLoRA) — Fits on 24GB L4 / 40GB A100 GPU

Qwen2-VL-7B in 4-bit NF4 Quantization:
├── Base Model Weights (4-bit)         ~5.2 GB
├── LoRA Adapter Weights (r=32)        ~0.5 GB
├── 8-bit Paged AdamW Optimizer        ~0.8 GB
├── Gradient Checkpointing             ~6.5 GB
└── CUDA Overhead & Allocator          ~4.0 GB
                               TOTAL: ~17.0 GB  ✅ (Fits inside 24GB L4 VRAM)

Training Dataset Breakdown

Source	Raw Samples	Primary Target	Description
WebSight	50,000	Vision-Code Alignment	Synthetic HTML/CSS screenshots and code
VISION2UI	20,000	Real UI Fine-Tuning	Real-world website screenshots to code
Sketch2Code	5,000	Wireframe Translation	Hand-drawn wireframe sketches to HTML
shadcn Scraping	5,000	Library RAG Alignment	Official shadcn component screenshots to code
Astryx Scraping	3,000	StyleX Component SFT	Meta Astryx component variants to code
Material UI	3,000	MUI Component SFT	MUI v6 component examples to code
Synthetic (Gemini API)	15,000	Complex Layout SFT	Multi-tier dashboards, pricing tables, forms
TOTAL	~101,000	—	Unified Multimodal Dataset

Google Colab Training (Zero-RAM Streaming)

To run Stage 1 training on Google Colab (Free T4 GPU) without running out of RAM or disk space:

Open notebooks/XERO_Training_Colab.ipynb in Colab.
Select T4 GPU (Runtime -> Change runtime type -> T4 GPU).
Run the setup cell:

# Create script inside Colab
!python notebooks/colab_train.py --stage 1 --max-samples 1000

Zero-RAM Feature: Uses load_dataset(..., streaming=True) so images are fetched 1-by-1 during training, keeping RAM usage below 200 MB.

Hugging Face AutoTrain Advanced

For hands-off, 1-click cloud training with zero session disconnects:

pip install autotrain-advanced

autotrain vlm \
  --model HuggingFaceTB/SmolVLM2-2.2B-Instruct \
  --data-path your-username/xero-ui-dataset \
  --project-name xero-ui-model \
  --lr 2e-4 \
  --epochs 3 \
  --batch-size 2 \
  --target-modules q_proj,v_proj \
  --push-to-hub \
  --token $HF_TOKEN

Base Model vs. Fine-Tuned Adapters

When training completes, PyTorch outputs small, modular LoRA adapter files:

adapter_model.safetensors (~50 MB to ~150 MB)
adapter_config.json

These adapter weights are uploaded to your Hugging Face repository (huggingface.co/your-username/xero-ui-model). In src/xero/config.py, set:

adapter_id: str = "your-username/xero-ui-model"

At runtime, XERO loads the lightweight base model (SmolVLM2-2.2B-Instruct) and overlays your trained LoRA adapter weights on top.

🔌 Google Colab MCP Integration

Google provides an official open-source MCP server (googlecolab/colab-mcp) allowing AI coding agents to control your Colab runtime directly.

Configuration (`mcp.json` / Client Settings)

{
  "mcpServers": {
    "colab-mcp": {
      "command": "uvx",
      "args": ["git+https://github.com/googlecolab/colab-mcp"]
    }
  }
}

This enables your AI agent to execute code cells, monitor VRAM usage, and resolve training errors inside Colab autonomously.

🎨 Figma Integration

Extract design tokens and component hierarchy from any Figma design file using the free REST API:

from xero.figma.extractor import FigmaExtractor

# Initialize extractor with your Personal Access Token
extractor = FigmaExtractor(token="your_figma_personal_access_token")

# Extract data from Figma file URL
data = extractor.extract("https://www.figma.com/file/ABC123/MyDesign")

# Format data into markdown prompt context
context = extractor.format_as_context(data)
print(context)

Extracted Design Data

Category	Source in Figma	Output Format
Colors	Solid fills across nodes	HEX (`#6366f1`), RGBA (`rgba(99, 102, 241, 1)`), Style Names
Typography	TEXT node properties	Font Family, Font Size (px), Weight, Line Height
Spacing	Auto Layout gaps & padding	Array of pixel values `[4, 8, 12, 16, 24, 32]`
Border Radius	Corner radius attributes	Array of pixel values `[2, 4, 8, 12, 9999]`
Component Tree	Canvas node structure	Simplified JSON hierarchy (3 levels deep)
Frame Renderings	Top-level FRAME nodes	Exported PNG image URLs

🚢 Deployment Pipeline

XERO deploys natively to Hugging Face Spaces using ZeroGPU (free shared A100 GPU compute):

graph LR
    SUBMIT["User Submits Request"] --> GRP["Gradio App (HF Space)"]
    GRP --> ZGPU{"ZeroGPU Allocator"}
    ZGPU -->|"60s GPU Lease"| INF["SmolVLM2 + LoRA Adapter"]
    INF --> RAG_RET["Retrieve RAG Components"]
    RAG_RET --> GEN["Generate Code"]
    GEN --> RENDER["Live HTML/React Preview"]

✅ Evaluation Suite

Run the automated evaluation benchmark suite:

python -m tests.evaluate

Evaluation Check	Description	Target Benchmark
Render Success Rate	% of outputs that produce valid JSX/HTML syntax	> 85%
Import Accuracy	Correct import statements for the specified library	> 90%
Library Compliance	Uses components from the requested library	> 85%
Placeholder-Free Rate	Absence of "lorem ipsum" or TODO markers	> 90%
Export Integrity	Contains valid default/named component export	> 85%

🗺️ Roadmap

🤝 Contributing

Contributions are welcome! To add a new component library manifest:

Create a JSON file in libraries/your-library.json following the manifest format.
Run python -m xero.rag.index --library your-library to build the vector embeddings.
Test retrieval with python -m xero.rag.index --query "test query" --library your-library.
Open a Pull Request!

📄 License

Apache 2.0 — free for personal, educational, and commercial use.

Built with ❤️ using SmolVLM2 · PEFT · TRL · Gradio · ChromaDB

Navigation Menu

Repository files navigation

🧠 XERO

📋 Table of Contents

✨ Key Features

🏗️ System Architecture

End-to-End Execution Sequence

📚 Supported Component Libraries

Manifest Structure & Adding New Libraries

📁 Project Structure

🚀 Quick Start

1. Installation

2. Index Component Libraries

3. Launch Local Web Interface

🏋️ Training Pipeline

3-Stage Progressive Training

VRAM Budget & Hardware Math

SmolVLM2-2.2B-Instruct (4-bit QLoRA) — Fits on 16GB T4 GPU

Qwen2-VL-7B-Instruct (4-bit QLoRA) — Fits on 24GB L4 / 40GB A100 GPU

Training Dataset Breakdown

Google Colab Training (Zero-RAM Streaming)

Hugging Face AutoTrain Advanced

Base Model vs. Fine-Tuned Adapters

🔌 Google Colab MCP Integration

Configuration (`mcp.json` / Client Settings)

🎨 Figma Integration

Extracted Design Data

🚢 Deployment Pipeline

✅ Evaluation Suite

🗺️ Roadmap

🤝 Contributing

📄 License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
libraries		libraries
notebooks		notebooks
scratch		scratch
src/xero		src/xero
tests		tests
.gitignore		.gitignore
README.md		README.md
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

🧠 XERO

📋 Table of Contents

✨ Key Features

🏗️ System Architecture

End-to-End Execution Sequence

📚 Supported Component Libraries

Manifest Structure & Adding New Libraries

📁 Project Structure

🚀 Quick Start

1. Installation

2. Index Component Libraries

3. Launch Local Web Interface

🏋️ Training Pipeline

3-Stage Progressive Training

VRAM Budget & Hardware Math

SmolVLM2-2.2B-Instruct (4-bit QLoRA) — Fits on 16GB T4 GPU

Qwen2-VL-7B-Instruct (4-bit QLoRA) — Fits on 24GB L4 / 40GB A100 GPU

Training Dataset Breakdown

Google Colab Training (Zero-RAM Streaming)

Hugging Face AutoTrain Advanced

Base Model vs. Fine-Tuned Adapters

🔌 Google Colab MCP Integration

Configuration (mcp.json / Client Settings)

🎨 Figma Integration

Extracted Design Data

🚢 Deployment Pipeline

✅ Evaluation Suite

🗺️ Roadmap

🤝 Contributing

📄 License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Configuration (`mcp.json` / Client Settings)