🚀 Orchestrator – Firecracker Build System

Orchestrator is an experimental, Rust-based build orchestration system designed to execute builds inside isolated Firecracker microVMs for secure, deterministic, and reproducible builds.

This project explores the design and implementation of a minimal CI-style pipeline focused on strong isolation guarantees, explicit build lifecycles, and transparent artifact handling — without relying on Docker or heavyweight orchestration platforms.

⚠️ Project Status: MVP / R&D Orchestrator is intentionally scoped and not production-hardened. It exists to explore microVM-based build isolation, Firecracker control flows, and end-to-end build orchestration in Rust.

What this project is:

A hands-on exploration of Firecracker-based build isolation
A reference implementation of a microVM-driven build pipeline
A systems-level Rust project focused on clarity and control

What this project is not:

A drop-in replacement for GitHub Actions, Tekton, or Buildkite
A multi-tenant or horizontally scalable CI platform

🎯 MVP Features

✅ Firecracker VM Control: Start/stop/kill microVMs with unique sockets
✅ Git Repository Support: Clone repos with specific branch/commit
✅ Build Execution: Run builds inside isolated VMs with shared workspace
✅ Artifact Storage: Collect and store build artifacts
✅ REST API: HTTP endpoints for triggering and monitoring builds
✅ Real-time Status: Track build progress through multiple stages

🏗️ Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   HTTP API    │───▶│  Build Pipeline  │───▶│ Firecracker   │
│   (Axum)      │    │   (Worker)      │    │   Manager     │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Git Manager  │    │ Artifact Manager │    │   VM Storage  │
│   (git2)      │    │   (Filesystem)  │    │  (Sockets)    │
└─────────────────┘    └──────────────────┘    └─────────────────┘

📋 Prerequisites

System Requirements

Linux (Firecracker is Linux-only)
Rust 1.70+
Firecracker v1.13+
sudo access (for VM management)

Install Dependencies

Ubuntu/Debian

# System dependencies
sudo apt-get update
sudo apt-get install -y \
    build-essential \
    cmake \
    git \
    pkg-config \
    libssl-dev \
    libffi-dev \
    python3-dev \
    debootstrap \
    e2fsprogs \
    qemu-utils \
    curl \
    wget

# Rust (if not installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# Firecracker
sudo apt-get install -y firecracker
# Or download from GitHub releases
wget https://github.com/firecracker-microvm/firecracker/releases/download/v1.13.1/firecracker-v1.13.1-x86_64.tgz
tar xzf firecracker-v1.13.1-x86_64.tgz
sudo cp release-v1.13.1-x86_64/firecracker /usr/local/bin/
sudo chmod +x /usr/local/bin/firecracker

CentOS/RHEL

sudo yum groupinstall -y "Development Tools"
sudo yum install -y \
    cmake \
    git \
    pkgconfig \
    openssl-devel \
    libffi-devel \
    python3-devel \
    debootstrap \
    e2fsprogs \
    qemu-img

🛠️ Setup

1. Clone Repository

git clone <repository-url>
cd orchestrator

2. Build Rust Project

cargo build --release

3. Create Builder Rootfs

# This creates an Alpine Linux rootfs with Rust toolchain
./setup_builder_rootfs.sh

4. Setup Firecracker Directory Structure

# Create base directory
mkdir -p ~/.orchestrator/firecracker/{kernel,rootfs,sockets,vms,logs}

# Create or install a kernel image (vmlinux) suitable for Firecracker.
# You can provide a prebuilt vmlinux via URL or build one locally using the included helper script.
# Example (download a prebuilt vmlinux):
#   ./setup_kernel.sh --download https://example.com/path/to/vmlinux
# Example (build from kernel source tree on your machine):
#   ./setup_kernel.sh --build 5.15
# If you already have a vmlinux file, copy it here:
#   cp /path/to/vmlinux ~/.orchestrator/firecracker/kernel/
# The rootfs should be created by setup_builder_rootfs.sh

5. Configure Environment

# Optional: Set custom directory
export ORCHESTRATOR_DIR=/custom/path

# Optional: Set server host/port
export HOST=0.0.0.0
export PORT=8080

🚀 Running the Orchestrator

Development Mode

cargo run

Production Mode

cargo run --release

Custom Configuration

ORCHESTRATOR_DIR=/data/orchestrator HOST=0.0.0.0 PORT=9090 cargo run --release

The server will start on http://127.0.0.1:8080 by default.

📡 API Usage

Submit a Build

curl -X POST http://localhost:8080/build \
  -H "Content-Type: application/json" \
  -d '{
    "repo_url": "https://github.com/ari/platform-repo",
    "branch": "feature-x",
    "commit": "abc123def456"
  }'

Response:

{
  "build_id": "build_01234567-89ab-cdef-0123-456789abcdef",
  "status": "queued"
}

Check Build Status

curl http://localhost:8080/build/build_01234567-89ab-cdef-0123-456789abcdef

Response:

{
  "id": "build_01234567-89ab-cdef-0123-456789abcdef",
  "status": "completed",
  "created_at": "2024-01-15T10:30:00Z",
  "updated_at": "2024-01-15T10:35:00Z",
  "repo_url": "https://github.com/ari/platform-repo",
  "branch": "feature-x",
  "commit": "abc123def456",
  "artifacts_count": 3,
  "log": [
    "[2024-01-15 10:30:00 UTC] Build queued",
    "[2024-01-15 10:30:05 UTC] Cloning repository",
    "[2024-01-15 10:30:15 UTC] Starting VM",
    "[2024-01-15 10:30:20 UTC] Running build",
    "[2024-01-15 10:35:00 UTC] Collecting artifacts",
    "[2024-01-15 10:35:05 UTC] Build completed"
  ]
}

List All Builds

curl http://localhost:8080/builds

Health Check

curl http://localhost:8080/health

🔄 Build Lifecycle

Queued → Build request received
CloningRepo → Git repository being cloned
StartingVM → Firecracker VM starting
RunningBuild → Build command executing in VM
CollectingArtifacts → Gathering build outputs
Completed → Build finished successfully
Failed → Build failed with error message

📁 Directory Structure

~/.orchestrator/
├── firecracker/
│   ├── kernel/
│   │   └── vmlinux              # Linux kernel
│   ├── rootfs/
│   │   └── builder.ext4          # Builder rootfs
│   ├── sockets/                  # VM API sockets
│   ├── vms/                     # VM-specific data
│   └── logs/                    # Firecracker logs
├── builds/
│   └── <build_id>/
│       ├── repo/                 # Cloned repository
│       └── workspace/            # Shared workspace
└── artifacts/
    └── <build_id>/
        ├── target/release/        # Build artifacts
        ├── manifest.json        # Artifact manifest
        └── ...                # Other build outputs

🔧 Configuration

Environment Variables

ORCHESTRATOR_DIR: Base directory (default: ~/.orchestrator)
HOST: Server host (default: 127.0.0.1)
PORT: Server port (default: 8080)
RUST_LOG: Log level (default: info)

Firecracker asset paths

FIRECRACKER_KERNEL: Absolute path to the kernel image (vmlinux). If unset, defaults to ~/.orchestrator/firecracker/kernel/vmlinux.
FIRECRACKER_ROOTFS: Absolute path to the builder root filesystem (ext4). If unset, defaults to ~/.orchestrator/firecracker/rootfs/builder.ext4.

You can quickly inspect the resolved paths with:

cargo run --bin paths

Build Configuration

Builds use these default settings:

VM Resources: 2 vCPUs, 2GB RAM
Build Command: cargo build --release
Workspace: /workspace (shared host↔VM)
Timeout: 10 minutes per build

🐛 Troubleshooting

Common Issues

"Firecracker binary not found"

# Install Firecracker
which firecracker
# Should output: /usr/local/bin/firecracker

"Kernel not found"

# Use the helper script to download or build a kernel, or ensure a vmlinux exists in the kernel dir
./setup_kernel.sh --download <url-to-vmlinux>
ls -la ~/.orchestrator/firecracker/kernel/vmlinux

"Rootfs not found"

# Create builder rootfs
./setup_builder_rootfs.sh
ls -la ~/.orchestrator/firecracker/rootfs/builder.ext4

"Permission denied" errors

# Check permissions
ls -la ~/.orchestrator/
sudo chown -R $USER:$USER ~/.orchestrator/

Build stuck in "StartingVM"

# Check Firecracker logs
tail -f ~/.orchestrator/firecracker/logs/<vm_id>.log

# Check if socket exists
ls -la ~/.orchestrator/firecracker/sockets/

Debug Mode

# Enable debug logging
RUST_LOG=debug cargo run

# Check VM status
./monitor_firecracker_vms.sh

Clean Reset

# Stop all VMs
./kill_firecracker_vms.sh all

# Clean orchestrator data
rm -rf ~/.orchestrator/

Remove kernel/rootfs artifacts only

# Delete all vmlinux and *.ext4 images under your orchestrator data dir
find "${ORCHESTRATOR_DIR:-$HOME/.orchestrator}" -type f \( -name 'vmlinux' -o -name '*.ext4' \) -print -delete

🧪 Development

Running Tests

cargo test

Code Structure

src/
├── main.rs              # Application entry point
├── api/                 # HTTP API handlers
├── build/               # Build data structures
├── firecracker/         # VM management
├── git/                 # Git operations
├── artifacts/            # Artifact collection
└── pipeline/            # Build orchestration

Adding New Features

Add new API endpoints in src/api/mod.rs
Extend build pipeline in src/pipeline/mod.rs
Update data structures in src/build/mod.rs
Add configuration options as needed

🚧 MVP Limitations

❌ No networking between VM and host
❌ No persistent VMs (fresh VM per build)
❌ No parallel builds (single worker)
❌ No GitHub webhooks
❌ No artifact publishing
❌ No preview URLs
❌ No user accounts
❌ No Kubernetes integration

🗺️ Roadmap (Post-MVP)

Stretch Goals

✅ SSH into VM for debugging
✅ Multiple builders (VM pool)
✅ Snapshot-based fast builders
✅ GitHub App for automatic PR builds
✅ Preview deployments
✅ Full pipeline definitions
✅ API for listing builds, logs
✅ Artifact signing
✅ Multi-tenant isolation
✅ VMs in Firecracker jailer
✅ WASI+WASM build environments

📄 License

MIT License - see LICENSE file for details.

🤝 Contributing

Fork the repository
Create feature branch
Make your changes
Add tests
Submit pull request

📞 Support

Create an issue for bugs or feature requests
Check existing issues before creating new ones
Provide detailed reproduction steps

Built with ❤️ for secure, reproducible builds

Name		Name	Last commit message	Last commit date
Latest commit History 22 Commits
docker/traefik		docker/traefik
src		src
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
README.md		README.md
SUCCESS.md		SUCCESS.md
bake_ubuntu_rust_rootfs.sh		bake_ubuntu_rust_rootfs.sh
create_firecracker_socket.sh		create_firecracker_socket.sh
downloaded_binary		downloaded_binary
kill_firecracker_vms.sh		kill_firecracker_vms.sh
monitor_firecracker_vms.sh		monitor_firecracker_vms.sh
vm_status.sh		vm_status.sh

ari-rocha/orchestrator

Folders and files

Latest commit

History

Repository files navigation

🚀 Orchestrator – Firecracker Build System

🎯 MVP Features

🏗️ Architecture

📋 Prerequisites

System Requirements

Install Dependencies

Ubuntu/Debian

CentOS/RHEL

🛠️ Setup

1. Clone Repository

2. Build Rust Project

3. Create Builder Rootfs

4. Setup Firecracker Directory Structure

5. Configure Environment

🚀 Running the Orchestrator

Development Mode

Production Mode

Custom Configuration

📡 API Usage

Submit a Build

Check Build Status

List All Builds

Health Check

🔄 Build Lifecycle

📁 Directory Structure

🔧 Configuration

Environment Variables

Firecracker asset paths

Build Configuration

🐛 Troubleshooting

Common Issues

"Firecracker binary not found"

"Kernel not found"

"Rootfs not found"

"Permission denied" errors

Build stuck in "StartingVM"

Debug Mode

Clean Reset

Remove kernel/rootfs artifacts only

🧪 Development

Running Tests

Code Structure

Adding New Features

🚧 MVP Limitations

🗺️ Roadmap (Post-MVP)

Stretch Goals

📄 License

🤝 Contributing

📞 Support

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages