[Feature]: Add MI300X as a GitHub runner/via SSH

[v01dXYZ]

What feature or enhancement are you proposing?

Add to the Github runner fleet instances with one MI300X GPU.

The two providers with cheap 1GPU VMs (1.99USD/hr) are:

• Digital Ocean: but GPUs are not available currently
• Hot Aisle: The one I used, it has a cli client to provision and delete VMs (beware if you provision and delete too many times, the CLI client stops to immediately delete instances and you have to rely on the admin panel instead).

The script to initialize the instance is the following one. It is complicated as it tries to parallelize downloading PyTorch and also uncompressing some heavy ROCM debs when upgrading ROCm. It runs around the 5mn mark (15ct).

set -e

function print_bar() {
    echo "####################################################################################################"
}
function print_between_bar() {
    print_bar
    echo "#  " $@
    print_bar
}

####################################################################################################
print_between_bar "Install uv"
####################################################################################################

curl -LsSf https://astral.sh/uv/install.sh | sh

export PATH=$HOME/.local/bin:$PATH

####################################################################################################
print_between_bar "Create venv and download pytorch in background (6GB)"
####################################################################################################

uv venv venv
. venv/bin/activate

torch_pyindex=https://download.pytorch.org/whl/rocm7.2
torch_download_url=$(uv pip install --force-reinstall --no-deps torch -i $torch_pyindex -v 2>&1 | grep -m1 -o "https://.*torch-.*whl" &)
kill $(jobs -l | grep "uv pip install" | cut -f2 -d" ") || echo "no worry, uv killed by SIGPIP"

sudo apt install -y aria2
aria2c -j8 -x8 --summary-interval=1 $torch_download_url &

####################################################################################################
print_between_bar "Upgrade"
####################################################################################################

sudo apt update -y
sudo apt-mark hold linux-generic linux-libc-dev
sudo DEBIAN_FRONTEND=noninteractive apt -y upgrade
sudo apt install -y vim emacs clang liblz4-dev libssl-dev libzstd-dev aria2

####################################################################################################
print_between_bar "Install apt-fast"
####################################################################################################

# otherwise add-apt-repository slow according to 
# https://askubuntu.com/questions/1110626/add-apt-repository-command-is-suddenly-very-slow
sudo sysctl net.ipv6.conf.all.disable_ipv6=1 
sudo add-apt-repository ppa:apt-fast/stable
sudo apt-get update -y
sudo DEBIAN_FRONTEND=noninteractive apt-get -y install apt-fast
sudo sysctl net.ipv6.conf.all.disable_ipv6=0 # put it back as it was before
# _APTMGR="apt-get"

####################################################################################################
print_between_bar "Update ROCM to 7.2.1 (copied from rocm docs)"
#  (with ROCM 7.2.0, we get error messages about GPU staying in low power mode)
####################################################################################################

wget https://repo.radeon.com/amdgpu-install/7.2.1/ubuntu/jammy/amdgpu-install_7.2.1.70201-1_all.deb
sudo apt install ./amdgpu-install_7.2.1.70201-1_all.deb
sudo apt update -y
sudo apt install -y python3-setuptools python3-wheel
sudo usermod -a -G render,video $LOGNAME # Add the current user to the render and video groups

####################################################################################################
print_between_bar "Update ROCm with some parallel uncompression"
####################################################################################################

mkdir $HOME/apt-cache-rocm
sudo APTCACHE=$HOME/apt-cache-rocm _MAXNUM=20 _MAXCONPERSERV=40 _SPLITCON=32 apt-fast install -d -y rocm

rm -rf unpacked packed pkgs_links
mkdir pkgs_links/

cd $HOME/apt-cache-rocm/
for i in *.deb; do
    ln -s $HOME/apt-cache-rocm/$i $HOME/pkgs_links
done

cd $HOME

mkdir unpacked packed
pkgs_to_unpack=$(du -hd0 apt-cache-rocm/* | grep "\([0-9]\{3,\}M\|[^ ]\+G\)" | cut -f2 | cut -d/ -f2)

for pkg in $pkgs_to_unpack; do
    echo "Processing $pkg $(date)"
    rm pkgs_links/$pkg
    pkg=$pkg bash -c 'set -e;\
                      dpkg-deb -R {apt-cache-rocm,unpacked}/$pkg ;\
                      echo $pkg UNPACK DONE $(date) ;\
                      dpkg-deb -Znone -b {unpacked,packed}/$pkg ;\
                      echo $pkg DONE $(date) ;\
                      sleep 1 ;\
                      while ! sudo dpkg -i packed/$pkg 2>/dev/null; do sleep 1; done' &
done
sudo dpkg -i pkgs_links/*.deb &

wait

sudo dpkg --configure -a

####################################################################################################
print_between_bar "Upgrade again"
####################################################################################################

sudo apt-mark hold amdgpu-dkms
sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y

####################################################################################################
print_between_bar "Install pytorch"
####################################################################################################

uv pip install ~/torch*.whl -i $torch_pyindex

####################################################################################################
print_between_bar "Clone and install genesis as editable"
####################################################################################################

git clone https://github.com/Genesis-Embodied-AI/genesis -j24
cd ~/genesis
uv pip install -e .[dev]

# disabled as normally you should build the wheel on another machine since LLVM is statically linked anyway
if false; then
    ####################################################################################################
    print_between_bar "Clone, build and install quadrants"
    ####################################################################################################

    git clone https://github.com/Genesis-Embodied-AI/quadrants --recurse-submodules -j24
    cd quadrants
    uv pip install --group dev
    uv pip install --group test

    uv pip install -r requirements_test_xdist.txt

    uv pip install pip

    # we put it in a shell script to easily call it later on
    echo 'QUADRANTS_CMAKE_ARGS="-DQD_WITH_AMDGPU=ON" ./build.py' >> build.sh
    chmod +x build.sh

    ./build.sh

    # install quadrants
    uv pip install dist/*.whl
fi

With ephemeral instances, the following options could be useful:

ssh  -o 'UserKnownHostsFile=/dev/null' \
  -o 'StrictHostKeyChecking=no' \
  -I <private_key> \
  hotaisle@$1

Motivation

Add to the Github runner fleet a ephemeral runner with a MI300X

Potential Benefit

Encourage developers to better target AMDGPU and improve performance on this platform as well. Allow a comparison with Nvidia GPUs and possibly advertise Genesis as cheaper to use than other engines with only Nvidia support.

What is the expected outcome of the implementation work?

• Early catch things that are not compatible with AMDGPU or lead to drastic performance degradation
• Speed and memory benchmarks to compare with Nvidia

Additional information

No response

[duburcqa]

Claude gave me this plan. What do you think about it?

 AMD GPU CI + Dev Environment (Hot Aisle, On-Demand)

 Context

 Genesis needs AMD MI300X for CI (unit tests on ROCm) and dev work (direct SSH). Using Hot Aisle's on-demand ephemeral model: $1.99/GPU/hr, per-minute billing, ROCm pre-installed, ~1-2 min spin-up.

 How it works

 Hot Aisle provides:
 - API: POST /teams/{team}/virtual_machines/ to create, DELETE to destroy
 - CLI: brew install hotaisle/tap/hotaisle — wraps the API
 - Cloud-init: user_data_url parameter on VM creation runs a script on first boot
 - SSH: Direct public IP, port 22, ROCm + Docker pre-installed on every VM

 CI flow: GHA workflow → create VM via API → SSH in and run tests → destroy VM
 Dev flow: hotaisle CLI → create VM → SSH in → destroy when done

 Files to create

 1. .github/workflows/production-amd.yml — CI workflow

 Uses the Hot Aisle API directly from a GHA workflow (no self-hosted runner needed). Provisions a VM, runs tests over SSH, tears it down.

 name: Production (AMD)

 on:
   push:
     branches: [main]
   pull_request:
     branches: [main]

 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
   cancel-in-progress: ${{ github.event_name == 'pull_request' }}

 env:
   HOTAISLE_TEAM: genesis          # Hot Aisle team name
   FORCE_COLOR: 1
   PY_COLORS: 1

 jobs:
   unit-tests:
     name: amd-unit_tests-${{ matrix.GS_ENABLE_NDARRAY == '0' && 'field' || 'ndarray' }}
     runs-on: ubuntu-24.04
     if: github.event_name == 'pull_request'

     strategy:
       fail-fast: true
       max-parallel: 1
       matrix:
         GS_ENABLE_NDARRAY: ["0", "1"]

     steps:
       - uses: actions/checkout@v4

       - name: Provision MI300X VM
         id: vm
         env:
           HOTAISLE_API_KEY: ${{ secrets.HOTAISLE_API_KEY }}
         run: |
           # Create VM via Hot Aisle API with cloud-init for env setup
           VM_NAME="ci-amd-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
           RESPONSE=$(curl -sf -X POST \
             "https://admin.hotaisle.app/api/teams/${HOTAISLE_TEAM}/virtual_machines/" \
             -H "Authorization: Bearer ${HOTAISLE_API_KEY}" \
             -H "Content-Type: application/json" \
             -d '{
               "cpu_cores": 24,
               "ram_capacity": 257698037760,
               "disk_capacity": 214748364800,
               "gpus": [{"count": 1, "manufacturer": "AMD", "model": "MI300X"}],
               "user_data_url": "https://raw.githubusercontent.com/Genesis-Embodied-AI/Genesis/main/infra/cloud-init-amd.yml"
             }')
           VM_IP=$(echo "$RESPONSE" | jq -r '.ip_address')
           VM_NAME=$(echo "$RESPONSE" | jq -r '.name')
           echo "vm_ip=${VM_IP}" >> "$GITHUB_OUTPUT"
           echo "vm_name=${VM_NAME}" >> "$GITHUB_OUTPUT"

           # Wait for SSH to become available
           for i in $(seq 1 60); do
             ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@${VM_IP} true 2>/dev/null && break
             sleep 5
           done

       - name: Run unit tests
         env:
           GS_ENABLE_NDARRAY: ${{ matrix.GS_ENABLE_NDARRAY }}
           HF_TOKEN: ${{ secrets.HF_TOKEN }}
         run: |
           VM_IP="${{ steps.vm.outputs.vm_ip }}"

           # Copy repo to VM
           rsync -az --delete -e "ssh -o StrictHostKeyChecking=no" \
             ./ root@${VM_IP}:/root/workspace/

           # Run tests
           ssh -o StrictHostKeyChecking=no root@${VM_IP} bash -s <<EOF
             set -e
             source /venv/bin/activate
             cd /root/workspace
             export GS_ENABLE_NDARRAY=${GS_ENABLE_NDARRAY}
             export QD_ENABLE_AMDGPU=1
             export QD_ENABLE_CUDA=0
             export HF_TOKEN=${HF_TOKEN}
             pytest -v -ra --backend gpu --dev --forked ./tests
           EOF

       - name: Destroy VM
         if: always()
         env:
           HOTAISLE_API_KEY: ${{ secrets.HOTAISLE_API_KEY }}
         run: |
           VM_NAME="${{ steps.vm.outputs.vm_name }}"
           if [ -n "${VM_NAME}" ]; then
             curl -sf -X DELETE \
               "https://admin.hotaisle.app/api/teams/${HOTAISLE_TEAM}/virtual_machines/${VM_NAME}?force=true" \
               -H "Authorization: Bearer ${HOTAISLE_API_KEY}"
           fi

 2. infra/cloud-init-amd.yml — First-boot provisioning

 Runs automatically on VM creation via user_data_url. Installs Genesis environment so tests can start immediately after SSH is up.

 #cloud-config
 runcmd:
   # Install uv
   - curl -LsSf https://astral.sh/uv/install.sh | sh
   - export PATH="$HOME/.local/bin:$PATH"

   # Create venv with ROCm PyTorch
   - uv venv --python '3.10' /venv
   - /venv/bin/pip install torch --index-url https://download.pytorch.org/whl/rocm6.4
   - /venv/bin/pip install --upgrade pip setuptools wheel

   # Signal readiness (cloud-init done)
   - touch /var/run/cloud-init-done

 Note: Genesis itself is installed from the repo copy that gets rsync'd by the CI workflow. The cloud-init only sets up the base environment (Python venv + ROCm PyTorch) to minimize boot time.

 For dev sessions, you'd also want to install Genesis in the cloud-init. A separate dev cloud-init template can add pip install ".[dev,render,usd]" and dev tools (tmux, htop, etc.).

 3. infra/amd-dev.sh — Dev session helper script

 Simple wrapper around the hotaisle CLI for dev work.

 #!/usr/bin/env bash
 # Provision/destroy an on-demand MI300X dev instance on Hot Aisle.
 #
 # Prerequisites: hotaisle CLI installed and configured (brew install hotaisle/tap/hotaisle)
 #
 # Usage:
 #   ./infra/amd-dev.sh up      # Create instance, print SSH command
 #   ./infra/amd-dev.sh down    # Destroy instance
 #   ./infra/amd-dev.sh ssh     # SSH into active instance
 #   ./infra/amd-dev.sh status  # Show instance status
 set -euo pipefail

 TEAM="genesis"
 VM_PREFIX="dev-amd"
 STATE_FILE="${HOME}/.cache/genesis/amd-dev-vm"

 case "${1:-status}" in
   up)
     echo "Creating MI300X dev instance..."
     RESPONSE=$(curl -sf -X POST \
       "https://admin.hotaisle.app/api/teams/${TEAM}/virtual_machines/" \
       -H "Authorization: Bearer ${HOTAISLE_API_KEY}" \
       -H "Content-Type: application/json" \
       -d '{
         "cpu_cores": 24,
         "ram_capacity": 257698037760,
         "disk_capacity": 214748364800,
         "gpus": [{"count": 1, "manufacturer": "AMD", "model": "MI300X"}],
         "user_data_url": "https://raw.githubusercontent.com/Genesis-Embodied-AI/Genesis/main/infra/cloud-init-amd-dev.yml"
       }')
     VM_IP=$(echo "$RESPONSE" | jq -r '.ip_address')
     VM_NAME=$(echo "$RESPONSE" | jq -r '.name')

     mkdir -p "$(dirname "$STATE_FILE")"
     echo "${VM_NAME} ${VM_IP}" > "$STATE_FILE"

     # Wait for SSH
     echo "Waiting for SSH..."
     for i in $(seq 1 60); do
       ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@${VM_IP} true 2>/dev/null && break
       sleep 5
     done

     echo ""
     echo "=== MI300X Dev Instance Ready ==="
     echo "SSH:     ssh root@${VM_IP}"
     echo "Destroy: ./infra/amd-dev.sh down"
     ;;

   down)
     if [ ! -f "$STATE_FILE" ]; then echo "No active instance."; exit 0; fi
     VM_NAME=$(awk '{print $1}' "$STATE_FILE")
     echo "Destroying ${VM_NAME}..."
     curl -sf -X DELETE \
       "https://admin.hotaisle.app/api/teams/${TEAM}/virtual_machines/${VM_NAME}?force=true" \
       -H "Authorization: Bearer ${HOTAISLE_API_KEY}"
     rm -f "$STATE_FILE"
     echo "Done."
     ;;

   ssh)
     if [ ! -f "$STATE_FILE" ]; then echo "No active instance. Run: $0 up"; exit 1; fi
     VM_IP=$(awk '{print $2}' "$STATE_FILE")
     exec ssh -o StrictHostKeyChecking=no root@${VM_IP}
     ;;

   status)
     if [ ! -f "$STATE_FILE" ]; then echo "No active instance."; exit 0; fi
     VM_NAME=$(awk '{print $1}' "$STATE_FILE")
     VM_IP=$(awk '{print $2}' "$STATE_FILE")
     echo "Active: ${VM_NAME} @ ${VM_IP}"
     echo "SSH:    ssh root@${VM_IP}"
     ;;

   *)
     echo "Usage: $0 {up|down|ssh|status}"
     exit 1
     ;;
 esac

 One-time setup

 1. brew install hotaisle/tap/hotaisle
 2. ssh admin.hotaisle.app → create team, add credits, generate API key
 3. Add HOTAISLE_API_KEY as GitHub repo secret
 4. Add HF_TOKEN as GitHub repo secret (if not already)
 5. Push the 3 files above

 Cost estimate

 - CI (2 matrix jobs × ~15 min each per PR): ~$1/PR
 - Dev session (4h/day, weekdays): ~$160/mo
 - Total: ~$180/mo (vs $1,440/mo persistent)

 Verification

 1. CI: Open test PR → production-amd.yml triggers → VM created → tests run → VM destroyed
 2. Dev: ./infra/amd-dev.sh up → SSH in → rocm-smi shows MI300X → run Genesis on AMD backend
 3. Cleanup: Check Hot Aisle dashboard for orphaned VMs after CI runs (the if: always() step should handle it)

[v01dXYZ]

@duburcqa It seems fair. Although it doesn't update ROCm, it seems to be a good first version that would need further tweaking and refinements.

Beware:

• the POST doesn't need to have constraint on memory, only gpu_model and gpu_count are needed
• the cloud init script needs to be uploaded before!
• install libegl1

Possible refinements:

• upgrade ROCm
• manage when there is no instance available
• add other tests (esp. benchkmark/mem-report)

[v01dXYZ]

The fix I'm made in #2680 allowed me to run the benchmarks, it is quite bad (but it could because of me misconfiguring the GPU, genesis or the test suite). On the other hand, I think the comparison is done with a 8xRTX Pro 6000, so maybe not so bad.

cf #2680 (comment)