ErenAri · Jun 18, 2026 · Jun 18, 2026 · Jun 18, 2026 · Jun 18, 2026 · Jun 18, 2026
@@ -41,6 +41,24 @@
 # Generated static compatibility site
 /public/compatibility/
 
+# Credentials & secrets — never commit (deploy / Hetzner / demo)
+# Real env files carry BPFCOMPAT_API_WRITE_KEY etc.; only the .example templates
+# are tracked. SSH private keys and TLS material must never land in the repo.
+*.env
+!*.env.example
+*.pem
+*.key
+*.ppk
+id_rsa
+id_ecdsa
+id_ed25519
+*_rsa
+*_ed25519
+known_hosts
+# Local deploy scratch (server IPs, installimage configs, notes)
+/.hetzner/
+/.secrets/
+
 # Common temporary files
 *.log
 *.tmp

@@ -405,6 +405,12 @@ azure-configure-tls:
 azure-rotate-registry-secret:
 	bash scripts/azure-rotate-registry-secret.sh
 
+hetzner-bootstrap-vm:
+	bash scripts/hetzner-bootstrap-vm.sh
+
+hetzner-configure-tls:
+	bash scripts/hetzner-configure-tls.sh
+
 clean:
 	rm -rf bin
 	rm -rf .bpfcompat

@@ -15,7 +15,7 @@ The core question is simple:
 > Will this `.bpf.o` load and attach on the kernels I care about, and if not,
 > what failed?
 
-**Live demo:** [demo.kernelguard.net](https://demo.kernelguard.net) — upload a
+**Live demo:** [bpfcompat.kernelguard.net](https://bpfcompat.kernelguard.net) — upload a
 `.bpf.o` and see the compatibility matrix.
 
 ## Why not just rely on CO-RE / BTFHub?

@@ -0,0 +1,85 @@
+# Hetzner demo runbook
+
+Move / host the public bpfcompat demo (`bpfcompat.kernelguard.net`) on a Hetzner
+bare-metal server. This replaces the Azure VM after the Azure credit expired.
+
+## Why bare metal (not Hetzner Cloud)
+
+The demo boots real x86 QEMU/KVM guests on each validation. **Hetzner Cloud
+(shared vCPU, dedicated vCPU, and ARM/CAX alike) does not expose nested
+virtualization**, so there is no `/dev/kvm` and QEMU silently falls back to TCG
+software emulation (~10x slower). Only a Hetzner **dedicated / Server Auction**
+host gives native KVM. The bootstrap script hard-fails if `/dev/kvm` is missing.
+
+## 0) Order the server (manual, your Hetzner account)
+
+1. Open the [Server Auction](https://www.hetzner.com/sb/) (no setup fee, prices
+   unaffected by the June 2026 adjustment). Use
+   [Server Radar](https://radar.iodev.org/) to filter.
+2. Pick any modern x86-64 box (e.g. Ryzen/EPYC, 32GB+ RAM, NVMe). All current
+   CPUs support VT-x/AMD-V; KVM works out of the box on bare metal.
+3. Install **Ubuntu 22.04 / 24.04 LTS** (via the install image or rescue
+   system). Add your SSH public key during install so `root` login works.
+4. Note the server's public IP -> `HETZNER_HOST`.
+
+## 1) Bootstrap toolchain + build + demo unit
+
+Installs the build toolchain and `qemu-kvm`, verifies `/dev/kvm`, builds
+`bpfcompat` + the static validator + examples, creates the `bpfcompat-demo`
+service user (in the `kvm` group), and installs the `bpfcompat-serve` systemd
+unit and `/etc/bpfcompat/serve.env` stub.
+
+```bash
+export HETZNER_HOST=<server-ip>
+# optional: export HETZNER_USER=root HETZNER_SSH_KEY=~/.ssh/id_ed25519
+make hetzner-bootstrap-vm
+```
+
+## 2) Set the write key and start the demo server
+
+The server binds `127.0.0.1:8080` only. Anonymous visitors may validate and read
+history; writes require the private key.
+
+```bash
+ssh root@$HETZNER_HOST \
+  "sudo sed -i \"s|^BPFCOMPAT_API_WRITE_KEY=.*|BPFCOMPAT_API_WRITE_KEY=$(openssl rand -hex 32)|\" /etc/bpfcompat/serve.env"
+ssh root@$HETZNER_HOST "sudo systemctl enable --now bpfcompat-serve.service"
+ssh root@$HETZNER_HOST "systemctl --no-pager status bpfcompat-serve.service"
+```
+
+## 3) Repoint DNS (Cloudflare)
+
+Update the `bpfcompat.kernelguard.net` `A` record from the old Azure IP
+(`20.91.218.19`) to `$HETZNER_HOST`. Keep it **DNS only (grey cloud)** so Caddy
+can complete the Let's Encrypt HTTP-01 challenge.
+
+## 4) Configure HTTPS + host firewall (Caddy + ufw)
+
+`ufw` opens only 22/80/443; the backend stays on `127.0.0.1:8080`.
+
+```bash
+export HETZNER_HOST=<server-ip>
+export BPFCOMPAT_DOMAIN=bpfcompat.kernelguard.net
+make hetzner-configure-tls
+```
+
+After completion:
+
+- UI:     `https://bpfcompat.kernelguard.net/`
+- Health: `https://bpfcompat.kernelguard.net/api/health`
+
+## 5) Verify, then decommission Azure
+
+```bash
+curl -fsS https://bpfcompat.kernelguard.net/api/health
+```
+
+Once green, tear down the Azure VM/resource group to stop any residual billing.
+
+## Security notes
+
+- Backend never listens publicly: `--addr 127.0.0.1:8080` behind Caddy + `ufw`.
+- `BPFCOMPAT_API_ENABLE_RUNTIME_EXECUTE=false` -- no host eBPF loading on the demo.
+- Public reports are sanitized server-side (`internal/api/sanitize.go`); host
+  paths, `vm_run_dir`, `qemu_command`, and `serial_log` never reach the browser.
+- `.bpfcompat/runs/**` (per-run SSH keys) lives only under `/var/lib/bpfcompat-demo`.
@@ -0,0 +1,26 @@
+# Environment for the public demo bpfcompat serve unit.
+# Copy to /etc/bpfcompat/serve.env (mode 0600) and edit before enabling the unit.
+#
+# Demo posture: anonymous visitors may run validations and read history/status,
+# but may NOT write to the registry or trigger host runtime execution. Generate a
+# strong random write key and keep it private (operators use it for writes).
+
+# Required: private write key. Generate with: openssl rand -hex 32
+BPFCOMPAT_API_WRITE_KEY=
+
+# Demo openness: allow anonymous validate + read-only history/status.
+BPFCOMPAT_API_ALLOW_ANONYMOUS_VALIDATE=true
+BPFCOMPAT_API_ALLOW_ANONYMOUS_READ=true
+BPFCOMPAT_API_ALLOW_ANONYMOUS_RUNTIME_DELIVERY=false
+
+# Host runtime execution stays OFF on the public demo.
+BPFCOMPAT_API_ENABLE_RUNTIME_EXECUTE=false
+BPFCOMPAT_API_RUNTIME_EXECUTE_KILL_SWITCH=false
+BPFCOMPAT_API_REDACT_RUNTIME_DETAILS=true
+
+# Do not mirror demo runs into a cloud registry.
+BPFCOMPAT_API_AUTO_SYNC_REGISTRY=false
+
+# Note: the run tree location is set via the --workdir flag in the unit's
+# ExecStart (/var/lib/bpfcompat-demo), not an env var. .bpfcompat/runs/** under
+# it holds per-run SSH keys, so it must stay inside the state dir only.
@@ -0,0 +1,39 @@
+[Unit]
+Description=bpfcompat public demo web/API server
+Documentation=https://github.com/kernel-guard/bpfcompat
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=simple
+User=bpfcompat-demo
+Group=bpfcompat-demo
+# The demo boots real QEMU/KVM guests, so the service user needs /dev/kvm.
+SupplementaryGroups=kvm
+EnvironmentFile=/etc/bpfcompat/serve.env
+# WorkingDirectory is the cloned repo so relative data paths (matrices/, vm/
+# profiles, examples) resolve; run artifacts are written to the writable
+# --workdir state dir instead.
+WorkingDirectory=/opt/bpfcompat-src
+ExecStart=/usr/local/bin/bpfcompat serve --addr 127.0.0.1:8080 --workdir /var/lib/bpfcompat-demo --matrix matrices/mvp.yaml
+Restart=on-failure
+RestartSec=3
+
+# Hardening compatible with running QEMU/KVM. Note this is deliberately looser
+# than bpfcompat-agent.service: the demo spawns qemu, which needs /dev/kvm and a
+# writable run tree, so ProtectSystem=strict and MemoryDenyWriteExecute are NOT
+# used here (they break KVM and the TCG JIT respectively).
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectHome=true
+ProtectSystem=full
+ReadWritePaths=/var/lib/bpfcompat-demo
+StateDirectory=bpfcompat-demo
+LogsDirectory=bpfcompat-demo
+DeviceAllow=/dev/kvm rw
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
+LockPersonality=true
+RestrictRealtime=true
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,131 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Bootstrap a Hetzner bare-metal (dedicated / Server Auction) host for the
+# bpfcompat public demo. Hetzner has no `az vm run-command` equivalent, so this
+# runs over plain SSH. Bare metal is required because Hetzner Cloud does NOT
+# expose nested virtualization (no /dev/kvm); the demo boots real QEMU/KVM
+# guests, so this script hard-fails if /dev/kvm is absent.
+#
+# Required env:
+#   HETZNER_HOST                      public IP or hostname of the server
+#
+# Optional:
+#   HETZNER_USER                      ssh user (default: root; Hetzner rescue/initial install gives root)
+#   HETZNER_SSH_KEY                   path to ssh private key (default: ssh-agent / ~/.ssh default)
+#   BPFCOMPAT_REPO_URL                git URL to clone (default: https://github.com/Kernel-Guard/bpfcompat.git)
+#   BPFCOMPAT_REPO_REF                branch/tag/commit to check out (default: main)
+
+HETZNER_HOST="${HETZNER_HOST:-}"
+HETZNER_USER="${HETZNER_USER:-root}"
+HETZNER_SSH_KEY="${HETZNER_SSH_KEY:-}"
+BPFCOMPAT_REPO_URL="${BPFCOMPAT_REPO_URL:-https://github.com/Kernel-Guard/bpfcompat.git}"
+BPFCOMPAT_REPO_REF="${BPFCOMPAT_REPO_REF:-main}"
+# bpfcompat needs a modern Go (see go.mod); distro packages lag, so we install
+# the official toolchain. Keep this in sync with the go directive in go.mod.
+GO_VERSION="${GO_VERSION:-1.25.0}"
+
+if [[ -z "$HETZNER_HOST" ]]; then
+  echo "[hetzner-bootstrap-vm] set HETZNER_HOST first" >&2
+  exit 1
+fi
+
+SSH_OPTS=(-o StrictHostKeyChecking=accept-new -o ConnectTimeout=20)
+if [[ -n "$HETZNER_SSH_KEY" ]]; then
+  SSH_OPTS+=(-i "$HETZNER_SSH_KEY")
+fi
+
+echo "[hetzner-bootstrap-vm] bootstrapping ${HETZNER_USER}@${HETZNER_HOST}"
+
+# shellcheck disable=SC2087  # heredoc is intentionally expanded locally for repo url/ref
+ssh "${SSH_OPTS[@]}" "${HETZNER_USER}@${HETZNER_HOST}" bash -s <<EOF
+set -euo pipefail
+export DEBIAN_FRONTEND=noninteractive
+
+echo "[remote] installing toolchain + qemu/kvm"
+sudo apt-get update -y
+sudo apt-get install -y \
+  build-essential ca-certificates curl git jq make pkg-config \
+  clang llvm libbpf-dev libelf-dev zlib1g-dev zstd \
+  qemu-system-x86 qemu-utils qemu-kvm openssh-client
+
+# bpfcompat requires Go ${GO_VERSION}; Ubuntu's apt golang is too old, so install
+# the official toolchain to /usr/local/go and symlink it onto the default PATH
+# (non-interactive ssh sessions do not source /etc/profile.d).
+if ! { command -v go >/dev/null 2>&1 && go version | grep -q "go${GO_VERSION} "; }; then
+  echo "[remote] installing Go ${GO_VERSION}"
+  curl -fsSL "https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz" -o /tmp/go.tgz
+  sudo rm -rf /usr/local/go
+  sudo tar -C /usr/local -xzf /tmp/go.tgz
+  sudo ln -sf /usr/local/go/bin/go /usr/local/bin/go
+  sudo ln -sf /usr/local/go/bin/gofmt /usr/local/bin/gofmt
+  rm -f /tmp/go.tgz
+fi
+go version
+
+# Bare metal MUST expose /dev/kvm. If it is missing the box is not suitable for
+# the demo (no native virt) -- fail loudly rather than silently fall back to TCG.
+if [ ! -e /dev/kvm ]; then
+  echo "[remote] FATAL: /dev/kvm missing -- this host cannot run KVM guests." >&2
+  echo "[remote] Confirm you ordered a bare-metal server (not Hetzner Cloud) with VT-x/AMD-V." >&2
+  exit 1
+fi
+echo "[remote] kvm_device=present"
+
+echo "[remote] creating bpfcompat-demo service user (in kvm group)"
+if ! getent group bpfcompat-demo >/dev/null; then
+  sudo groupadd --system bpfcompat-demo
+fi
+if ! id -u bpfcompat-demo >/dev/null 2>&1; then
+  sudo useradd --system --gid bpfcompat-demo --groups kvm \
+    --home-dir /var/lib/bpfcompat-demo --shell /usr/sbin/nologin bpfcompat-demo
+else
+  sudo usermod -aG kvm bpfcompat-demo
+fi
+sudo install -d -m 0750 -o bpfcompat-demo -g bpfcompat-demo /var/lib/bpfcompat-demo
+
+echo "[remote] cloning ${BPFCOMPAT_REPO_URL} @ ${BPFCOMPAT_REPO_REF}"
+sudo rm -rf /opt/bpfcompat-src
+sudo git clone "${BPFCOMPAT_REPO_URL}" /opt/bpfcompat-src
+cd /opt/bpfcompat-src
+sudo git checkout "${BPFCOMPAT_REPO_REF}"
+
+echo "[remote] building bpfcompat + static validator + examples"
+sudo make build
+sudo make validator-static
+sudo make examples
+
+echo "[remote] installing /usr/local/bin/bpfcompat"
+sudo install -m 0755 ./bin/bpfcompat /usr/local/bin/bpfcompat
+
+# The repo tree is root-owned (cloned via sudo) and the service runs as the
+# unprivileged bpfcompat-demo user with WorkingDirectory here. It writes a few
+# runtime dirs relative to the repo (UI report copies, API state), so make those
+# writable by the service user. VM run overlays go to the separate --workdir.
+sudo install -d -o bpfcompat-demo -g bpfcompat-demo /opt/bpfcompat-src/reports /opt/bpfcompat-src/.bpfcompat-api
+
+echo "[remote] installing demo serve systemd unit + env stub"
+sudo install -d -m 0750 /etc/bpfcompat
+if [ ! -f /etc/bpfcompat/serve.env ]; then
+  sudo install -m 0600 packaging/systemd/bpfcompat-serve.env.example /etc/bpfcompat/serve.env
+  echo "[remote] created /etc/bpfcompat/serve.env -- set BPFCOMPAT_API_WRITE_KEY before enabling"
+fi
+sudo install -m 0644 packaging/systemd/bpfcompat-serve.service /etc/systemd/system/bpfcompat-serve.service
+sudo systemctl daemon-reload
+
+echo "[remote] bpfcompat version:"
+/usr/local/bin/bpfcompat version || /usr/local/bin/bpfcompat --version || true
+EOF
+
+cat <<MSG
+
+[hetzner-bootstrap-vm] done. Next:
+  1. Set the write key on the server:
+       ssh ${HETZNER_USER}@${HETZNER_HOST} \\
+         "sudo sed -i 's|^BPFCOMPAT_API_WRITE_KEY=.*|BPFCOMPAT_API_WRITE_KEY='\$(openssl rand -hex 32)'|' /etc/bpfcompat/serve.env"
+  2. Enable the demo server (binds 127.0.0.1:8080):
+       ssh ${HETZNER_USER}@${HETZNER_HOST} "sudo systemctl enable --now bpfcompat-serve.service"
+       ssh ${HETZNER_USER}@${HETZNER_HOST} "systemctl --no-pager status bpfcompat-serve.service"
+  3. Point DNS A record at ${HETZNER_HOST}, then run:
+       HETZNER_HOST=${HETZNER_HOST} BPFCOMPAT_DOMAIN=demo.kernelguard.net make hetzner-configure-tls
+MSG