diff --git a/.github/federation.json b/.github/federation.json index 74f28ec..40a4126 100644 --- a/.github/federation.json +++ b/.github/federation.json @@ -1,5 +1,15 @@ { "sources": [ + { + "repo": "AMD-AGI/Hyperloom", + "license": "MIT", + "skills": [ + { + "path": "examples/skills/hyperloom-workload-optimizer", + "as": "hyperloom-workload-optimizer" + } + ] + }, { "repo": "AMD-AGI/TraceLens", "license": "MIT", diff --git a/skills/hyperloom-workload-optimizer/SKILL.md b/skills/hyperloom-workload-optimizer/SKILL.md index 9916baf..0499520 100644 --- a/skills/hyperloom-workload-optimizer/SKILL.md +++ b/skills/hyperloom-workload-optimizer/SKILL.md @@ -23,449 +23,122 @@ See LICENSE for license information. # Hyperloom Workload Optimizer -You are the catalog entry point for Hyperloom optimization on AMD Instinct GPUs. -Bootstrap the workspace, prepare the runtime environment, collect workload -parameters, then install, launch, and monitor the optimizer. This skill owns the -orchestration and the launcher gates; environment prep and workload intake are -delegated to the skills the Hyperloom wheel installs, and -`@${HYPERLOOM_SKILL_PATH}` (`inference_optimizer`) is the execution baseline. +You are the entry point for Hyperloom optimization on AMD Instinct GPUs. Your job +is the bootstrap: confirm the workspace, install the Hyperloom wheel, run +`/hyperloom-setup`, then hand the run to the skill that owns it. -Do not manually optimize inside chat unless debugging. +The wheel installs the skills that own everything after setup: `hyperloom-setup` +for credentials and run mode, the demo skills for a workload preset, and +`inference_optimizer` for the launcher gates and monitoring. They ship with the +runtime, so they always match the installed version. -## Prerequisites - -- AMD Instinct GPU host (MI300X / MI325X / MI355X) with ROCm -- `/dev/kfd` and `/dev/dri` present; `amd-smi` or `rocm-smi` works -- Python 3.10+ and network access to install the Hyperloom wheel -- Anthropic (or compatible) LLM credentials for agent backends -- A dedicated agent workspace directory - -Every command in this skill runs on that GPU host. Confirm the shell you are in -is on it before Phase 0, so a bootstrap does not land on a machine with no GPU. - -The Hyperloom **runtime** ships via `pip install` of the published wheel. - -## What Hyperloom runs - -The CLI starts a Python Coordinator that coordinates: - -- **Orchestration** — baseline, explore, specialist, integrate_patch, sweep -- **Kernel** — trace_analyze, run_optimization, integrate -- **Critic** — proposal review (default `--critic-agent`) -- **Robustness** — health monitoring and RCA (default `--robustness-agent`) - -State lives under a **session directory** per run; run-state root is -`$USER_DATA_PATH` (default `/workspace/hyperloom`), independent of the install -directory (`INSTALL_DIR`, where the wheel and `.env` live) and may point to -shared storage. Layout: `$USER_DATA_PATH/runtime/` (install.sh outputs, -`kernel-agent.env.sh`), `logs/`, and `//` per session -holding `manifest.json`, `state.json`, `runs/`, `reports/`, `optimizer_runs/`. - -## Workflow overview - -Match `hyperloom-custom-advanced` section order — do **not** ask workload -questions while writing `.env` or during `/hyperloom-setup`. - -- **Phase 0 Bootstrap** — `pip install`, `/hyperloom-setup` → `.env` (credentials + run mode only) -- **Phase 1 Environment** — custom-advanced §Setup Configuration (baremetal: confirm host; docker: start container + setup inside, contract in [setup.md](setup.md)) -- **Phase 2 Workload intake** — custom-advanced §Advanced Configuration → Model Resolution → show launch plan → user confirms -- **Phase 3 Execute** — install.sh → preflight → launch → monitor → report - -Load `hyperloom-custom-advanced` at Phase 1 and follow its sections in order -(discovery: `.cursor/` / `.claude/` / `.agents/skills/hyperloom-custom-advanced/SKILL.md`). -If it is not on disk, stop and tell the user to restart the agent so the newly -installed skills are picked up — do not improvise the environment or workload -sections from memory, since the wheel is the source of truth for both. -For deeper optimizer behavior read `@${HYPERLOOM_SKILL_PATH}` (`inference_optimizer`); -Iron Rules + CLI reference: [reference.md](reference.md). +This is the agent-facing form of the human quickstart in +[examples/README.md](https://github.com/AMD-AGI/Hyperloom/blob/main/examples/README.md). -## Iron Rules (launcher gates) +## Out of scope for this skill -Run order is always **IR-2 → IR-1 → launch**. Full text in [reference.md](reference.md). +- Do not run `python -m hyperloom.inference_optimizer.cli optimize` yourself. +- Do not implement a GPU preflight, launcher gate, or background launch here. The + installed skills own those, including the Iron Rules and the resume path. +- Do not ask for workload values (model, TP/EP, concurrency, ISL/OSL, precision, + objective, budget) while installing or while setup is running. They belong to + the run skill, after setup finishes. +- Do not optimize by hand in chat. -- **IR-1 — GPU unoccupied.** Before every `optimize` (fresh or `--resume`), every - visible GPU must have zero foreign serving PIDs (`sglang.launch_server` / - `vllm.entrypoints` / `Magpie`) and ≲ 500 MiB VRAM in use. -- **IR-2 — install.sh before launch.** Run `install.sh` and source - `kernel-agent.env.sh` in the **same shell** that spawns `optimize`. -- **Resume carve-out:** `--resume` may skip install only when `install.sh` exited - 0 earlier in the same shell, `kernel-agent.env.sh` is still sourced, and the - session's `manifest.json` exists. Any failure → re-run `install.sh`. - -## Phase discipline (do not skip) - -One phase at a time. Each phase asks only its own questions, waits for the -user's answers, completes its exit condition, then moves on. Never batch -questions from different phases into one prompt. In particular, never ask -workload questions (model, framework, TP/EP, precision, ISL/OSL, hours…) during -Phase 0 or Phase 1 — those belong to Phase 2 only. - -## Phase 0 — Bootstrap - -Skip completed steps (idempotent). Ask only about the install directory and -credentials/run mode here. Do not ask about the model or workload yet. - -### Confirm the install directory +## Prerequisites -The wheel installs into a target directory with `pip install --target `, -which also holds `.env` and runtime artifacts. Do not silently use the current -directory. Show the resolved current directory (`pwd`) and confirm it with the -user, or let them choose another dedicated path. Wait for the answer, then `cd` -into the chosen directory before installing. +- AMD Instinct GPU host (MI300X / MI325X / MI355X) with ROCm, `/dev/kfd` and + `/dev/dri` present, and `amd-smi` or `rocm-smi` working. +- Python 3.10+ and `pip` on the machine that runs the install. +- LLM credentials: Anthropic API access, or the AMD LLM gateway. +- A dedicated empty directory, opened in the agent as the workspace. -### Install the Hyperloom wheel +Confirm the shell you are in is on the GPU host before installing. Setup may later +point Docker at a different target host; until it does, everything here runs where +the agent is. -Skip when `hyperloom/` (wheel) or `src/hyperloom/` (source) already exists in the -confirmed directory. +## Step 1: Confirm the workspace -The runtime is published to PyPI as `hyperloom-inference-optimizer`. List the -releases, tell the user the newest one, and ask whether to install it or a -version they name. +The current directory is both the install target and the agent workspace. Confirm +with the user that it is a dedicated directory before installing: setup creates or +updates `.env` in it. Do not switch to another directory on your own, and do not +install into an existing project unless the user accepts the `.env` change. -List with `--pre` so prereleases are visible, and install an exact `==` version -so a later bootstrap installs the same runtime. +## Step 2: Install the Hyperloom wheel ```bash -cd "$INSTALL_DIR" # the directory confirmed above -pip index versions hyperloom-inference-optimizer --pre -pip install hyperloom-inference-optimizer== --target . -``` - -Confirm `hyperloom/inference_optimizer/assets/install.sh` exists. Restart the -agent if wheel skills are not visible. - -### Credentials and run mode - -Run `/hyperloom-setup` (installed to `.cursor/skills/hyperloom-setup/`). It -writes `.env`, sets `USER_DATA_PATH`, `HYPERLOOM_RUN_MODE`, and -`HYPERLOOM_SKILL_PATH`, and on bare metal runs `install_baremetal.sh`. - -**Phase 0 is done when all hold:** - -- `hyperloom/inference_optimizer/assets/install.sh` exists -- `.env` exists with non-placeholder LLM secrets -- `USER_DATA_PATH`, `HYPERLOOM_RUN_MODE`, and `HYPERLOOM_SKILL_PATH` are set - -More bootstrap detail: [setup.md](setup.md). - -## Phase 1 — Environment prep - -Load `hyperloom-custom-advanced` and follow its **Setup Configuration** -section only. - -**Baremetal (`HYPERLOOM_RUN_MODE=baremetal`):** confirm `install_baremetal.sh` -finished and the serving framework from setup is importable. Do not ask workload -questions yet. - -**Docker (`HYPERLOOM_RUN_MODE=docker`):** image choice, `docker run`, and the -in-container setup are owned entirely by custom-advanced Setup Configuration — -follow it, do not restate its commands or flags here. Do not ask workload -questions until the container is up and in-container setup succeeded, and never -run `optimize` on the host. - -Phase 1 is done when the target environment (host or container) is ready. - -## Phase 2 — Workload intake - -Enter only after Phase 0 and Phase 1 exit conditions hold. This is the first and -only phase that asks workload questions. - -Now follow custom-advanced **Advanced Configuration**, **Default Values**, and -**Model Resolution**. Use the agent's structured question UI when available. -Never copy API keys into chat output. - -| Field | CLI flag | Default | Notes | -|---|---|---|---| -| Model path | `--model` | required | Local dir with `config.json`, or HF cache | -| Framework | `--framework` | `sglang` | or `vllm`; prefer `.env` `FRAMEWORK` when set | -| TP / EP | `--tp` / `--ep` | `1` / `1` | tensor / expert parallel | -| CONC | `--conc` | `64` | client concurrency | -| ISL / OSL | `--isl` / `--osl` | `1024` / `1024` | input / output seq lengths | -| PRECISION | `--precision` | `bf16` | match checkpoint; `fp8` for FP8 models | -| MAX_HOURS | `--max-hours` | CLI `2.0` | offer `3` (quick) or `12` (full); see below | -| TARGET_GAIN | `--target-gain` | `30` | desired % gain | - -**Optional:** `--no-framework-agent`, `--no-kernel`, `--no-enable-conc-sweep`, -`--no-enable-roofline`, `--gpu-type`, `--server-args`, `--compare-against-gpu`, -`--quantize` prelude. - -Infer `PRECISION` from the model name when obvious (e.g. an `FP8` model implies -`--precision fp8`) and confirm it — do not silently keep the `bf16` default. - -### Budget and flags — offer these three - -Offer all three and let the user pick one. The flags in each are a set: pass them -together, and do not ask for a budget and then ask separately which phases to run. -The two demos take the workload and flags of the Hyperloom demo skill of the same -budget — treat those as given and skip the table above. The user may name their -own model instead of the demo's; for the 3-hour demo keep it at 8B or below. -Confirm everything in the launch plan. Only **Custom** collects workload answers. - -**1. 3-hour demo** (`hyperloom-qwen3-8b-3h`) — `Qwen/Qwen3-8B` unless the user -names another 8B-or-smaller model, TP=1, CONC=64, ISL=OSL=1024, -`--precision bf16`, serving and config parameters only, no kernel rewrites. -Resolve the model per custom-advanced Model Resolution; download it from Hugging -Face when it is not already local. Match `--precision` to the chosen checkpoint. -Expect a modest validated gain, or an honest 0% when the workload has no -parameter headroom. - -```text ---max-hours 3 --precision bf16 --target-gain 30 ---max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01 ---no-kernel --no-enable-conc-sweep --no-enable-roofline +pip install hyperloom-inference-optimizer --target . ``` -`0.50` is the share *before* redistribution. With `--no-kernel`, KERNEL_AGENT is -disabled and its freed share is added on top, so `0.50` becomes ~0.99 of the wall -clock for OPTIMIZE. Raising `0.50` buys almost nothing — the post-redistribution -share is capped at one full wall clock and the excess is discarded. +Install the current release unless the user asks for a specific version. It is +normal for the directory to hold many Python package folders afterwards; the user +does not need to inspect them. -Do **not** pass `--no-framework-agent` here: it skips OPTIMIZE entirely, which is -the one phase this profile relies on. `--no-kernel` is what makes it "no kernel -rewrites". +## Step 3: Run `/hyperloom-setup` -**2. 12-hour demo** (`hyperloom-qwen3-14b-fp8-12h`) — `Qwen/Qwen3-14B-FP8` -unless the user names another model, TP=1, CONC=64, ISL=OSL=1024, -`--precision fp8` matched to the chosen checkpoint, every lever with kernel -rewrites included. The kernel agent needs room to profile, rewrite and -revalidate, which is where the larger gains come from. +The wheel installs `hyperloom-setup` into the agent's skill directories +(`.agents/skills/`, `.claude/skills/`, `.cursor/skills/`). Run it: ```text ---max-hours 12 --precision fp8 --target-gain 50 ---max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42 -``` - -Do not add `--max-minutes-explore-pct`: configuration search and source landing -are two arms of one phase with one budget, so that spelling is an alias onto -`--max-minutes-framework-pct` and silently overwrites it. - -**3. Custom** — the user brings their own model or workload instead of taking a -demo. Walk through the fields in the table above and the phase toggles, one -question at a time, and derive the flags from the answers rather than asking for -flags. Whichever levers they pick, a budget of 3 hours or less keeps the 3-hour -demo's flag set. -Optional flags come from the list above; show the full flag list in the launch -plan either way. - -### Confirmation gate (required before Phase 3) - -The Coordinator has no in-loop `setup` / `classify` — a value not asked here is -silently lost to its default. Before running any Phase 3 command, present the -full launch plan (including defaulted fields) and get explicit user confirmation. - -Print the plan in the reply body as this aligned block: - -```text -Launch plan — please confirm: - MODEL_PATH /wekafs/models/Qwen3-14B-FP8 - FRAMEWORK vllm - TP=1 EP=1 CONC=64 - ISL=1024 OSL=1024 - PRECISION=fp8 - MAX_HOURS=3 TARGET_GAIN=30% - profile 3-hour demo — no kernel rewrites, no conc sweep, no roofline - flags --max-minutes-framework-pct 0.50 - --max-minutes-sweep-pct 0.01 - --no-kernel --no-enable-conc-sweep --no-enable-roofline - RUN_MODE baremetal -``` - -Never put the plan inside the confirmation prompt itself. A prompt renders as one -wrapped paragraph, which collapses the alignment above into an unreadable blob the -user has to search for `MAX_HOURS` in. Keep the prompt to a single short question -such as `Approve this launch plan?`, and if you offer a "change something" option, -name the field to change rather than making the user retype it as free text. - -Do **not** run `install.sh` or launch `optimize` until the user approves this plan. - -### Persist the plan (required — shells do not share exports) - -Agent shells do not persist exports between calls, so write the confirmed values -to `$RUN_DIR/workload.env` right after approval. Every Phase 3 block sources it; -without this, launch silently falls back to `${TP:-1}` / `${CONC:-64}` defaults -and `--model ""`. Fill each value from the approved plan. - -```bash -export USER_DATA_PATH="${USER_DATA_PATH:?run /hyperloom-setup first}" -export RUN_DIR="${USER_DATA_PATH}/optimizer_runs" -mkdir -p "$RUN_DIR" -# Quoted heredoc (<<'EOF'): values are written literally, so a MODEL_PATH with -# spaces, $, or $(...) is not expanded or executed. Edit each value to the plan. -cat > "$RUN_DIR/workload.env" <<'EOF' -export MODEL_PATH=/wekafs/models/Qwen3-14B-FP8 -export FRAMEWORK=vllm -export TP=1 -export EP=1 -export CONC=64 -export ISL=1024 -export OSL=1024 -export PRECISION=fp8 -export MAX_HOURS=3 -export TARGET_GAIN=30 -# The whole flag set for the approved profile, space-separated. The 3-hour -# demo is shown; a 12-hour run swaps in its own set. -export OPT_FLAGS="--max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01 --no-kernel --no-enable-conc-sweep --no-enable-roofline" -EOF -``` - -## Phase 3 — Install (IR-2) - -`INSTALL_DIR` is the directory confirmed in Phase 0, the one holding `hyperloom/` -and `.env`. Every Phase 3 block below rebuilds it from the current directory, so -run them from there; the check refuses a directory that is not it. - -Resolve paths for wheel or source layout: - -```bash -export INSTALL_DIR="$(pwd -P)" -[ -d "${INSTALL_DIR}/hyperloom" ] || [ -d "${INSTALL_DIR}/src/hyperloom" ] || { - echo "ERROR: ${INSTALL_DIR} holds no hyperloom/ -- cd to the Phase 0 install directory" >&2; exit 1; } -set -a; . "${INSTALL_DIR}/.env"; set +a -export USER_DATA_PATH="${USER_DATA_PATH:?USER_DATA_PATH missing}" -. "${USER_DATA_PATH}/optimizer_runs/workload.env" # confirmed Phase 2 values -export PYTHONPATH="${INSTALL_DIR}:${PYTHONPATH:-}" -ulimit -Sn 65536 || true - -INSTALL_SH="${INSTALL_DIR}/hyperloom/inference_optimizer/assets/install.sh" -[ -f "$INSTALL_SH" ] || INSTALL_SH="${INSTALL_DIR}/src/hyperloom/inference_optimizer/assets/install.sh" - -bash "$INSTALL_SH" -. "${KERNEL_AGENT_ENV:-${USER_DATA_PATH}/runtime/kernel-agent.env.sh}" -export PYTHONPATH="${INSTALL_DIR}:${PYTHONPATH:-}" -``` - -In Docker mode, run this inside the container. - -## Phase 3 — Preflight (IR-1) - -`install.sh` exports `$PYTHON`; the fallback below covers agent sandboxes that do -not persist exports between shell calls. - -```bash -export SKILL_DIR="${SKILL_DIR:?absolute path of the directory holding this SKILL.md}" -. "${USER_DATA_PATH}/optimizer_runs/workload.env" # confirmed Phase 2 values -export PYTHON="${PYTHON:-$(command -v python3)}" -"$PYTHON" "${SKILL_DIR}/scripts/preflight.py" -``` - -The gate exits non-zero — do not launch — when `MODEL_PATH` is missing or has no -`config.json`, torch sees no GPU, a foreign serving process still holds a card, -or any GPU holds more than `IR1_VRAM_LIMIT_MIB` (default 500) MiB. - -It also blocks when VRAM cannot be read at all: no `amd-smi`/`rocm-smi` on -`PATH`, a probe that exits non-zero, or output it cannot parse. An unreadable -probe cannot rule out a busy GPU, and a foreign process holding VRAM under a -different name would slip through. Confirm the GPUs are idle by hand before -re-running with `IR1_ALLOW_UNVERIFIED_VRAM=1`. - -Never print API keys or tokens. `scripts/tests/test_preflight.py` covers the -probe shapes this gate must reject. - -## Phase 3 — Launch - -After IR-2 and IR-1 pass, launch. `setsid nohup` is required for runs longer than -5 minutes, so the run outlives the agent shell. - -```bash -export INSTALL_DIR="$(pwd -P)" -export SKILL_DIR="${SKILL_DIR:?absolute path of the directory holding this SKILL.md}" -bash "${SKILL_DIR}/scripts/launch.sh" -``` - -Every workload value comes from the confirmed `workload.env`; the script has no -`${VAR:-default}` fallbacks, so a missing value fails loudly instead of launching -a different config. Put any optional Phase 2 flags (`--no-kernel`, -`--no-framework-agent`, `--gpu-type`, `--model-class`, `--server-args`, -`--compare-against-gpu`, `--quantize`, phase budget flags) into `OPT_FLAGS` in -`workload.env`. `OPT_FLAGS` is word-split, so quote any flag value that contains -spaces, e.g. -`export OPT_FLAGS='--server-args "--foo bar"'`. - -### Launch health check (30 s after start) - -Required after every launch and resume. The PID recorded at launch is the -**setsid wrapper**, which exits immediately — it is NOT the optimizer. This reads -the real `.pid` and `.session_dir` from the launch-info JSON, rewrites the PID -file so the monitor watches the right process, and records both in -`$RUN_DIR/last_launch.env` for the later phases. - -```bash -export INSTALL_DIR="$(pwd -P)" -export SKILL_DIR="${SKILL_DIR:?absolute path of the directory holding this SKILL.md}" -bash "${SKILL_DIR}/scripts/launch_health.sh" -``` - -It exits non-zero when the launch-info JSON never appeared, no optimizer process -can be found, or `session_dir` is still unset — inspect the reported run log in -those cases. Never guess `session_dir` from a timestamp; concurrent sessions -share `USER_DATA_PATH`. - -## Phase 3 — Monitor - -Poll at most every 5 minutes unless debugging a startup failure. Use the state -reader the wheel ships rather than parsing `state.json` by hand — it also prints -the recent lifecycle events. - -```bash -export INSTALL_DIR="$(pwd -P)" -. "${USER_DATA_PATH}/optimizer_runs/last_launch.env" # SESSION_DIR from launch -STATE_TOOL="${INSTALL_DIR}/hyperloom/inference_optimizer/tools/read_optimizer_state.py" -[ -f "$STATE_TOOL" ] || STATE_TOOL="${INSTALL_DIR}/src/hyperloom/inference_optimizer/tools/read_optimizer_state.py" -"${PYTHON:-python3}" "$STATE_TOOL" "$SESSION_DIR" -``` - -For recent action counts grouped by category, the wheel also ships -`tools/event_counts.py`, invoked the same way. - -Report session id + log path, `baseline_tput` / `current_best` / -`cumulative_gain`, explore accepted/rejected, last kernel opt (correctness, -speedup, KEEP/REVERT), and process-alive vs `stop_reason`. See -[reference.md](reference.md) Report fields. - -## Resume - -Resume runs in a fresh shell. Re-run the IR-2 and IR-1 gates first, exactly as for -a fresh launch — the script does not re-check them. - -```bash -export INSTALL_DIR="$(pwd -P)" -export SKILL_DIR="${SKILL_DIR:?absolute path of the directory holding this SKILL.md}" -bash "${SKILL_DIR}/scripts/resume.sh" -bash "${SKILL_DIR}/scripts/launch_health.sh" +/hyperloom-setup ``` -It resumes the session recorded in `last_launch.env` and always passes -`--resume-from` explicitly, because a bare `--resume` auto-picks the newest -session and can target the wrong run. Resume writes its own log -(`run_resume-*.log`) so the original run log is preserved. Reuse the IR-2 -carve-out rules; re-run `install.sh` if the shell or env changed. - -| `stop_reason` | Action | -|---|---| -| `time_exhausted` | `--resume` same session | -| `no_more_leverage` | stop; resume only if user changes strategy | -| `policy_loop` | inspect `policy_denial_history`; clear stale prunes | - -## Expected optimizer flow - -1. Establish `baseline_tput`. -2. Coordinator runs roofline/profile analysis after baseline. -3. `explore` tests serving parameters incrementally. -4. Kernel-agent runs on hot paths with compile + correctness evidence. -5. `sweep` validates concurrency around the best candidate. -6. Final report under `$SESSION_DIR/reports/`. - -## When to defer - -- **Plain serving only** — use `serving-llms-on-instinct`. -- **ROCm driver broken** — diagnose the ROCm stack first (e.g. a `rocm-doctor` - skill if published); do not start the optimizer on a broken driver. -- **Edge cases** — read `@${HYPERLOOM_SKILL_PATH}` for multi-node, atom - framework (IR-8), critic/robustness backends, cache topology, and the - full failure matrix. - -## Further reading - -- Bootstrap detail: [setup.md](setup.md) -- Iron Rules + CLI reference: [reference.md](reference.md) -- Authoritative runtime skill: `hyperloom/inference_optimizer/SKILL.md` +It is interactive and owns credentials, `USER_DATA_PATH`, the run mode +(`docker` recommended, or `baremetal`), the Docker target host, and the bare-metal +framework install. It writes `.env` and stops before any optimization. Run it once +per workspace; the run skills reuse those values. + +Let setup ask its own questions. Do not preempt them, do not restate its option +lists, and never ask the user to paste an API key into chat. + +If the agent does not list `hyperloom-setup` after the install, the skill +directories were written after the agent scanned them. Tell the user to restart +the agent, then run it again. Do not substitute your own setup steps. + +## Step 4: Hand off to a run skill + +Setup ends by offering a run and loading the matching skill, so normally you just +follow it. When the user asks for a run directly, load the skill by name and follow +its instructions instead of this one: + +- `hyperloom-qwen3-8b-3h` — short no-kernel Qwen3-8B run; best first end-to-end check. +- `hyperloom-qwen3-14b-fp8-12h` — medium-length Qwen3-14B-FP8 run. +- `hyperloom-qwen3-14b-fp8-12h-forge` — the same run on the KernelForge kernel backend. +- `hyperloom-custom-advanced` — explicit model, framework, workload, budget, and phase toggles. + +A preset keeps its workload even if the user supplies their own `MODEL_PATH`; +tensor parallelism, concurrency, sequence lengths, precision, and budget are not +retuned for that model. When those need to change, use `hyperloom-custom-advanced`. + +To resume a stopped session, follow the optimizer skill at `.env` +`HYPERLOOM_SKILL_PATH`; it owns the resume path and the gates a relaunch still has +to clear. + +## What to expect during a run + +Optimization runs for hours in the background. Do not stream the log. + +Before launch the run skill shows a plan: resolved model path, run mode, framework, +TP, concurrency, ISL/OSL, precision, budget, and `USER_DATA_PATH`. Get the user's +go-ahead on that plan before the optimizer starts — it then owns the GPU for hours. +After launch it reports the optimizer PID, run log, launch-info JSON, session +directory, `state.json`, and the first health check. + +During the run, report a short status about every 300 seconds: process alive, +current phase, `stop_reason`, baseline and current best throughput, cumulative +gain, the latest benchmark or candidate decision, and the most relevant log lines. +Never print API keys, tokens, or custom headers. + +## Troubleshooting + +- Many package folders in the workspace after `pip install --target .` is expected. +- `/hyperloom-setup` not listed: the install landed after the agent scanned for + skills. Restart the agent and check `.claude/skills/hyperloom-setup/` exists. +- `ImportError: libamdhip64.so.7` or `libhipblas.so.3`: the framework torch wheel + wants different ROCm user-space libraries; align `ROCM_PATH` and + `LD_LIBRARY_PATH`. +- `hipDeviceAttributePciChipId` missing during an AITER build: `hipcc` is using + older ROCm headers; put the matching ROCm `bin` first on `PATH`. +- Anything past setup (preflight failures, launch, phases, gains) belongs to the + installed run and optimizer skills. Read those rather than reproducing their + checks here. diff --git a/skills/hyperloom-workload-optimizer/evals/evals.json b/skills/hyperloom-workload-optimizer/evals/evals.json index d823b49..00dd102 100644 --- a/skills/hyperloom-workload-optimizer/evals/evals.json +++ b/skills/hyperloom-workload-optimizer/evals/evals.json @@ -6,36 +6,18 @@ "prompt": "I want to optimize vLLM inference throughput on an MI300X. What are the first steps before launching Hyperloom?", "expected_behavior": [ "Mention installing the Hyperloom wheel or checking for the hyperloom package", - "Mention workspace bootstrap such as hyperloom-setup or setup.md" + "Mention workspace bootstrap by running the hyperloom-setup skill" ], "unexpected_behavior": [ "Start a plain vLLM docker serve as the primary answer without the optimization loop", "Launch the optimizer immediately after setup without collecting TP, concurrency, ISL, OSL, and precision or confirming a launch plan" ] }, - { - "id": "hyperloom-launcher-gates", - "skill_should_trigger": true, - "note": "The two Iron Rules are the whole reason a launch is safe, so they are graded as prose and pinned to the literal script names. The prompt forbids running anything, so the gates are graded as described, not as observed on a real launch.", - "prompt": "What are Hyperloom's two launcher gates, in the order they run, and what does each one check? Answer in three or four sentences. Do not run anything.", - "expected_behavior": [ - "Describe IR-2: install.sh must have been run and kernel-agent.env.sh sourced in the same shell that spawns the optimizer", - "Mention a GPU preflight check for stale serving processes or VRAM in use (IR-1)" - ], - "logs_contain": [ - "install.sh", - "kernel-agent.env.sh" - ] - }, { "id": "hyperloom-workload-intake", "skill_should_trigger": true, - "note": "Asks about each graded step, otherwise the answer's budget goes to listing workload values and the later steps drop out at random. Whether the answer names the values is deliberately not graded: a judge reads \"the values it needs\" as a request the prompt forbids and splits both ways on it, and logs_contain cannot stand in because those names are all over SKILL.md, which the run reads. Collecting them before a launch is graded by hyperloom-optimize-vllm-first-steps.", - "prompt": "The environment is already set up. Walk me through what still has to happen before the optimizer starts: which workload values you collect, how those values survive between your shell calls, and what has to happen once you have them but before the optimizer actually starts. Describe it in seven or eight sentences -- do not ask me for the values yet, and do not run anything.", - "expected_behavior": [ - "Say it will present a launch plan and get user confirmation before launching the optimizer", - "Explain that confirmed workload values are persisted (e.g. to a workload.env file) and sourced at launch, since agent shells do not keep exports between calls" - ] + "note": "Trigger only. Every judged expectation tried here has graded both ways over byte-identical inputs, because the prompt forbids running anything and the judge then reads a described launch plan as a plan never presented. That the plan is confirmed before a launch is graded where a run actually reaches it, by hyperloom-optimize-vllm-first-steps and hyperloom-bootstrap-phase-discipline.", + "prompt": "The environment is already set up. Walk me through what still has to happen before the optimizer starts: which workload values get collected, and what has to happen once they are known but before the optimizer actually starts. Describe it in five or six sentences -- do not ask me for the values yet, and do not run anything." }, { "id": "hyperloom-bootstrap-phase-discipline", diff --git a/skills/hyperloom-workload-optimizer/reference.md b/skills/hyperloom-workload-optimizer/reference.md deleted file mode 100644 index 4519a88..0000000 --- a/skills/hyperloom-workload-optimizer/reference.md +++ /dev/null @@ -1,142 +0,0 @@ -# Hyperloom Workload Optimizer Reference - -Iron Rules, CLI flags, and launcher contracts for -[SKILL.md](SKILL.md). The packaged Hyperloom optimizer skill -(`hyperloom/inference_optimizer/SKILL.md` after wheel install) is the -authoritative source for edge cases. - -## Table of contents - -1. [Iron Rules](#iron-rules) -2. [CLI workload flags](#cli-workload-flags) -3. [Critic and robustness backends](#critic-and-robustness-backends) -4. [Framework selection](#framework-selection) -5. [Failure signals](#failure-signals) -6. [Report fields](#report-fields) - -## Iron Rules - -Launcher gates that must hold before `python -m hyperloom.inference_optimizer.cli optimize`. - -### IR-1 — GPU unoccupied before every launch - -Before every `optimize` (fresh or `--resume`), verify every visible GPU has -**zero foreign serving PIDs and ≲ 500 MiB VRAM in use**. Leftover -`sglang.launch_server` / `vllm.entrypoints` / `Magpie` processes silently -degrade the next baseline. - -### IR-2 — install.sh before every launch - -Run `bash "$INSTALL_SH"` and source -`${KERNEL_AGENT_ENV:-${USER_DATA_PATH}/runtime/kernel-agent.env.sh}` in the -**same shell** that spawns `optimize`. Skipping install fails after baseline: -missing TraceLens/GEAK, hung Ray tasks, or `401` on kernel-opt gateway calls. - -**Resume carve-out:** `--resume` may skip install only when all hold: - -1. `install.sh` exited 0 earlier in the *same shell* -2. `kernel-agent.env.sh` is still sourced -3. The resumed session's `manifest.json` exists - -Any failure → treat as a fresh launch and re-run `install.sh`. - -### IR-3 — KB + PR Monitor (soft degrade) - -`_preflight()` runs `preflight_kb.sh`. Exit `1` auto-enables `--degraded-kb` / -`--degraded-pr`; launch continues. IR-3 never aborts. - -### IR-4 — OPTIMIZE phase contracts (Coordinator-internal) - -Configuration search and source landing are two arms of **one** phase -(OPTIMIZE / FRAMEWORK_AGENT) sharing one budget. The separate EXPLORE phase and -its `--explore-force-exit-*` flags no longer exist. - -- **IR-4:** OPTIMIZE is specialist-informed; GPU specialists lease cards via - `gpu_research_lane` and must not touch production serving on port 8888. - Specialists author patches into an isolated worktree; `integrate_patch` does - the `git apply` plus throughput/accuracy gate after Critic review. -- Plateau signals are advisory; the per-phase budgets are the hard gates. - -### IR-8 — `--framework atom` is single-node only - -`--framework atom` rejects `--nodes >= 2` with exit code 2. - -## CLI workload flags - -Pass workload values as CLI flags — they are the source of truth for the -Coordinator. - -| Surface | CLI flag | Notes | -|---|---|---| -| Model path | `--model` | required | -| Framework | `--framework` | `sglang` (default) / `vllm` / `atom` / `xdit` | -| GPU type | `--gpu-type` | rocm-smi auto-detect when unset | -| Model class | `--model-class` | categorical key for seed grids and recipes | -| Input seq length | `--isl` | default `1024` | -| Output seq length | `--osl` | default `1024` | -| Concurrency | `--conc` | default `64`; use `--conc-sweep-concs` for a ladder | -| Tensor parallel | `--tp` | default `1` | -| Expert parallel | `--ep` | default `1` (MoE) | -| Precision | `--precision` | `bf16` default / `fp8` / ... | -| Budget | `--max-hours` | CLI parser default `2.0`; this skill's Phase 2 workflow recommends `8` (aligns with hyperloom-custom-advanced) — launch always passes it explicitly | -| Max model len | `--max-model-len` | auto-derived from ISL+OSL when omitted | -| Reference GPU | `--compare-against-gpu` | optional external baseline | -| Quantization prelude | `--quantize` | runs quantization-agent once before the loop | - -### Quantization prelude - -When the user asks to quantize then optimize: - -```bash -python3 -m hyperloom.inference_optimizer.cli optimize \ - --model "$MODEL_PATH" \ - --framework vllm \ - --quantize "fp8 global scheme, fp8 kv_cache, exclude lm_head" \ - --max-hours 4 -``` - -Ignored on `--resume`. - -## Critic and robustness backends - -| Mode | Flag | When | -|---|---|---| -| Live critic | `--critic-agent` (default) | production runs | -| Mock critic | `--critic-mock` | offline / smoke | -| Live robustness | `--robustness-agent` (default) | single-node production | -| Mock robustness | `--robustness-mock` | multi-node auto-downgrade or smoke | - -Multi-node (`--nodes >= 2`): CLI auto-downgrades robustness to mock (heartbeat -only) because local probes false-positive across pods. - -## Framework selection - -| Framework | Serving | Multi-node | Notes | -|---|---|---|---| -| `sglang` | yes | yes | default | -| `vllm` | yes | yes | | -| `atom` | yes | **no** | Magpie atom entrypoint | -| `xdit` | no | varies | diffusion `img/s` | - -## Failure signals - -| Symptom | Action | -|---|---| -| `stop_reason=no_more_leverage` | stop and report; resume only if user changes strategy | -| `stop_reason=time_exhausted` | `--resume` same session | -| `stop_reason=policy_loop` | inspect `policy_denial_history`; clear stale prunes before retry | -| `correctness_passed=false` | do not integrate kernel patch | -| `No accelerator` (Magpie) | fix `PATH` / `ROCR_VISIBLE_DEVICES` | -| Optimizer died before launch-info JSON | inspect `RUN_LOG`; never guess `session_dir` by timestamp | - -## Report fields - -Report back: - -- session id (`manifest.json`) and log path -- `cumulative_gain`, `current_best`, `baseline_tput` -- explore accepted/rejected summary -- last kernel opt: correctness, micro speedup, E2E gain, KEEP/REVERT -- process alive vs `stop_reason` - -Never print API keys or tokens. diff --git a/skills/hyperloom-workload-optimizer/scripts/_env.sh b/skills/hyperloom-workload-optimizer/scripts/_env.sh deleted file mode 100644 index 9846e1f..0000000 --- a/skills/hyperloom-workload-optimizer/scripts/_env.sh +++ /dev/null @@ -1,60 +0,0 @@ -# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. -# -# See LICENSE for license information. - -# Shared Phase 3 preamble, sourced by launch.sh, launch_health.sh and resume.sh. -# Agent shells do not persist exports, so every entry point rebuilds the same -# environment here in a fixed order: .env, then workload.env, then the -# kernel-agent env written by install.sh. -# -# Sourced, not executed: a failure here exits the calling script. - -: "${INSTALL_DIR:="$(pwd -P)"}" -cd "$INSTALL_DIR" || exit 1 - -# Defaulting to the current directory is only right when the caller is already -# in the install directory, so say which directory was wrong rather than failing -# later on a path built from it. -if [ ! -d "$INSTALL_DIR/hyperloom" ] && [ ! -d "$INSTALL_DIR/src/hyperloom" ]; then - echo "ERROR: $INSTALL_DIR holds no hyperloom/ or src/hyperloom/ -- set INSTALL_DIR to the directory the wheel was installed into" >&2 - exit 1 -fi - -if [ -f "$INSTALL_DIR/.env" ]; then - # Values containing spaces must be double-quoted in .env, otherwise this - # source fails with exit 127. hyperloom-setup writes them quoted. - set -a - . "$INSTALL_DIR/.env" - set +a -fi - -: "${USER_DATA_PATH:?USER_DATA_PATH missing -- run the Hyperloom setup skill first}" -export INSTALL_DIR USER_DATA_PATH -export RUN_DIR="${USER_DATA_PATH}/optimizer_runs" -mkdir -p "$RUN_DIR" - -WORKLOAD_ENV="${RUN_DIR}/workload.env" -if [ ! -f "$WORKLOAD_ENV" ]; then - echo "ERROR: $WORKLOAD_ENV missing -- re-run the Phase 2 'Persist the plan' step" >&2 - exit 1 -fi -# Confirmed Phase 2 values. Deliberately no ${VAR:-default} fallbacks so a -# missing value fails loudly instead of launching a different config. -. "$WORKLOAD_ENV" -: "${MODEL_PATH:?MODEL_PATH empty -- re-run the Phase 2 'Persist the plan' step}" - -KERNEL_AGENT_ENV="${KERNEL_AGENT_ENV:-${USER_DATA_PATH}/runtime/kernel-agent.env.sh}" -if [ ! -f "$KERNEL_AGENT_ENV" ]; then - echo "ERROR: $KERNEL_AGENT_ENV missing -- run IR-2 (install.sh) first" >&2 - exit 1 -fi -export KERNEL_AGENT_ENV -. "$KERNEL_AGENT_ENV" - -export PYTHON="${PYTHON:-$(command -v python3)}" -export PATH="$(dirname "$PYTHON"):/usr/local/bin:$PATH" -export PYTHONPATH="${INSTALL_DIR}:${PYTHONPATH:-}" - -# Run handles are recorded here because the health check, monitor and resume -# steps each run in a fresh shell. -LAST_LAUNCH_ENV="${RUN_DIR}/last_launch.env" diff --git a/skills/hyperloom-workload-optimizer/scripts/launch.sh b/skills/hyperloom-workload-optimizer/scripts/launch.sh deleted file mode 100644 index 0c7f8d5..0000000 --- a/skills/hyperloom-workload-optimizer/scripts/launch.sh +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env bash -# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. -# -# See LICENSE for license information. -# -# Start a fresh optimize run in the background. Run IR-2 (install.sh) and IR-1 -# (preflight.py) first; this script does not re-check the gates. -# -# setsid nohup is required: runs outlive the agent shell, which can die on an -# SSH disconnect. Every workload value comes from workload.env. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" -# shellcheck source=./_env.sh -. "${SCRIPT_DIR}/_env.sh" - -RUN_TAG="$(basename "$MODEL_PATH")-$(date +%Y%m%d_%H%M%S)" -RUN_LOG="${RUN_DIR}/run_${RUN_TAG}.log" -PID_FILE="${RUN_DIR}/run_${RUN_TAG}.pid" -LAUNCH_INFO_FILE="${RUN_DIR}/launch_${RUN_TAG}.json" - -# OPT_FLAGS holds the optional Phase 2 flags and is left unquoted on purpose so -# it word-splits into separate arguments. -# shellcheck disable=SC2086 -setsid nohup "$PYTHON" -m hyperloom.inference_optimizer.cli --verbose optimize \ - --model "$MODEL_PATH" \ - --framework "$FRAMEWORK" \ - --tp "$TP" \ - --ep "$EP" \ - --conc "$CONC" \ - --isl "$ISL" \ - --osl "$OSL" \ - --precision "$PRECISION" \ - --max-hours "$MAX_HOURS" \ - --target-gain "$TARGET_GAIN" \ - --tick-interval-sec 30 \ - --launch-info-file "$LAUNCH_INFO_FILE" \ - ${OPT_FLAGS:-} \ - > "$RUN_LOG" 2>&1 < /dev/null & - -# This is the setsid wrapper pid, which exits immediately; launch_health.sh -# replaces it with the real optimizer pid from the launch-info JSON. -echo $! > "$PID_FILE" - -cat > "$LAST_LAUNCH_ENV" <&2 - exit 1 -fi -. "$LAST_LAUNCH_ENV" - -sleep "${LAUNCH_HEALTH_DELAY_SEC:-30}" - -if [ ! -f "$LAUNCH_INFO_FILE" ]; then - echo "ERROR: $LAUNCH_INFO_FILE not written; inspect $RUN_LOG" >&2 - exit 1 -fi - -read_json() { - "$PYTHON" -c 'import json,sys;print(json.load(open(sys.argv[1])).get(sys.argv[2],""))' \ - "$1" "$2" 2>/dev/null || true -} - -REAL_PID="$(read_json "$LAUNCH_INFO_FILE" pid)" -if [ -z "$REAL_PID" ]; then - REAL_PID="$(pgrep -f 'hyperloom.inference_optimizer.cli .*optimize' | head -1 || true)" -fi -if [ -z "$REAL_PID" ]; then - echo "ERROR: optimizer pid not found; inspect $RUN_LOG" >&2 - exit 1 -fi -echo "$REAL_PID" > "$PID_FILE" - -if [ -d "/proc/${REAL_PID}" ]; then - echo "optimizer_alive=true pid=${REAL_PID}" -else - echo "ERROR: pid ${REAL_PID} is already gone; inspect $RUN_LOG" >&2 - exit 1 -fi - -SESSION_DIR="$(read_json "$LAUNCH_INFO_FILE" session_dir)" -if [ -z "$SESSION_DIR" ]; then - echo "ERROR: no session_dir yet in $LAUNCH_INFO_FILE; inspect $RUN_LOG" >&2 - exit 1 -fi - -cat > "$LAST_LAUNCH_ENV" < None: - print(f"IR-1 BLOCK: {message}", file=sys.stderr) - - -def _warn(message: str) -> None: - print(f"IR-1 WARNING: {message}", file=sys.stderr) - - -def check_model_path() -> bool: - path = os.environ.get("MODEL_PATH", "").strip() - if not path: - _fail("MODEL_PATH is empty; re-run the Phase 2 'Persist the plan' step") - return False - if not os.path.isdir(path): - _fail(f"MODEL_PATH is not a directory: {path}") - return False - if not os.path.isfile(os.path.join(path, "config.json")): - _fail(f"MODEL_PATH has no config.json: {path}") - return False - print(f"model_path_ok={path}") - return True - - -def check_torch_gpus() -> bool: - try: - import torch - except Exception as exc: - _fail(f"torch is not importable ({type(exc).__name__}); run install.sh first") - return False - try: - available = torch.cuda.is_available() - count = torch.cuda.device_count() if available else 0 - except Exception as exc: - _fail(f"torch GPU probe raised {type(exc).__name__}: {str(exc)[:200]}") - return False - print(f"torch_cuda_available={available} torch_cuda_device_count={count}") - if not available or count == 0: - _fail("torch sees no GPU; check ROCm, /dev/kfd and /dev/dri") - return False - return True - - -def check_foreign_processes() -> bool: - # Report the matched pattern and pid only; a serving cmdline can carry tokens. - found = [] - for pid in filter(str.isdigit, os.listdir("/proc")): - try: - with open(f"/proc/{pid}/cmdline", "rb") as handle: - raw = handle.read() - except OSError: - continue - text = raw.replace(b"\0", b" ").decode("utf-8", "ignore") - if not text: - continue - for pattern in FOREIGN_SERVING_PATTERNS: - if pattern in text: - found.append((pid, pattern)) - break - for pid, pattern in found: - print(f"foreign_serving_process pid={pid} matched={pattern}") - if found: - _fail(f"{len(found)} foreign serving process(es) still hold the GPUs") - return False - print("foreign_serving_processes=0") - return True - - -def _to_mib(value: object, unit: object) -> float: - if isinstance(value, bool) or not isinstance(value, (int, float, str)): - raise VramUnreadable(f"unsupported VRAM value type: {type(value).__name__}") - try: - number = float(value) - except (TypeError, ValueError): - raise VramUnreadable(f"VRAM value is not numeric: {value!r}") from None - key = str(unit or "mib").strip().lower() - if key not in UNIT_TO_MIB: - raise VramUnreadable(f"unknown VRAM unit: {unit!r}") - return number * UNIT_TO_MIB[key] - - -def _extract_used_vram(entry: object) -> float: - if not isinstance(entry, dict): - raise VramUnreadable(f"GPU entry is {type(entry).__name__}, expected an object") - containers = [entry] - for key in MEM_CONTAINER_KEYS: - nested = entry.get(key) - if isinstance(nested, dict): - containers.append(nested) - for container in containers: - for key in USED_VRAM_KEYS: - if key not in container: - continue - reading = container[key] - if isinstance(reading, dict): - if "value" not in reading: - raise VramUnreadable(f"{key} object has no 'value' field") - return _to_mib(reading.get("value"), reading.get("unit")) - return _to_mib(reading, "mib") - raise VramUnreadable(f"no used-VRAM field found; keys={sorted(entry)[:8]}") - - -def _run_probe(argv: list[str]) -> str: - try: - completed = subprocess.run( - argv, capture_output=True, text=True, timeout=PROBE_TIMEOUT_SEC - ) - except subprocess.TimeoutExpired: - raise VramUnreadable(f"{argv[0]} timed out after {PROBE_TIMEOUT_SEC}s") from None - except OSError as exc: - raise VramUnreadable(f"{argv[0]} could not be executed: {exc}") from None - if completed.returncode != 0: - detail = (completed.stderr or completed.stdout or "").strip()[:200] - raise VramUnreadable(f"{argv[0]} exited {completed.returncode}: {detail}") - if not completed.stdout.strip(): - raise VramUnreadable(f"{argv[0]} produced no output") - return completed.stdout - - -def _parse_json(payload: str, tool: str) -> object: - try: - return json.loads(payload) - except ValueError as exc: - raise VramUnreadable(f"{tool} output is not valid JSON: {exc}") from None - - -def _iter_gpu_entries(parsed: object, tool: str) -> list[tuple[object, object]]: - if isinstance(parsed, list): - return list(enumerate(parsed)) - if isinstance(parsed, dict): - # A known container key wins over the per-key scan below, so a sibling - # metadata object is not mistaken for a GPU entry. - for key in GPU_LIST_KEYS: - nested = parsed.get(key) - if isinstance(nested, list): - return list(enumerate(nested)) - entries = [(key, value) for key, value in parsed.items() if isinstance(value, dict)] - if entries: - return entries - raise VramUnreadable(f"{tool} output has no recognizable GPU list") - - -def read_vram_usage() -> list[tuple[object, float]]: - if shutil.which("amd-smi"): - tool = "amd-smi" - payload = _run_probe(["amd-smi", "metric", "-m", "--json"]) - elif shutil.which("rocm-smi"): - tool = "rocm-smi" - payload = _run_probe(["rocm-smi", "--showmeminfo", "vram", "--json"]) - else: - raise VramUnreadable("neither amd-smi nor rocm-smi is on PATH") - - entries = _iter_gpu_entries(_parse_json(payload, tool), tool) - readings: list[tuple[object, float]] = [] - for device, entry in entries: - try: - readings.append((device, _extract_used_vram(entry))) - except VramUnreadable as exc: - raise VramUnreadable(f"{tool} gpu {device}: {exc}") from None - if not readings: - raise VramUnreadable(f"{tool} reported no GPUs") - return readings - - -def check_vram() -> bool: - limit = int(os.environ.get("IR1_VRAM_LIMIT_MIB", DEFAULT_VRAM_LIMIT_MIB)) - allow_unverified = os.environ.get("IR1_ALLOW_UNVERIFIED_VRAM", "").strip() == "1" - try: - readings = read_vram_usage() - except VramUnreadable as exc: - if allow_unverified: - _warn( - f"VRAM unreadable ({exc}); proceeding because " - "IR1_ALLOW_UNVERIFIED_VRAM=1. Confirm the GPUs are idle by hand." - ) - return True - _fail( - f"VRAM unreadable ({exc}). A busy GPU cannot be ruled out, so the " - "launch is blocked. Fix the probe, or set " - "IR1_ALLOW_UNVERIFIED_VRAM=1 after confirming the GPUs are idle." - ) - return False - - over_limit = [] - for device, mib in readings: - marker = "OVER_LIMIT" if mib > limit else "ok" - print(f"gpu {device}: used_vram_mib={mib:.0f} ({marker})") - if mib > limit: - over_limit.append(device) - if over_limit: - _fail(f"GPU(s) {over_limit} hold more than {limit} MiB; stop them first") - return False - return True - - -def main() -> int: - checks = ( - check_model_path, - check_torch_gpus, - check_foreign_processes, - check_vram, - ) - passed = True - for check in checks: - if not check(): - passed = False - print(f"IR1_RESULT={'PASS' if passed else 'BLOCK'}") - return 0 if passed else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/skills/hyperloom-workload-optimizer/scripts/resume.sh b/skills/hyperloom-workload-optimizer/scripts/resume.sh deleted file mode 100644 index f93b80c..0000000 --- a/skills/hyperloom-workload-optimizer/scripts/resume.sh +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env bash -# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. -# -# See LICENSE for license information. -# -# Resume the session recorded at launch. Re-run IR-2 (install.sh) and IR-1 -# (preflight.py) first, exactly as for a fresh launch; this script does not -# re-check the gates. -# -# --resume-from is always passed explicitly: a bare --resume auto-picks the -# newest session and can target the wrong run. Resume writes its own log so the -# original run log is preserved. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" -# shellcheck source=./_env.sh -. "${SCRIPT_DIR}/_env.sh" - -if [ -z "${SESSION_DIR:-}" ] && [ -f "$LAST_LAUNCH_ENV" ]; then - . "$LAST_LAUNCH_ENV" -fi -if [ -z "${SESSION_DIR:-}" ]; then - echo "ERROR: SESSION_DIR unknown -- read .session_dir from the launch-info JSON" >&2 - exit 1 -fi -if [ ! -f "${SESSION_DIR}/state.json" ]; then - echo "ERROR: ${SESSION_DIR} has no state.json; the CLI refuses to resume it" >&2 - exit 1 -fi - -RESUME_TAG="resume-$(date +%Y%m%d_%H%M%S)" -RUN_LOG="${RUN_DIR}/run_${RESUME_TAG}.log" -PID_FILE="${RUN_DIR}/run_${RESUME_TAG}.pid" -LAUNCH_INFO_FILE="${RUN_DIR}/launch_${RESUME_TAG}.json" - -# shellcheck disable=SC2086 -setsid nohup "$PYTHON" -m hyperloom.inference_optimizer.cli --verbose optimize \ - --resume --resume-from "$SESSION_DIR" \ - --tick-interval-sec 30 \ - --launch-info-file "$LAUNCH_INFO_FILE" \ - ${OPT_FLAGS:-} \ - > "$RUN_LOG" 2>&1 < /dev/null & - -echo $! > "$PID_FILE" - -cat > "$LAST_LAUNCH_ENV" </dev/null || true - done - rm -rf "$WORK" -} -trap cleanup EXIT - -fail() { - echo "[FAIL] $1" >&2 - exit 1 -} - -# launch.sh and resume.sh background the optimizer with setsid nohup, so the stub -# writes its argv dump after they return. Poll for it rather than racing. -wait_for_argv() { - local path="$1" deadline=$((SECONDS + 10)) - while [ ! -e "$path" ]; do - [ "$SECONDS" -lt "$deadline" ] || fail "timed out waiting for $path" - sleep 0.1 - done -} - -# --- stub optimizer ------------------------------------------------------- -# Only "-m hyperloom..." is simulated; every other invocation (the launch-info -# JSON reader inside launch_health.sh) goes to the real interpreter. -cat > "$WORK/stub_python" < "\${STUB_ARGV_OUT}.tmp" -mv "\${STUB_ARGV_OUT}.tmp" "\${STUB_ARGV_OUT}" - -mkdir -p "\${STUB_SESSION_DIR}" -echo '{}' > "\${STUB_SESSION_DIR}/manifest.json" -echo '{"phase": "PRELUDE"}' > "\${STUB_SESSION_DIR}/state.json" - -# Report a forked child as the optimizer pid. The launcher's \$! is the wrapper, -# which is a different process -- that gap is what the health check must close. -sleep 30 & -child=\$! - -if [ -n "\$info" ]; then - printf '{"pid": %s, "session_dir": "%s"}\n' "\$child" "\${STUB_SESSION_DIR}" > "\$info" -fi - -wait "\$child" -STUB -chmod +x "$WORK/stub_python" - -# --- fixture -------------------------------------------------------------- -# Each fixture is self-contained: .env and workload.env hold absolute paths, so a -# variant must be generated fresh rather than copied from another root. -make_fixture() { - local root="$1" - # hyperloom/ is what `pip install --target` leaves behind, and what marks this - # root as an install directory rather than whatever the caller happened to cd to. - mkdir -p "$root/model" "$root/data/runtime" "$root/data/optimizer_runs" \ - "$root/hyperloom/inference_optimizer" - - # A value with a space and a colon, double-quoted the way hyperloom-setup - # writes it. Unquoted, the .env source in _env.sh would fail with exit 127. - cat > "$root/.env" < "$root/model/config.json" - echo 'export KERNEL_AGENT_MARKER=1' > "$root/data/runtime/kernel-agent.env.sh" - - cat > "$root/data/optimizer_runs/workload.env" < "$WORK/launch.out" - -LAST_LAUNCH="$ROOT/data/optimizer_runs/last_launch.env" -[ -f "$LAST_LAUNCH" ] || fail "launch.sh did not write last_launch.env" -# shellcheck disable=SC1090 -. "$LAST_LAUNCH" -WRAPPER_PID="$(cat "$PID_FILE")" -STARTED_PIDS+=("$WRAPPER_PID") - -grep -q "^export LAUNCH_INFO_FILE=" "$LAST_LAUNCH" \ - || fail "last_launch.env is missing LAUNCH_INFO_FILE" -echo "[ok] launch.sh recorded the run handles on disk" - -wait_for_argv "$STUB_ARGV_OUT" -for expected in --model --framework --tp --ep --conc --isl --osl --precision \ - --max-hours --target-gain; do - grep -qx -- "$expected" "$STUB_ARGV_OUT" || fail "$expected not passed to the CLI" -done -grep -qx -- "--no-kernel" "$STUB_ARGV_OUT" \ - || fail "OPT_FLAGS did not word-split into a separate argument" -grep -qx -- "$ROOT/model" "$STUB_ARGV_OUT" || fail "MODEL_PATH not passed" -echo "[ok] every workload.env value reached the CLI, OPT_FLAGS word-split" - -# --- health check --------------------------------------------------------- -bash "$SCRIPTS_DIR/launch_health.sh" > "$WORK/health.out" -grep -q "optimizer_alive=true" "$WORK/health.out" || fail "health check reported no live optimizer" - -REPORTED_PID="$(cat "$PID_FILE")" -STARTED_PIDS+=("$REPORTED_PID") -JSON_PID="$("$REAL_PYTHON" -c 'import json,sys;print(json.load(open(sys.argv[1]))["pid"])' "$LAUNCH_INFO_FILE")" -[ "$WRAPPER_PID" != "$JSON_PID" ] \ - || fail "fixture is not exercising the gap: wrapper pid equals the optimizer pid" -[ "$REPORTED_PID" = "$JSON_PID" ] \ - || fail "pid file holds $REPORTED_PID, expected the optimizer pid $JSON_PID" -echo "[ok] pid file holds the optimizer pid, not the setsid wrapper" - -grep -q "^export SESSION_DIR=" "$LAST_LAUNCH" || fail "SESSION_DIR was not persisted" -echo "[ok] session_dir persisted for the monitor and resume steps" - -# --- health check failure path ------------------------------------------- -# A run whose launch-info JSON never appeared: the handles exist but the file -# does not, which is what a crash during startup looks like. -BROKEN="$WORK/broken" -make_fixture "$BROKEN" -cat > "$BROKEN/data/optimizer_runs/last_launch.env" < "$WORK/health_fail.out" 2>&1; then - fail "health check passed even though the launch-info JSON was missing" -fi -grep -q "launch_broken.json not written" "$WORK/health_fail.out" \ - || fail "health check did not name the missing launch-info JSON" -echo "[ok] health check blocks when the launch-info JSON never appeared" - -# --- resume --------------------------------------------------------------- -export STUB_ARGV_OUT="$WORK/argv_resume.txt" -bash "$SCRIPTS_DIR/resume.sh" > "$WORK/resume.out" -# shellcheck disable=SC1090 -. "$LAST_LAUNCH" -STARTED_PIDS+=("$(cat "$PID_FILE")") - -wait_for_argv "$WORK/argv_resume.txt" -grep -qx -- "--resume-from" "$WORK/argv_resume.txt" \ - || fail "resume.sh did not pass --resume-from explicitly" -if grep -qx -- "--model" "$WORK/argv_resume.txt"; then - fail "resume.sh passed --model, which resume must omit" -fi -case "$RUN_LOG" in - *run_resume-*) ;; - *) fail "resume.sh reused the original run log: $RUN_LOG" ;; -esac -echo "[ok] resume targets the recorded session and writes its own log" - -# --- missing gate inputs -------------------------------------------------- -NOPLAN="$WORK/noplan" -make_fixture "$NOPLAN" -rm -f "$NOPLAN/data/optimizer_runs/workload.env" -if (cd "$NOPLAN" && INSTALL_DIR="$NOPLAN" bash "$SCRIPTS_DIR/launch.sh") \ - > "$WORK/noplan.out" 2>&1; then - fail "launch.sh started without workload.env" -fi -grep -q "workload.env missing" "$WORK/noplan.out" \ - || fail "launch.sh did not name the missing workload.env" -echo "[ok] launch.sh refuses to start without the confirmed plan" - -# A directory that is not the install directory -- what the caller gets when the -# shell is left somewhere else and INSTALL_DIR falls back to the current one. -WRONGDIR="$WORK/wrongdir" -make_fixture "$WRONGDIR" -rm -rf "$WRONGDIR/hyperloom" -# INSTALL_DIR is unset so the script falls back to the current directory, which -# is the case the check exists for; the earlier export would otherwise mask it. -if (cd "$WRONGDIR" && env -u INSTALL_DIR bash "$SCRIPTS_DIR/launch.sh") \ - > "$WORK/wrongdir.out" 2>&1; then - fail "launch.sh started from a directory holding no hyperloom/" -fi -grep -q "holds no hyperloom/" "$WORK/wrongdir.out" \ - || fail "launch.sh did not name the directory that is missing hyperloom/" -echo "[ok] launch.sh refuses to start outside the install directory" - -NOINSTALL="$WORK/noinstall" -make_fixture "$NOINSTALL" -rm -f "$NOINSTALL/data/runtime/kernel-agent.env.sh" -if (cd "$NOINSTALL" && INSTALL_DIR="$NOINSTALL" bash "$SCRIPTS_DIR/launch.sh") \ - > "$WORK/noinstall.out" 2>&1; then - fail "launch.sh started without the kernel-agent env from install.sh" -fi -grep -q "run IR-2" "$WORK/noinstall.out" \ - || fail "launch.sh did not point at IR-2 when the kernel-agent env was missing" -echo "[ok] launch.sh refuses to start before IR-2 has run" - -echo -echo "all launch-flow checks passed" diff --git a/skills/hyperloom-workload-optimizer/scripts/tests/test_preflight.py b/skills/hyperloom-workload-optimizer/scripts/tests/test_preflight.py deleted file mode 100644 index d0caa02..0000000 --- a/skills/hyperloom-workload-optimizer/scripts/tests/test_preflight.py +++ /dev/null @@ -1,232 +0,0 @@ -# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. -# -# See LICENSE for license information. - -"""Regression tests for the IR-1 VRAM gate in ../preflight.py. - -Each case installs a fake amd-smi that reports a busy GPU in a different output -shape, then asserts the gate blocks the launch. Before the fail-closed rewrite, -an unexpected shape or a non-zero probe exit let a GPU holding ~140 GiB pass as -idle, so these cases are the reproduction for that bug. - -Run standalone; no pytest or third-party dependency required: - - python3 scripts/tests/test_preflight.py -""" - -from __future__ import annotations - -import importlib.util -import os -import stat -import sys -import tempfile -from pathlib import Path - -BUSY_MIB = 143360 -SCRIPT = Path(__file__).resolve().parents[1] / "preflight.py" - - -def _load_preflight(): - spec = importlib.util.spec_from_file_location("preflight_under_test", SCRIPT) - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -preflight = _load_preflight() - - -def _install_fake_smi(directory: Path, name: str, body: str) -> None: - path = directory / name - path.write_text(f"#!/bin/sh\n{body}\n") - path.chmod(path.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) - - -def _emit(payload: str) -> str: - quoted = payload.replace("'", "'\\''") - return f"printf '%s' '{quoted}'" - - -CASES: list[tuple[str, str | None, bool, dict[str, str]]] = [ - ( - "expected shape, GPU busy", - _emit( - '[{"gpu": 0, "mem_usage": {"total_vram": {"value": 196592, "unit": "MB"},' - ' "used_vram": {"value": %d, "unit": "MB"}}}]' % BUSY_MIB - ), - False, - {}, - ), - ( - "expected shape, GPU idle", - _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": 283, "unit": "MB"}}}]'), - True, - {}, - ), - ( - "top-level object instead of list, GPU busy", - _emit('{"gpu_0": {"mem_usage": {"used_vram": {"value": %d, "unit": "MB"}}}}' % BUSY_MIB), - False, - {}, - ), - ( - "gpu list wrapped in a top-level container key, GPU busy", - _emit( - '{"gpu_data": [{"gpu": 0, "mem_usage": {"used_vram":' - ' {"value": %d, "unit": "MB"}}}]}' % BUSY_MIB - ), - False, - {}, - ), - ( - "gpu list wrapped in a top-level container key, GPU idle", - _emit( - '{"gpu_data": [{"gpu": 0, "mem_usage": {"used_vram":' - ' {"value": 283, "unit": "MB"}}}]}' - ), - True, - {}, - ), - ( - "wrapped gpu list alongside a sibling metadata key, GPU idle", - _emit( - '{"gpu_data": [{"gpu": 0, "mem_usage": {"used_vram":' - ' {"value": 283, "unit": "MB"}}}],' - ' "metadata": {"version": "26.2.2"}}' - ), - True, - {}, - ), - ( - "scalar used_vram instead of value/unit map, GPU busy", - _emit('[{"gpu": 0, "mem_usage": {"used_vram": %d}}]' % BUSY_MIB), - False, - {}, - ), - ( - "container renamed mem_usage -> mem, GPU busy", - _emit('[{"gpu": 0, "mem": {"used_vram": {"value": %d, "unit": "MB"}}}]' % BUSY_MIB), - False, - {}, - ), - ( - "used_vram key absent (renamed vram_used), GPU busy", - _emit('[{"gpu": 0, "mem_usage": {"vram_used": {"value": %d, "unit": "MB"}}}]' % BUSY_MIB), - False, - {}, - ), - ( - "no used-VRAM field at all", - _emit('[{"gpu": 0, "mem_usage": {"total_vram": {"value": 196592, "unit": "MB"}}}]'), - False, - {}, - ), - ( - "amd-smi exits non-zero (driver error)", - 'echo "Unable to communicate with the amdgpu driver" >&2\nexit 1', - False, - {}, - ), - ( - "warning banner before JSON, GPU busy", - 'echo "WARNING: amdgpu version mismatch"\n' - + _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": %d, "unit": "MB"}}}]' % BUSY_MIB), - False, - {}, - ), - ( - "used_vram explicitly null", - _emit('[{"gpu": 0, "mem_usage": {"used_vram": null}}]'), - False, - {}, - ), - ( - "empty GPU list", - _emit("[]"), - False, - {}, - ), - ( - "GiB unit, GPU busy", - _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": 140, "unit": "GiB"}}}]'), - False, - {}, - ), - ( - "MB unit just under the MiB ceiling", - _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": 500, "unit": "MB"}}}]'), - True, - {}, - ), - ( - "unknown unit", - _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": 12, "unit": "furlongs"}}}]'), - False, - {}, - ), - ( - "second GPU unreadable while first is idle", - _emit( - '[{"gpu": 0, "mem_usage": {"used_vram": {"value": 100, "unit": "MB"}}},' - ' {"gpu": 1, "mem_usage": {}}]' - ), - False, - {}, - ), - ( - "no probe tool on PATH", - None, - False, - {}, - ), - ( - "no probe tool on PATH with explicit override", - None, - True, - {"IR1_ALLOW_UNVERIFIED_VRAM": "1"}, - ), - ( - "unreadable probe with explicit override", - 'exit 1', - True, - {"IR1_ALLOW_UNVERIFIED_VRAM": "1"}, - ), -] - - -def run_case(body: str | None, extra_env: dict[str, str]) -> bool: - saved_env = dict(os.environ) - with tempfile.TemporaryDirectory() as tmp: - fake_dir = Path(tmp) - if body is not None: - _install_fake_smi(fake_dir, "amd-smi", body) - try: - # Isolate PATH so only the fake probe (if any) is discoverable. - os.environ["PATH"] = str(fake_dir) - os.environ.pop("IR1_ALLOW_UNVERIFIED_VRAM", None) - os.environ["IR1_VRAM_LIMIT_MIB"] = "500" - os.environ.update(extra_env) - return preflight.check_vram() - finally: - os.environ.clear() - os.environ.update(saved_env) - - -def main() -> int: - failures = 0 - for name, body, expected_pass, extra_env in CASES: - actual_pass = run_case(body, extra_env) - ok = actual_pass == expected_pass - if not ok: - failures += 1 - want = "PASS" if expected_pass else "BLOCK" - got = "PASS" if actual_pass else "BLOCK" - print(f"[{'ok' if ok else 'FAIL'}] {name}: want {want}, got {got}") - print() - print(f"{len(CASES) - failures}/{len(CASES)} cases behaved as expected") - return 1 if failures else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/skills/hyperloom-workload-optimizer/setup.md b/skills/hyperloom-workload-optimizer/setup.md deleted file mode 100644 index ef3ae43..0000000 --- a/skills/hyperloom-workload-optimizer/setup.md +++ /dev/null @@ -1,115 +0,0 @@ -# Hyperloom Workspace Bootstrap - -Use this file from [SKILL.md](SKILL.md) Phase 0. The optimizer runtime ships in the -Hyperloom Python wheel; this skill does not bundle it. - -Steps 0–2 only prepare the workspace. Return to [SKILL.md](SKILL.md) and continue -with Phase 1 (environment prep), Phase 2 (workload intake), then Phase 3 -(install, launch, monitor). - -## Step 0 — Confirm the install directory - -The wheel installs into a target directory (`pip install --target `) that -also holds `.env` and runtime artifacts. Do not silently use the current -directory. Show the resolved `pwd` and confirm it with the user, or let them -pick another dedicated path. `cd` into the chosen directory before installing. - -## Step 1 — Install the Hyperloom wheel - -Skip when `hyperloom/` already exists in the directory (wheel layout) or -`src/hyperloom/` exists (source checkout). - -The runtime is published to PyPI as `hyperloom-inference-optimizer`. List the -releases, tell the user the newest one, and ask whether to install it or a -version they name. - -List with `--pre` so prereleases are visible, and install an exact `==` version -so a later bootstrap installs the same runtime. - -```bash -cd "$INSTALL_DIR" # the directory confirmed in Step 0 -pip index versions hyperloom-inference-optimizer --pre -pip install hyperloom-inference-optimizer== --target . -``` - -After install, confirm: - -- `hyperloom/inference_optimizer/assets/install.sh` exists -- `.cursor/skills/hyperloom-custom-advanced/SKILL.md` exists (or `.claude/` / - `.agents/` equivalent) - -Restart the agent if the new skills are not visible. - -## Step 2 — Credentials and run mode - -**Preferred:** run the bundled setup skill installed by the wheel: - -- Cursor / Codex: `$hyperloom-setup` or load `hyperloom-setup` from - `.cursor/skills/hyperloom-setup/SKILL.md` -- Claude Code: `/hyperloom-setup` - -That skill owns the credential questions, the run-mode question, and the bare-metal -host setup backend (`install_baremetal.sh`). It is the source of truth for which -LLM providers are accepted and which variable names they use, so do not -reimplement it here or guess variable names from memory. - -If it is not available, restart the agent and re-check that -`.cursor/skills/hyperloom-setup/SKILL.md` (or the `.claude/` / `.agents/` -equivalent) exists. If it is still missing, the wheel install is incomplete — -stop and report that, rather than hand-writing `.env`. - -When it returns, verify `.env` before continuing. It must define -`HYPERLOOM_RUN_MODE` (`baremetal` or `docker`), `USER_DATA_PATH`, -`HYPERLOOM_SKILL_PATH`, and one complete LLM credential set. Check the values are -real, not placeholders. `HYPERLOOM_SKILL_PATH` must point at the packaged -optimizer skill that exists on disk — `hyperloom/inference_optimizer/SKILL.md` -for a wheel install, `src/hyperloom/inference_optimizer/SKILL.md` for a source -checkout. - -Inspect `.env` with `grep` rather than echoing it, and never print a key value -into chat. Phase 3 loads `.env` with a shell `source`, so any value containing a -space must stay double-quoted — `hyperloom-setup` writes -`ANTHROPIC_CUSTOM_HEADERS="Ocp-Apim-Subscription-Key: ${ANTHROPIC_API_KEY}"` for -exactly that reason. A hand-edited `.env` that drops those quotes makes the launch -shell fail with exit 127. - -Do not write your own live probe of the LLM endpoint. Hyperloom's `optimize` -preflight already probes `/models` with the operator's auth and custom -headers, and refuses to start on an auth failure, so a wrong key fails there in -the first seconds rather than mid-run. Two of its outcomes are only warnings, and -both must be surfaced to the user instead of scrolled past: - -- `gateway has no /models route (HTTP 404/405) ... Proceeding` — the key is - unverified, not verified. Expected when `ANTHROPIC_BASE_URL` is native - `https://api.anthropic.com`, which has no `/models` at that path. -- `gateway catalog unreachable ... Proceeding with custom orchestration model - support enabled` — this is the auth failure downgraded to a warning because a - custom model id was allowed. A bad key looks like this instead of an error. - -## Docker run mode — the Phase 1 contract - -When `HYPERLOOM_RUN_MODE=docker`, `hyperloom-custom-advanced` owns the container -steps (image, `docker run` flags, setup inside the container). Follow it rather -than inventing flags. Regardless of how it gets there, all of the following must -hold before Phase 3 launches anything, and every later command in this skill runs -*inside* the container: - -- the container is running and you have a shell in it -- `/dev/kfd` and `/dev/dri` are mapped, and `amd-smi` or `rocm-smi` works inside -- the install directory from Step 0 and `USER_DATA_PATH` are mounted at the same - absolute paths inside the container, so `.env` and run state resolve identically -- `MODEL_PATH` resolves inside the container -- the setup backend has been run inside the container - -If any of these is unmet, fix it before Phase 3. Launching with a half-prepared -container produces failures that look like optimizer bugs. - -## Step 3 — Return to SKILL.md - -Bootstrap is complete. Go back to [SKILL.md](SKILL.md) and continue with Phase 1 -(environment prep), Phase 2 (workload intake), then Phase 3 (install, preflight, -launch, monitor). - -Read `@${HYPERLOOM_SKILL_PATH}` only for edge cases the catalog skill does not -cover: multi-node, `--framework atom`, critic/robustness backend selection, -aiter cache topology, and the full failure matrix. diff --git a/walkthroughs/hyperloom-workload-optimizer.md b/walkthroughs/hyperloom-workload-optimizer.md index cfab32a..9d5a15a 100644 --- a/walkthroughs/hyperloom-workload-optimizer.md +++ b/walkthroughs/hyperloom-workload-optimizer.md @@ -39,6 +39,8 @@ Use these values for the placeholders in the prompts below: - ``: `vllm` or `sglang` - ``: `MI300X`, `MI325X`, or `MI355X` +- ``: absolute path to a local model directory holding `config.json`. + Drop that sentence from a demo prompt to run the model the demo names. You do **not** need to decide these before Step 2: @@ -74,17 +76,18 @@ You should see `hyperloom-workload-optimizer` in the list. In the dedicated workspace, ask the agent: ```text -Install Hyperloom and set up the execution environment for on . +Install Hyperloom in this workspace and run setup for on . ``` -This step prepares the workspace and execution environment only: +This step installs Hyperloom and records how the run will happen. It launches +nothing: -1. **Phase 0, Bootstrap:** confirm the install directory, install a pinned - `hyperloom-inference-optimizer` release from PyPI with `pip install --target .`, - then run `/hyperloom-setup` to write `.env` (credentials + run mode only). -2. **Phase 1, Environment prep:** choose Docker or bare metal, then prepare - that environment. On bare metal, confirm the host stack; in Docker, start a - long-running container and run the in-container setup first. +1. **Install.** Confirm the install directory, then install the + `hyperloom-inference-optimizer` release from PyPI with `pip install --target .`. +2. **Setup.** Run `/hyperloom-setup`. It asks for credentials, `USER_DATA_PATH` + and the run mode, then writes `.env`. On bare metal it also prepares the host + stack and can install the serving framework. In Docker it writes `.env` only — + the container belongs to Step 3. Choose **Docker** when you want the validated, reproducible ROCm and framework stack and can run containers with GPU devices mapped in. Choose **bare metal** @@ -93,12 +96,12 @@ containers are unavailable. Bare metal is more sensitive to host packages and can modify the environment, so prefer Docker for first-time walkthroughs when it is available. -Save model selection, workload choices and launch approval for Step 3. +Save model selection, workload choices and launch approval for Step 3. Setup +offers to start a run when it finishes; decline it and come back here at Step 3. Verify the setup handoff: ```bash -ls hyperloom/inference_optimizer/assets/install.sh test -f .env grep -E '^(USER_DATA_PATH|HYPERLOOM_RUN_MODE)=' .env ``` @@ -106,38 +109,39 @@ grep -E '^(USER_DATA_PATH|HYPERLOOM_RUN_MODE)=' .env ## Step 3: Launch an optimization -Start this step only after Step 2 has written `.env` and prepared the execution -environment. Step 3 reuses the Docker or bare-metal run mode recorded in `.env`; -do not choose it again here. +Start this step only after Step 2 has written `.env`. Step 3 reuses the Docker or +bare-metal run mode recorded there; do not choose it again here. -There are three ways to run. Paste the prompt that matches, replacing -`` and `` with your target. The flags in each are a set, so -pass them together. The two demos carry the same workload and flags as the -matching Hyperloom demo skill; you can point either one at your own model, and -**Custom** is for changing the workload itself. +There are three ways to run. Ask for the one you want; the agent loads the +matching demo skill the wheel installed, and that skill owns the workload preset, +the budget and every optimizer flag. Nothing here restates them, so a change to +the CLI reaches you through the wheel rather than through this page. -**1. 3-hour demo.** Serving and config parameters only; the kernel agent is -off. Swap in your own model if you want; with a 3-hour budget keep it at 8B or -below. +In Docker mode that skill is also what starts the container and runs setup inside +it, on the host `/hyperloom-setup` recorded. Expect that to happen here, not in +Step 2. + +Each demo names the model it was tuned around, and either can run yours instead — +give the path, or drop that sentence to take the demo's own. The preset workload +does not change with the model, so if yours is much larger or a different +architecture, use **Custom** and set the values yourself. + +**1. 3-hour demo (Qwen3-8B).** Serving and config parameters only; the kernel +agent is off. The shortest end-to-end check; with a 3-hour budget keep the model +at 8B or below. ```text -Optimize Qwen/Qwen3-8B with on : TP=1, conc=64, ISL=1024, -OSL=1024, precision bf16, target-gain 30, max-hours 3, serving parameters only: ---max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01 ---no-kernel --no-enable-conc-sweep --no-enable-roofline. -Launch and monitor. +Run the 3-hour Hyperloom Qwen3-8B demo with on . Use the +model at instead of the demo default. Launch and monitor. ``` -**2. 12-hour demo.** Every lever, kernel rewrites included. The kernel agent -needs room to profile, rewrite and revalidate hot kernels, which is where the -larger gains come from. +**2. 12-hour demo (Qwen3-14B-FP8).** Every lever, kernel rewrites included. The +kernel agent needs room to profile, rewrite and revalidate hot kernels, which is +where the larger gains come from. ```text -Optimize Qwen/Qwen3-14B-FP8 with on : TP=1, conc=64, -ISL=1024, OSL=1024, precision fp8, target-gain 50, max-hours 12, all components -enabled: ---max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42. -Launch and monitor. +Run the 12-hour Hyperloom Qwen3-14B-FP8 demo with on . Use +the model at instead of the demo default. Launch and monitor. ``` **3. Custom.** Ask for a run and let the agent take you through the choices: @@ -152,9 +156,9 @@ Optimize a model with Hyperloom on this host. Walk me through the choices. Whichever of the three you use, the agent shows the full launch plan (every resolved value and every flag) and waits for your approval before it starts. -Once workload values are resolved, the agent should run `install.sh` (IR-2), -the GPU preflight (IR-1), launch `hyperloom.inference_optimizer.cli optimize` -with those values as CLI flags, then poll `state.json`. +Once the values are resolved, the agent clears the runtime install and GPU +preflight gates the optimizer skill owns, starts the run in the background, then +polls the session state. ## Step 4: Read results @@ -186,5 +190,7 @@ Use these entries when a step fails. - Resume: `Resume the most recent Hyperloom session for .` - Ran the 3-hour demo and want kernel rewrites? Start the 12-hour run from - Step 3 rather than raising `--max-hours` on the 3-hour demo's flags. -- Advanced flags: see `reference.md` in the skill folder. + Step 3 rather than stretching the 3-hour demo's budget. +- Advanced flags: use **Custom** in Step 3 and let the agent derive them, or read + the [optimizer skill](https://github.com/AMD-AGI/Hyperloom/blob/main/src/hyperloom/inference_optimizer/SKILL.md) + the wheel installs.