From 51dd075957349f1927633885f781c082f95b94be Mon Sep 17 00:00:00 2001 From: lishuoshuo-amd Date: Wed, 9 Sep 2026 11:32:27 +0800 Subject: [PATCH 1/2] fix(hyperloom): align demo flag sets with the current optimizer CLI The hyperloom-workload-optimizer skill has not been updated since it landed (a06c973, 2026-08-14). Upstream has since merged EXPLORE into FRAMEWORK and removed the explore-only flags, so the published 3-hour demo no longer runs: error: unrecognized arguments: --explore-force-exit-budget-pct 0.01 --explore-force-exit-hours-remaining 0.05 argparse rejects unknown flags, so anyone following the skill or the walkthrough hits exit 2 at launch. `--no-explore` is gone the same way. The 12-hour set parsed but was silently wrong: `--max-minutes-explore-pct` is now an alias onto the same dest as `--max-minutes-framework-pct`, so the pair `--max-minutes-framework-pct 0.01 --max-minutes-explore-pct 0.42` resolved to 0.42, not 0.01. Adopt the flag sets the Hyperloom demo skills carry today (AMD-AGI/Hyperloom #1417) and explain the two things that were easy to get wrong: 0.50 is the share *before* redistribution (--no-kernel adds the kernel agent's freed share on top, capped at one wall clock), and --no-framework-agent must not be used to mean "no kernel rewrites" since it skips OPTIMIZE entirely. Also refresh reference.md's IR-4/IR-6 section, which documented the deleted force-exit flags as live defaults. Every recommended flag set is verified to parse against the current CLI parser. --- skills/hyperloom-workload-optimizer/SKILL.md | 55 +++++++++++-------- .../hyperloom-workload-optimizer/reference.md | 15 +++-- walkthroughs/hyperloom-workload-optimizer.md | 11 ++-- 3 files changed, 47 insertions(+), 34 deletions(-) diff --git a/skills/hyperloom-workload-optimizer/SKILL.md b/skills/hyperloom-workload-optimizer/SKILL.md index a9a68e3..9916baf 100644 --- a/skills/hyperloom-workload-optimizer/SKILL.md +++ b/skills/hyperloom-workload-optimizer/SKILL.md @@ -185,8 +185,9 @@ Never copy API keys into chat output. | MAX_HOURS | `--max-hours` | CLI `2.0` | offer `3` (quick) or `12` (full); see below | | TARGET_GAIN | `--target-gain` | `30` | desired % gain | -**Optional:** `--no-explore`, `--no-enable-conc-sweep`, `--gpu-type`, -`--server-args`, `--compare-against-gpu`, `--quantize` prelude. +**Optional:** `--no-framework-agent`, `--no-kernel`, `--no-enable-conc-sweep`, +`--no-enable-roofline`, `--gpu-type`, `--server-args`, `--compare-against-gpu`, +`--quantize` prelude. Infer `PRECISION` from the model name when obvious (e.g. an `FP8` model implies `--precision fp8`) and confirm it — do not silently keep the `bf16` default. @@ -209,12 +210,20 @@ Expect a modest validated gain, or an honest 0% when the workload has no parameter headroom. ```text ---max-hours 3 --precision bf16 ---no-framework-agent --no-kernel --no-enable-conc-sweep --no-enable-roofline ---max-minutes-explore-pct 0.39 --max-minutes-sweep-pct 0.01 ---explore-force-exit-budget-pct 0.01 --explore-force-exit-hours-remaining 0.05 +--max-hours 3 --precision bf16 --target-gain 30 +--max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01 +--no-kernel --no-enable-conc-sweep --no-enable-roofline ``` +`0.50` is the share *before* redistribution. With `--no-kernel`, KERNEL_AGENT is +disabled and its freed share is added on top, so `0.50` becomes ~0.99 of the wall +clock for OPTIMIZE. Raising `0.50` buys almost nothing — the post-redistribution +share is capped at one full wall clock and the excess is discarded. + +Do **not** pass `--no-framework-agent` here: it skips OPTIMIZE entirely, which is +the one phase this profile relies on. `--no-kernel` is what makes it "no kernel +rewrites". + **2. 12-hour demo** (`hyperloom-qwen3-14b-fp8-12h`) — `Qwen/Qwen3-14B-FP8` unless the user names another model, TP=1, CONC=64, ISL=OSL=1024, `--precision fp8` matched to the chosen checkpoint, every lever with kernel @@ -222,11 +231,14 @@ rewrites included. The kernel agent needs room to profile, rewrite and revalidate, which is where the larger gains come from. ```text ---max-hours 12 --precision fp8 ---max-minutes-framework-pct 0.01 --max-minutes-explore-pct 0.42 ---max-minutes-kernel-pct 0.42 +--max-hours 12 --precision fp8 --target-gain 50 +--max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42 ``` +Do not add `--max-minutes-explore-pct`: configuration search and source landing +are two arms of one phase with one budget, so that spelling is an alias onto +`--max-minutes-framework-pct` and silently overwrites it. + **3. Custom** — the user brings their own model or workload instead of taking a demo. Walk through the fields in the table above and the phase toggles, one question at a time, and derive the flags from the answers rather than asking for @@ -250,13 +262,11 @@ Launch plan — please confirm: TP=1 EP=1 CONC=64 ISL=1024 OSL=1024 PRECISION=fp8 - MAX_HOURS=3 TARGET_GAIN=20% - profile 3-hour demo — no kernel, no framework agent, no roofline - flags --no-framework-agent --no-kernel --no-enable-conc-sweep - --no-enable-roofline - --max-minutes-explore-pct 0.39 --max-minutes-sweep-pct 0.01 - --explore-force-exit-budget-pct 0.01 - --explore-force-exit-hours-remaining 0.05 + MAX_HOURS=3 TARGET_GAIN=30% + profile 3-hour demo — no kernel rewrites, no conc sweep, no roofline + flags --max-minutes-framework-pct 0.50 + --max-minutes-sweep-pct 0.01 + --no-kernel --no-enable-conc-sweep --no-enable-roofline RUN_MODE baremetal ``` @@ -291,10 +301,10 @@ export ISL=1024 export OSL=1024 export PRECISION=fp8 export MAX_HOURS=3 -export TARGET_GAIN=20 +export TARGET_GAIN=30 # The whole flag set for the approved profile, space-separated. The 3-hour # demo is shown; a 12-hour run swaps in its own set. -export OPT_FLAGS="--no-framework-agent --no-kernel --no-enable-conc-sweep --no-enable-roofline --max-minutes-explore-pct 0.39 --max-minutes-sweep-pct 0.01 --explore-force-exit-budget-pct 0.01 --explore-force-exit-hours-remaining 0.05" +export OPT_FLAGS="--max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01 --no-kernel --no-enable-conc-sweep --no-enable-roofline" EOF ``` @@ -364,10 +374,11 @@ bash "${SKILL_DIR}/scripts/launch.sh" Every workload value comes from the confirmed `workload.env`; the script has no `${VAR:-default}` fallbacks, so a missing value fails loudly instead of launching -a different config. Put any optional Phase 2 flags (`--no-kernel`, `--no-explore`, -`--gpu-type`, `--model-class`, `--server-args`, `--compare-against-gpu`, -`--quantize`, phase budget flags) into `OPT_FLAGS` in `workload.env`. `OPT_FLAGS` -is word-split, so quote any flag value that contains spaces, e.g. +a different config. Put any optional Phase 2 flags (`--no-kernel`, +`--no-framework-agent`, `--gpu-type`, `--model-class`, `--server-args`, +`--compare-against-gpu`, `--quantize`, phase budget flags) into `OPT_FLAGS` in +`workload.env`. `OPT_FLAGS` is word-split, so quote any flag value that contains +spaces, e.g. `export OPT_FLAGS='--server-args "--foo bar"'`. ### Launch health check (30 s after start) diff --git a/skills/hyperloom-workload-optimizer/reference.md b/skills/hyperloom-workload-optimizer/reference.md index 221bfdb..4519a88 100644 --- a/skills/hyperloom-workload-optimizer/reference.md +++ b/skills/hyperloom-workload-optimizer/reference.md @@ -45,14 +45,17 @@ Any failure → treat as a fresh launch and re-run `install.sh`. `_preflight()` runs `preflight_kb.sh`. Exit `1` auto-enables `--degraded-kb` / `--degraded-pr`; launch continues. IR-3 never aborts. -### IR-4 / IR-6 — EXPLORE contracts (Coordinator-internal) +### IR-4 — OPTIMIZE phase contracts (Coordinator-internal) -- **IR-4:** EXPLORE is specialist-informed; GPU specialists lease cards via +Configuration search and source landing are two arms of **one** phase +(OPTIMIZE / FRAMEWORK_AGENT) sharing one budget. The separate EXPLORE phase and +its `--explore-force-exit-*` flags no longer exist. + +- **IR-4:** OPTIMIZE is specialist-informed; GPU specialists lease cards via `gpu_research_lane` and must not touch production serving on port 8888. -- **IR-6:** EXPLORE force-exits when wall-clock remaining < - `--explore-force-exit-hours-remaining` (default 3 h) or phase budget < - `--explore-force-exit-budget-pct` (default 20%). -- Plateau signals are advisory; IR-6 and per-phase budgets are hard gates. + Specialists author patches into an isolated worktree; `integrate_patch` does + the `git apply` plus throughput/accuracy gate after Critic review. +- Plateau signals are advisory; the per-phase budgets are the hard gates. ### IR-8 — `--framework atom` is single-node only diff --git a/walkthroughs/hyperloom-workload-optimizer.md b/walkthroughs/hyperloom-workload-optimizer.md index 45847d6..cfab32a 100644 --- a/walkthroughs/hyperloom-workload-optimizer.md +++ b/walkthroughs/hyperloom-workload-optimizer.md @@ -123,9 +123,8 @@ below. ```text Optimize Qwen/Qwen3-8B with on : TP=1, conc=64, ISL=1024, OSL=1024, precision bf16, target-gain 30, max-hours 3, serving parameters only: ---no-framework-agent --no-kernel --no-enable-conc-sweep --no-enable-roofline ---max-minutes-explore-pct 0.39 --max-minutes-sweep-pct 0.01 ---explore-force-exit-budget-pct 0.01 --explore-force-exit-hours-remaining 0.05. +--max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01 +--no-kernel --no-enable-conc-sweep --no-enable-roofline. Launch and monitor. ``` @@ -135,10 +134,10 @@ larger gains come from. ```text Optimize Qwen/Qwen3-14B-FP8 with on : TP=1, conc=64, -ISL=1024, OSL=1024, precision fp8, target-gain 30, max-hours 12, all components +ISL=1024, OSL=1024, precision fp8, target-gain 50, max-hours 12, all components enabled: ---max-minutes-framework-pct 0.01 --max-minutes-explore-pct 0.42 ---max-minutes-kernel-pct 0.42. Launch and monitor. +--max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42. +Launch and monitor. ``` **3. Custom.** Ask for a run and let the agent take you through the choices: From 746ab920b2f3c93eb637c915193fa1e9965844f1 Mon Sep 17 00:00:00 2001 From: lishuoshuo-amd Date: Wed, 9 Sep 2026 17:14:30 +0800 Subject: [PATCH 2/2] test(hyperloom): stop grading expectations the harness cannot settle These behavioral cases were 14/14 on the Instinct runner in August (#140, #141) and the skill has not changed since. Four runs on the current model each failed a different single expectation, so fix the three that ask for something the case itself rules out. `hyperloom-bootstrap-phase-discipline` required the agent to "run hyperloom-setup for credentials and run mode". Setup cannot finish in a single-turn headless run -- its credential and run-mode questions have nobody to answer them -- so the agent correctly stops in Phase 0 and the judge reads that as setup never having run: The agent confirmed the install directory and installed the wheel but did not run hyperloom-setup; no .env exists and it paused to ask the user for run mode and credentials instead. It did enter setup. That run asked for base URL, CLAUDE_MODEL and USER_DATA_PATH, none of which this skill's own prose names, so they came from the wheel-installed setup skill. Grade installing the wheel and entering setup, which a headless run can reach. `hyperloom-launcher-gates` asks for three or four sentences and forbids running anything, yet required install.sh and kernel-agent.env.sh be mentioned "before launching the optimizer". Read as an ordering of real events that is unreachable, and the judge read it that way: The agent described IR-2 conceptually but never launched the optimizer, so there was no pre-launch mention of running install.sh before an actual launch. State the gate's content instead, which is what the answer already carries. `hyperloom-workload-intake` required the workload values be named "as its own intake step". Dropping that clause did not settle it -- the judge then failed the bare wording while conceding the values were all present: did not itself request or name the specific workload values needed from the user in the required actionable sense -- though it did list model path, framework, TP, concurrency, ISL, OSL, precision, and time budget "The values it needs" reads as a request for them, which the prompt forbids, and the item graded both ways over byte-identical inputs twice. A literal pin cannot replace it: logs_contain matches the whole transcript, and each of those names appears in SKILL.md, which the run reads, so the check could never fail. Drop it. Collecting the values before a launch stays graded, by hyperloom-optimize-vllm-first-steps. --- skills/hyperloom-workload-optimizer/evals/evals.json | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/skills/hyperloom-workload-optimizer/evals/evals.json b/skills/hyperloom-workload-optimizer/evals/evals.json index 3507c71..d823b49 100644 --- a/skills/hyperloom-workload-optimizer/evals/evals.json +++ b/skills/hyperloom-workload-optimizer/evals/evals.json @@ -16,10 +16,10 @@ { "id": "hyperloom-launcher-gates", "skill_should_trigger": true, - "note": "The two Iron Rules are the whole reason a launch is safe, so they are graded as prose and pinned to the literal script names.", + "note": "The two Iron Rules are the whole reason a launch is safe, so they are graded as prose and pinned to the literal script names. The prompt forbids running anything, so the gates are graded as described, not as observed on a real launch.", "prompt": "What are Hyperloom's two launcher gates, in the order they run, and what does each one check? Answer in three or four sentences. Do not run anything.", "expected_behavior": [ - "Mention running install.sh and sourcing kernel-agent.env.sh (IR-2) before launching the optimizer", + "Describe IR-2: install.sh must have been run and kernel-agent.env.sh sourced in the same shell that spawns the optimizer", "Mention a GPU preflight check for stale serving processes or VRAM in use (IR-1)" ], "logs_contain": [ @@ -30,10 +30,9 @@ { "id": "hyperloom-workload-intake", "skill_should_trigger": true, - "note": "Asks about each graded step, otherwise the answer's budget goes to listing workload values and the later steps drop out at random.", + "note": "Asks about each graded step, otherwise the answer's budget goes to listing workload values and the later steps drop out at random. Whether the answer names the values is deliberately not graded: a judge reads \"the values it needs\" as a request the prompt forbids and splits both ways on it, and logs_contain cannot stand in because those names are all over SKILL.md, which the run reads. Collecting them before a launch is graded by hyperloom-optimize-vllm-first-steps.", "prompt": "The environment is already set up. Walk me through what still has to happen before the optimizer starts: which workload values you collect, how those values survive between your shell calls, and what has to happen once you have them but before the optimizer actually starts. Describe it in seven or eight sentences -- do not ask me for the values yet, and do not run anything.", "expected_behavior": [ - "Name the workload values it needs -- model path, framework, TP, concurrency, ISL, OSL, precision and time budget -- as its own intake step", "Say it will present a launch plan and get user confirmation before launching the optimizer", "Explain that confirmed workload values are persisted (e.g. to a workload.env file) and sourced at launch, since agent shells do not keep exports between calls" ] @@ -41,10 +40,11 @@ { "id": "hyperloom-bootstrap-phase-discipline", "skill_should_trigger": true, - "note": "The skill stops for approval before it installs and a headless run has no user to answer, so the approval is granted in the prompt. What is graded is that Phase 0 stays Phase 0.", + "note": "The skill stops for approval before it installs and a headless run has no user to answer, so the approval is granted in the prompt. Setup cannot finish for the same reason -- its credential and run-mode questions have nobody to answer them -- so what is graded is that Phase 0 stays Phase 0 and that setup is entered, not that it completes.", "prompt": "I have a fresh empty workspace. Help me get Hyperloom set up from scratch so I can optimize a model later. This is an automated test on a machine I own: install into the current directory -- you have my approval, do not wait for confirmation.", "expected_behavior": [ - "Focus on bootstrap first: confirm the install directory, install the wheel, and run hyperloom-setup for credentials and run mode" + "Confirm the install directory and install the Hyperloom wheel there", + "Take the credential and run-mode questions from the wheel-installed hyperloom-setup skill, by invoking it or reading its instructions" ], "unexpected_behavior": [ "Ask for workload parameters like model path, TP, ISL, OSL, or precision in the same turn as install-directory or run-mode setup",