Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 33 additions & 22 deletions skills/hyperloom-workload-optimizer/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,9 @@ Never copy API keys into chat output.
| MAX_HOURS | `--max-hours` | CLI `2.0` | offer `3` (quick) or `12` (full); see below |
| TARGET_GAIN | `--target-gain` | `30` | desired % gain |

**Optional:** `--no-explore`, `--no-enable-conc-sweep`, `--gpu-type`,
`--server-args`, `--compare-against-gpu`, `--quantize` prelude.
**Optional:** `--no-framework-agent`, `--no-kernel`, `--no-enable-conc-sweep`,
`--no-enable-roofline`, `--gpu-type`, `--server-args`, `--compare-against-gpu`,
`--quantize` prelude.

Infer `PRECISION` from the model name when obvious (e.g. an `FP8` model implies
`--precision fp8`) and confirm it — do not silently keep the `bf16` default.
Expand All @@ -209,24 +210,35 @@ Expect a modest validated gain, or an honest 0% when the workload has no
parameter headroom.

```text
--max-hours 3 --precision bf16
--no-framework-agent --no-kernel --no-enable-conc-sweep --no-enable-roofline
--max-minutes-explore-pct 0.39 --max-minutes-sweep-pct 0.01
--explore-force-exit-budget-pct 0.01 --explore-force-exit-hours-remaining 0.05
--max-hours 3 --precision bf16 --target-gain 30
--max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01
--no-kernel --no-enable-conc-sweep --no-enable-roofline
```

`0.50` is the share *before* redistribution. With `--no-kernel`, KERNEL_AGENT is
disabled and its freed share is added on top, so `0.50` becomes ~0.99 of the wall
clock for OPTIMIZE. Raising `0.50` buys almost nothing — the post-redistribution
share is capped at one full wall clock and the excess is discarded.

Do **not** pass `--no-framework-agent` here: it skips OPTIMIZE entirely, which is
the one phase this profile relies on. `--no-kernel` is what makes it "no kernel
rewrites".

**2. 12-hour demo** (`hyperloom-qwen3-14b-fp8-12h`) — `Qwen/Qwen3-14B-FP8`
unless the user names another model, TP=1, CONC=64, ISL=OSL=1024,
`--precision fp8` matched to the chosen checkpoint, every lever with kernel
rewrites included. The kernel agent needs room to profile, rewrite and
revalidate, which is where the larger gains come from.

```text
--max-hours 12 --precision fp8
--max-minutes-framework-pct 0.01 --max-minutes-explore-pct 0.42
--max-minutes-kernel-pct 0.42
--max-hours 12 --precision fp8 --target-gain 50
--max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42
```

Do not add `--max-minutes-explore-pct`: configuration search and source landing
are two arms of one phase with one budget, so that spelling is an alias onto
`--max-minutes-framework-pct` and silently overwrites it.

**3. Custom** — the user brings their own model or workload instead of taking a
demo. Walk through the fields in the table above and the phase toggles, one
question at a time, and derive the flags from the answers rather than asking for
Expand All @@ -250,13 +262,11 @@ Launch plan — please confirm:
TP=1 EP=1 CONC=64
ISL=1024 OSL=1024
PRECISION=fp8
MAX_HOURS=3 TARGET_GAIN=20%
profile 3-hour demo — no kernel, no framework agent, no roofline
flags --no-framework-agent --no-kernel --no-enable-conc-sweep
--no-enable-roofline
--max-minutes-explore-pct 0.39 --max-minutes-sweep-pct 0.01
--explore-force-exit-budget-pct 0.01
--explore-force-exit-hours-remaining 0.05
MAX_HOURS=3 TARGET_GAIN=30%
profile 3-hour demo — no kernel rewrites, no conc sweep, no roofline
flags --max-minutes-framework-pct 0.50
--max-minutes-sweep-pct 0.01
--no-kernel --no-enable-conc-sweep --no-enable-roofline
RUN_MODE baremetal
```

Expand Down Expand Up @@ -291,10 +301,10 @@ export ISL=1024
export OSL=1024
export PRECISION=fp8
export MAX_HOURS=3
export TARGET_GAIN=20
export TARGET_GAIN=30
# The whole flag set for the approved profile, space-separated. The 3-hour
# demo is shown; a 12-hour run swaps in its own set.
export OPT_FLAGS="--no-framework-agent --no-kernel --no-enable-conc-sweep --no-enable-roofline --max-minutes-explore-pct 0.39 --max-minutes-sweep-pct 0.01 --explore-force-exit-budget-pct 0.01 --explore-force-exit-hours-remaining 0.05"
export OPT_FLAGS="--max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01 --no-kernel --no-enable-conc-sweep --no-enable-roofline"
EOF
```

Expand Down Expand Up @@ -364,10 +374,11 @@ bash "${SKILL_DIR}/scripts/launch.sh"

Every workload value comes from the confirmed `workload.env`; the script has no
`${VAR:-default}` fallbacks, so a missing value fails loudly instead of launching
a different config. Put any optional Phase 2 flags (`--no-kernel`, `--no-explore`,
`--gpu-type`, `--model-class`, `--server-args`, `--compare-against-gpu`,
`--quantize`, phase budget flags) into `OPT_FLAGS` in `workload.env`. `OPT_FLAGS`
is word-split, so quote any flag value that contains spaces, e.g.
a different config. Put any optional Phase 2 flags (`--no-kernel`,
`--no-framework-agent`, `--gpu-type`, `--model-class`, `--server-args`,
`--compare-against-gpu`, `--quantize`, phase budget flags) into `OPT_FLAGS` in
`workload.env`. `OPT_FLAGS` is word-split, so quote any flag value that contains
spaces, e.g.
`export OPT_FLAGS='--server-args "--foo bar"'`.

### Launch health check (30 s after start)
Expand Down
12 changes: 6 additions & 6 deletions skills/hyperloom-workload-optimizer/evals/evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@
{
"id": "hyperloom-launcher-gates",
"skill_should_trigger": true,
"note": "The two Iron Rules are the whole reason a launch is safe, so they are graded as prose and pinned to the literal script names.",
"note": "The two Iron Rules are the whole reason a launch is safe, so they are graded as prose and pinned to the literal script names. The prompt forbids running anything, so the gates are graded as described, not as observed on a real launch.",
"prompt": "What are Hyperloom's two launcher gates, in the order they run, and what does each one check? Answer in three or four sentences. Do not run anything.",
"expected_behavior": [
"Mention running install.sh and sourcing kernel-agent.env.sh (IR-2) before launching the optimizer",
"Describe IR-2: install.sh must have been run and kernel-agent.env.sh sourced in the same shell that spawns the optimizer",
"Mention a GPU preflight check for stale serving processes or VRAM in use (IR-1)"
],
"logs_contain": [
Expand All @@ -30,21 +30,21 @@
{
"id": "hyperloom-workload-intake",
"skill_should_trigger": true,
"note": "Asks about each graded step, otherwise the answer's budget goes to listing workload values and the later steps drop out at random.",
"note": "Asks about each graded step, otherwise the answer's budget goes to listing workload values and the later steps drop out at random. Whether the answer names the values is deliberately not graded: a judge reads \"the values it needs\" as a request the prompt forbids and splits both ways on it, and logs_contain cannot stand in because those names are all over SKILL.md, which the run reads. Collecting them before a launch is graded by hyperloom-optimize-vllm-first-steps.",
"prompt": "The environment is already set up. Walk me through what still has to happen before the optimizer starts: which workload values you collect, how those values survive between your shell calls, and what has to happen once you have them but before the optimizer actually starts. Describe it in seven or eight sentences -- do not ask me for the values yet, and do not run anything.",
"expected_behavior": [
"Name the workload values it needs -- model path, framework, TP, concurrency, ISL, OSL, precision and time budget -- as its own intake step",
"Say it will present a launch plan and get user confirmation before launching the optimizer",
"Explain that confirmed workload values are persisted (e.g. to a workload.env file) and sourced at launch, since agent shells do not keep exports between calls"
]
},
{
"id": "hyperloom-bootstrap-phase-discipline",
"skill_should_trigger": true,
"note": "The skill stops for approval before it installs and a headless run has no user to answer, so the approval is granted in the prompt. What is graded is that Phase 0 stays Phase 0.",
"note": "The skill stops for approval before it installs and a headless run has no user to answer, so the approval is granted in the prompt. Setup cannot finish for the same reason -- its credential and run-mode questions have nobody to answer them -- so what is graded is that Phase 0 stays Phase 0 and that setup is entered, not that it completes.",
"prompt": "I have a fresh empty workspace. Help me get Hyperloom set up from scratch so I can optimize a model later. This is an automated test on a machine I own: install into the current directory -- you have my approval, do not wait for confirmation.",
"expected_behavior": [
"Focus on bootstrap first: confirm the install directory, install the wheel, and run hyperloom-setup for credentials and run mode"
"Confirm the install directory and install the Hyperloom wheel there",
"Take the credential and run-mode questions from the wheel-installed hyperloom-setup skill, by invoking it or reading its instructions"
],
"unexpected_behavior": [
"Ask for workload parameters like model path, TP, ISL, OSL, or precision in the same turn as install-directory or run-mode setup",
Expand Down
15 changes: 9 additions & 6 deletions skills/hyperloom-workload-optimizer/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,17 @@ Any failure → treat as a fresh launch and re-run `install.sh`.
`_preflight()` runs `preflight_kb.sh`. Exit `1` auto-enables `--degraded-kb` /
`--degraded-pr`; launch continues. IR-3 never aborts.

### IR-4 / IR-6 — EXPLORE contracts (Coordinator-internal)
### IR-4 — OPTIMIZE phase contracts (Coordinator-internal)

- **IR-4:** EXPLORE is specialist-informed; GPU specialists lease cards via
Configuration search and source landing are two arms of **one** phase
(OPTIMIZE / FRAMEWORK_AGENT) sharing one budget. The separate EXPLORE phase and
its `--explore-force-exit-*` flags no longer exist.

- **IR-4:** OPTIMIZE is specialist-informed; GPU specialists lease cards via
`gpu_research_lane` and must not touch production serving on port 8888.
- **IR-6:** EXPLORE force-exits when wall-clock remaining <
`--explore-force-exit-hours-remaining` (default 3 h) or phase budget <
`--explore-force-exit-budget-pct` (default 20%).
- Plateau signals are advisory; IR-6 and per-phase budgets are hard gates.
Specialists author patches into an isolated worktree; `integrate_patch` does
the `git apply` plus throughput/accuracy gate after Critic review.
- Plateau signals are advisory; the per-phase budgets are the hard gates.

### IR-8 — `--framework atom` is single-node only

Expand Down
11 changes: 5 additions & 6 deletions walkthroughs/hyperloom-workload-optimizer.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,9 +123,8 @@ below.
```text
Optimize Qwen/Qwen3-8B with <framework> on <gpu_type>: TP=1, conc=64, ISL=1024,
OSL=1024, precision bf16, target-gain 30, max-hours 3, serving parameters only:
--no-framework-agent --no-kernel --no-enable-conc-sweep --no-enable-roofline
--max-minutes-explore-pct 0.39 --max-minutes-sweep-pct 0.01
--explore-force-exit-budget-pct 0.01 --explore-force-exit-hours-remaining 0.05.
--max-minutes-framework-pct 0.50 --max-minutes-sweep-pct 0.01
--no-kernel --no-enable-conc-sweep --no-enable-roofline.
Launch and monitor.
```

Expand All @@ -135,10 +134,10 @@ larger gains come from.

```text
Optimize Qwen/Qwen3-14B-FP8 with <framework> on <gpu_type>: TP=1, conc=64,
ISL=1024, OSL=1024, precision fp8, target-gain 30, max-hours 12, all components
ISL=1024, OSL=1024, precision fp8, target-gain 50, max-hours 12, all components
enabled:
--max-minutes-framework-pct 0.01 --max-minutes-explore-pct 0.42
--max-minutes-kernel-pct 0.42. Launch and monitor.
--max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42.
Launch and monitor.
```

**3. Custom.** Ask for a run and let the agent take you through the choices:
Expand Down
Loading