fix(hyperloom): align demo flag sets with the current optimizer CLI - #207
Merged
Merged
Conversation
The hyperloom-workload-optimizer skill has not been updated since it landed (a06c973, 2026-08-14). Upstream has since merged EXPLORE into FRAMEWORK and removed the explore-only flags, so the published 3-hour demo no longer runs: error: unrecognized arguments: --explore-force-exit-budget-pct 0.01 --explore-force-exit-hours-remaining 0.05 argparse rejects unknown flags, so anyone following the skill or the walkthrough hits exit 2 at launch. `--no-explore` is gone the same way. The 12-hour set parsed but was silently wrong: `--max-minutes-explore-pct` is now an alias onto the same dest as `--max-minutes-framework-pct`, so the pair `--max-minutes-framework-pct 0.01 --max-minutes-explore-pct 0.42` resolved to 0.42, not 0.01. Adopt the flag sets the Hyperloom demo skills carry today (AMD-AGI/Hyperloom #1417) and explain the two things that were easy to get wrong: 0.50 is the share *before* redistribution (--no-kernel adds the kernel agent's freed share on top, capped at one wall clock), and --no-framework-agent must not be used to mean "no kernel rewrites" since it skips OPTIMIZE entirely. Also refresh reference.md's IR-4/IR-6 section, which documented the deleted force-exit flags as live defaults. Every recommended flag set is verified to parse against the current CLI parser.
lishuoshuo-amd
temporarily deployed
to
behavioral-instinct
September 9, 2026 07:19 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
lishuoshuo-amd
had a problem deploying
to
behavioral-instinct
September 9, 2026 07:19 — with
GitHub Actions
Failure
GitHub Actions
Failure
lishuoshuo-amd
had a problem deploying
to
behavioral-instinct
September 9, 2026 07:50 — with
GitHub Actions
Failure
GitHub Actions
Failure
lishuoshuo-amd
had a problem deploying
to
behavioral-instinct
September 9, 2026 08:09 — with
GitHub Actions
Failure
GitHub Actions
Failure
lishuoshuo-amd
force-pushed
the
fix/hyperloom-demo-flags-align-1417
branch
from
September 9, 2026 08:42
6ed0fe3 to
8a028eb
Compare
lishuoshuo-amd
had a problem deploying
to
behavioral-instinct
September 9, 2026 08:42 — with
GitHub Actions
Error
GitHub Actions
Error
lishuoshuo-amd
had a problem deploying
to
behavioral-instinct
September 9, 2026 08:42 — with
GitHub Actions
Failure
GitHub Actions
Failure
lishuoshuo-amd
had a problem deploying
to
behavioral-instinct
September 9, 2026 08:54 — with
GitHub Actions
Failure
GitHub Actions
Failure
These behavioral cases were 14/14 on the Instinct runner in August (#140, #141) and the skill has not changed since. Four runs on the current model each failed a different single expectation, so fix the three that ask for something the case itself rules out. `hyperloom-bootstrap-phase-discipline` required the agent to "run hyperloom-setup for credentials and run mode". Setup cannot finish in a single-turn headless run -- its credential and run-mode questions have nobody to answer them -- so the agent correctly stops in Phase 0 and the judge reads that as setup never having run: The agent confirmed the install directory and installed the wheel but did not run hyperloom-setup; no .env exists and it paused to ask the user for run mode and credentials instead. It did enter setup. That run asked for base URL, CLAUDE_MODEL and USER_DATA_PATH, none of which this skill's own prose names, so they came from the wheel-installed setup skill. Grade installing the wheel and entering setup, which a headless run can reach. `hyperloom-launcher-gates` asks for three or four sentences and forbids running anything, yet required install.sh and kernel-agent.env.sh be mentioned "before launching the optimizer". Read as an ordering of real events that is unreachable, and the judge read it that way: The agent described IR-2 conceptually but never launched the optimizer, so there was no pre-launch mention of running install.sh before an actual launch. State the gate's content instead, which is what the answer already carries. `hyperloom-workload-intake` required the workload values be named "as its own intake step". Dropping that clause did not settle it -- the judge then failed the bare wording while conceding the values were all present: did not itself request or name the specific workload values needed from the user in the required actionable sense -- though it did list model path, framework, TP, concurrency, ISL, OSL, precision, and time budget "The values it needs" reads as a request for them, which the prompt forbids, and the item graded both ways over byte-identical inputs twice. A literal pin cannot replace it: logs_contain matches the whole transcript, and each of those names appears in SKILL.md, which the run reads, so the check could never fail. Drop it. Collecting the values before a launch stays graded, by hyperloom-optimize-vllm-first-steps.
lishuoshuo-amd
force-pushed
the
fix/hyperloom-demo-flags-align-1417
branch
from
September 9, 2026 09:14
57c0dc3 to
746ab92
Compare
Collaborator
Author
|
hyperloom-workload-optimizer is green on the Instinct runner (4/4 cases, 14/14 expectations); the remaining red legs belong to skills this PR does not touch, with lemonade on Windows already red here before this branch had any eval change. |
danielholanda
approved these changes
Sep 9, 2026
lishuoshuo-amd
added a commit
that referenced
this pull request
Sep 16, 2026
The demo prompts carried the full flag set, so a CLI change meant a PR here even once the skill itself is federated -- which is what #207 was. The flags live in the demo skills the wheel installs, so the prompts now name the demo and let those skills answer for the values. Same reason drops the module path and install.sh from the launch description and the Step 2 check: they are the wheel's to move. A model path is worth spelling out, though, since it is the one value a user brings, so it joins <framework> and <gpu_type> as a placeholder. What stays is the artifact paths under USER_DATA_PATH. Those are what the run hands back to the user, not internals. Co-authored-by: Cursor <cursoragent@cursor.com>
lishuoshuo-amd
added a commit
that referenced
this pull request
Sep 16, 2026
The demo prompts carried the full flag set, so a CLI change meant a PR here even once the skill itself is federated -- which is what #207 was. The flags live in the demo skills the wheel installs, so the prompts now name the demo and let those skills answer for the values. Same reason drops the module path and install.sh from the launch description and the Step 2 check: they are the wheel's to move. A model path is worth spelling out, though, since it is the one value a user brings, so it joins <framework> and <gpu_type> as a placeholder. What stays is the artifact paths under USER_DATA_PATH. Those are what the run hands back to the user, not internals.
lishuoshuo-amd
added a commit
that referenced
this pull request
Sep 16, 2026
The expectation failed and passed on byte-identical inputs: the case passed at 06:54, then failed at 09:47 with only walkthrough commits in between, which a behavioral run never reads. #207 recorded the same flap on this case for a different wording, so this is the case rather than the wording -- its prompt forbids running anything, and the judge then reads a described launch plan as a plan that was never presented. Confirming the plan before a launch is still graded, by the two cases a run actually reaches: hyperloom-optimize-vllm-first-steps and hyperloom-bootstrap-phase-discipline both assert the optimizer does not start before it, and both pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The hyperloom-workload-optimizer skill has not been updated since it landed (a06c973, 2026-08-14). Upstream has since merged EXPLORE into FRAMEWORK and removed the explore-only flags, so the published 3-hour demo no longer runs:
error: unrecognized arguments: --explore-force-exit-budget-pct 0.01
--explore-force-exit-hours-remaining 0.05
argparse rejects unknown flags, so anyone following the skill or the walkthrough hits exit 2 at launch.
--no-exploreis gone the same way.The 12-hour set parsed but was silently wrong:
--max-minutes-explore-pctis now an alias onto the same dest as--max-minutes-framework-pct, so the pair--max-minutes-framework-pct 0.01 --max-minutes-explore-pct 0.42resolved to 0.42, not 0.01.Adopt the flag sets the Hyperloom demo skills carry today (AMD-AGI/Hyperloom #1417) and explain the two things that were easy to get wrong: 0.50 is the share before redistribution (--no-kernel adds the kernel agent's freed share on top, capped at one wall clock), and --no-framework-agent must not be used to mean "no kernel rewrites" since it skips OPTIMIZE entirely.
Also refresh reference.md's IR-4/IR-6 section, which documented the deleted force-exit flags as live defaults.
Every recommended flag set is verified to parse against the current CLI parser.