Relocate the enablement subsystem and make ENABLEMENT the sixth phase - #1491
Open
ZhengGong-amd wants to merge 17 commits into
Open
Relocate the enablement subsystem and make ENABLEMENT the sixth phase#1491ZhengGong-amd wants to merge 17 commits into
ZhengGong-amd wants to merge 17 commits into
Conversation
The two report sub-trees were spelled by hand in five places. reports/bringup/ had no owner at all: bringup/persist.py and bringup/trees.py each appended the literal to reports_dir() independently, and breakdown/collectors/sessions.py bypassed enablement_dir() to spell the enablement setting-script path twice. Adds bringup_dir(), enablement_builds_dir() and enablement_stacks_dir() alongside the existing enablement_dir(), and exports BRINGUP_SEGMENT / ENABLEMENT_SEGMENT so session_package.py's glob patterns -- which cannot take a Path -- derive from the same source rather than repeating the string. enablement_stacks_dir() covers <session>/enablement/stacks/, a second root outside reports/ that nothing owned either. Co-authored-by: Cursor <cursoragent@cursor.com>
The classifier had no production consumer inside the agent runtime -- all of its call sites are in the orchestrator, two of them (bringup/ladder.py and framework/adapters.py) at module level. Living under agents/framework/ meant the orchestrator imported an agent's Python in-process, which makes the subprocess JSON contract that is supposed to be the API surface decorative. common/ is the only non-inverting home: placing it under enablement/ would force bringup and framework to import enablement, which imports both of them. The module has zero first-party imports, so it satisfies the common/ layering guard as-is, and FailureSignature belongs to the same vocabulary as the BootObservation and LadderStage already in common/bringup.py. All 15 failure-kind string values are unchanged -- they are persisted in session breakdowns. Co-authored-by: Cursor <cursoragent@cursor.com>
enablement_ops.py held the repo's only reverse dependency edge: a function-local import of orchestrator.framework.paths wrapped in a bare `except Exception: pass`, both of which existed only to hide the cycle orchestrator -> agents.framework.enablement_ops -> orchestrator.framework.paths. As enablement/mandate.py that becomes an ordinary top-level import and the guard goes with it. The narrower try/except around resolve_kernel_search_roots stays: the probe itself can fail on a host with no source trees, which is a different failure from the module being unimportable. _enablement_artifacts.py was a private module of phases/ whose only production consumer was the enablement lane. It becomes enablement/artifacts.py and loses the underscore. This also removes the enablement -> phases module-level edge. Both modules keep their module-level imports of agents.framework.keywords and repo_map. That is the legal orchestrator -> agents direction, and keywords.py is genuinely shared agent-side, so it must not move. Co-authored-by: Cursor <cursoragent@cursor.com>
Sixteen lines re-exporting one function and one constant. repo_url_for_framework now comes straight from agents.framework.repo_map, which is what enablement/params.py already did. DISCOVER_FAILURE_RETRY_LIMIT moves to phases/framework.py rather than to framework/artifacts.py: artifacts.py classifies candidate outcomes, and a discovery retry bound has nothing to do with that. Its only two production readers are in phases/framework.py, so it lands beside them and the module count drops by one instead of staying flat. Co-authored-by: Cursor <cursoragent@cursor.com>
bringup/argv_preflight.py reached framework/adapters.get_adapter() through a function-local import purely to call argv_parser_source(), which pinned the whole adapter registry -- venv creation, pip installs, ROCm probes -- at the bringup layer. That is what blocked the acquisition half from moving. The coupling turned out to be thinner than the class hierarchy suggests: argv_parser_source has exactly one caller in the repo and all four implementations are a bare `return "<source string>"` with no calls into the acquisition side. Extracting the strings into framework/adapter_parsers.py lets bringup import a 65-line lookup and nothing else, so no class hierarchy is split and no shared base module is needed. With that edge gone, adapters, stack_actions, localization, build_actions, build_utils and targeted_build move to enablement/runtime/ intact. The cluster depends on nothing in orchestrator/ outside itself, so the move adds no reverse edges. framework/ is left holding paths.py, artifacts.py and adapter_parsers.py. Two things that would have failed silently: - build_lifecycle._driver_command spawned `python -m hyperloom.orchestrator.framework.targeted_build`. A stale string there raises at build-spawn time, not import time, so it now derives the path from the module's own __name__. - test_git_foreign_checkout_callers reads a module's source by path to scan for unguarded git calls. Left pointing at the old path it would have scanned a 33-line shim and silently protected nothing. Also carries the attempt_root fix: enqueue_targeted_build pre-generates the task_id so it can fill action.attempt_root before the row is written. The executor's private _attempt_root() and the re-derive fallback in enablement/build.py both go away, and the params no longer keep the enqueue-time default they were documented as keeping. Co-authored-by: Cursor <cursoragent@cursor.com>
Twenty-six test files under inference_optimizer/tests/ imported only hyperloom.orchestrator.* and belonged in the packages they test. The style guide already asks for **/tests/ next to the code under test, and pyproject's src/**/tests glob collects the new directories with no config change. The relocation is not a bare file move, because a test taken out of inference_optimizer/tests/ silently loses that package's 466-line conftest -- including the autouse _isolate_session_layout_env, which clears the session-dir pin and points MULTI_NODE_STATE_FILE at a missing sentinel. So: - orchestrator/conftest.py takes _isolate_session_layout_env, launch_backend and virtual_clock. It sits at the orchestrator root rather than src/, whose scope would be every test package in the repo. - orchestrator/tests/_helpers.py takes init_git_repo, git_commit_all, patch_integrate_patch_roots and variant_result. These were pulled in through `from .conftest import`, which pytest does not make available across packages, so they need a real importable module. Every new tests/ directory gets an __init__.py. The repo is split on this today, but with importmode=prepend two same-named files in two non-package directories collide in sys.modules, and several of these basenames are generic enough to collide later. test_argv_refusal_round.py stays put: it imports fixtures from test_bringup_round_scenario.py, which is one of four files that genuinely depend on inference_optimizer.protocol / session / breakdown and belong where they are. Co-authored-by: Cursor <cursoragent@cursor.com>
enablement/__init__.py was four lines with no __all__. It now exports the three mandate names that callers outside the package address, and deliberately not the four CoordinatorCollaborator subclasses: those are resolved by Coordinator._COLLAB_MODULES through dotted strings, and exporting them would invite instantiation outside the coordinator. This is the shape bringup/__init__.py already uses. The guard follows kernelforge's test_rename_completeness.py: a git ls-files sweep, an allowlist whose every entry carries a written justification, and a self-validating test that fails when an entry stops exempting anything. It enforces two rules: - No pre-relocation dotted path may come back. The relocation left no shims, so any such reference is a live regression rather than a deprecation. - agents/framework/* must not import orchestrator/*. This is the rule that would have caught the enablement_ops reverse edge the day it landed. Scoping it to agents/framework rather than all of agents/ means it lands with an empty allowlist; the repo-wide version would have needed twenty entries, nearly all for agents/kernel/tools, and traded one real fix for a list nobody maintains. It also asserts the relocated modules import and that _driver_command's spawn argv still matches targeted_build's own __name__, since both are string-typed and fail late. adjustment.md records where the implementation departed from enablement-refactor-2.plan.md and why. Co-authored-by: Cursor <cursoragent@cursor.com>
Three fallbacks that could not fire. enqueue_targeted_build fills attempt_root, so the re-derive in _route_succeeded_build and TargetedBuildExecutor._attempt_root were unreachable; both are gone, and the enqueue now fills the field only when the caller left it unset instead of overwriting an explicit one. The broad `except Exception` around resolve_kernel_search_roots guarded nothing -- the function returns an empty tuple for "nothing here to search" rather than raising -- so it and the test that asserted the swallow are gone too. Seven docstrings described the move rather than the code: "Moved from", "Hoisted from", "previously reached these through". A reader of the module does not need its history, and the style guide asks for neither. orchestrator/conftest.py had copied four definitions that still existed in inference_optimizer/tests/conftest.py. They move to orchestrator/tests/_fixtures.py and both conftests import them, which is what registers a fixture; neither package sits under a shared ancestor conftest, and one at src/ would scope them to the whole repo. The relocation guard shrinks: the empty agents/framework allowlist drove a loop that could only ever match nothing, and two exemptions covered docstring lines this commit deletes. adjustment.md keeps the decisions and drops the plan-step narration. Also repoints a comment in common/provenance.py that named targeted_build by its old path. Co-authored-by: Cursor <cursoragent@cursor.com>
The phase chain is now: PRELUDE -> ENABLEMENT -> FRAMEWORK_AGENT -> KERNEL_AGENT -> SWEEP -> CLOSE ENABLEMENT is entered from PRELUDE when enablement is admitted (--enablement != off, not multi-node) and at least one baseline has failed (baseline_failure_streak >= 1). A healthy run never enters the phase. compute_next_phase gains three keyword arguments -- enablement_enabled, enablement_stalled (from RoundStore.consecutive_stalled), and enablement_in_flight -- passed by the already-async _advance_phase_if_needed. machine_state.py acquires no new first-party imports. Normal exit requires all three: baseline_tput > 0, no open revalidation window, and no queued/running enablement targeted_build or integrate_patch. The third conjunct is what makes the close-guard collapse sound: without it a build outliving the round would re-open validation_pending from inside FRAMEWORK_AGENT through _maybe_rearm_authored_lane. Terminal exits: server_argv_invalid, environment_fault (written by the lane's terminal helpers via stop_reason, routed by _global_terminal), and enablement_attempts_exhausted (consecutive_stalled >= ENABLEMENT_MAX_ATTEMPTS). The close guard is deleted: - enablement_close_guard_active() from shared_state.py - MAX_SKIP_TO_CLOSE_SUPPRESSIONS constant - skip_to_close_suppressions field from EnablementRound - the suppression block and observation in intent_router.py The drain exit condition makes validation_pending implies phase == ENABLEMENT a consequence of the machine, not of a separate predicate. Both the three-strike gate (baseline_failure_streak >= 3) and the combined backstop (_BASELINE_MAX_TOTAL_FAILURES = 3) in writeback.py are suppressed while phase == ENABLEMENT. Without this, the cap of 8 rounds is unreachable because 3 failures terminate the run first. The dead stop reason "enablement_stalled" is removed from STOP_REASON_VOCAB (it had no production writer). ENABLEMENT_MAX_ATTEMPTS moves from coordinator.py to machine_state.py, removing the import inversion through lane.py. Budget: ENABLEMENT 5%, FRAMEWORK_AGENT down from 40% to 38%, KERNEL_AGENT down from 50% to 47%. Sum stays 1.0; work >= 0.8. Allowlist: PRELUDE's set plus specialist, integrate_patch and targeted_build. targeted_build is added because cancel_queued_not_allowed fires on every transition and targeted_build was in no allowlist; it stays non-proposable because allowed_actions_for() subtracts COORDINATOR_INTERNAL_ACTIONS. Surfaces updated: _PHASE_ORIENTATION (critic gets an ENABLEMENT entry), _BASELINE_RECOVERY_PHASES (rules F1/F2 render in ENABLEMENT too), orchestration.md (phase-goal section, roofline tag), v6.py phase_map, attribution.py phase_buckets, render.py cycle_reloop set, CLI flag pair, failure_recovery.md reference tag, and all external docs. Three three-strike tests set enablement_mode="off" to keep exercising the documented fast-fail path. Six close-guard and suppression-counter tests are deleted; one is rewritten against the phase check the machine now owns. The SVG diagram is re-laid-out with a sixth box (dashed orange border for ENABLEMENT, indicating conditional entry); the PNG should be regenerated from the SVG. Co-authored-by: Cursor <cursoragent@cursor.com>
The ENABLEMENT branch of compute_next_phase re-derived three exits the lane already writes to stop_reason, which _global_terminal routes ahead of every phase branch — dead code, and a second place to keep the vocabulary in sync. Drop the branch's terminal arm and the enablement_stalled argument that fed it; the branch now decides only the normal exit. Fold the rest of the duplication the phase change introduced: - _enablement_work_in_flight re-implemented the lane's _enablement_in_flight and got the discriminator wrong: it looked for params["enablement"], which TargetedBuildAction.to_state never sets, so a running build read as drained. Call the lane's own query, and only from inside the phase, since it renews the round lease as a side effect. - The admission predicate lived in both the lane and the phase machine. It belongs to the lane; the phase machine reaches it through the collaborator registry. Hoisting it also absorbs the is_multi_node check that sat halfway down the pump, past work a multi-node host cannot use. - writeback derived in_enablement twice in one block; derive it once, next to the eval-suppression flag it is read beside. - _on_enter_enablement only logged, duplicating the ENTER lifecycle event the dispatcher already emits for every phase. enablement_stalled is no longer written by anything now that the cap stops with enablement_attempts_exhausted, so drop its report explanation and point the three tests that sampled it at the reason that is actually reachable. Add the enablement key to the PhaseBreakdown TypedDict so the schema again describes what the collector emits, and trim the comments that restated their own code or narrated the move. Co-authored-by: Cursor <cursoragent@cursor.com>
…rminal The lane helper was renamed in a prior commit; seven call sites in test_environment_fault_round.py still used the old name, causing an AttributeError at module collection time that silently dropped all 13 tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Neither enablement_entered nor enablement_done appeared in any test; enablement_enabled=True was never passed. Five focused cases now cover: - PRELUDE with a baseline_failure_streak routes to ENABLEMENT - the branch is skipped when enablement_enabled=False - ENABLEMENT exits (enablement_done) once tput is set and work is drained - the phase holds while work is in flight - the phase holds while validation_pending is set Co-authored-by: Cursor <cursoragent@cursor.com>
EnablementBreakdown reached the JSON output but was invisible in the Markdown report. The new renderer surfaces admission status, round outcomes, a bounded rounds table, and a build-attempts table. Skipped automatically when the enablement section is empty, so sessions that never triggered enablement are unaffected. Co-authored-by: Cursor <cursoragent@cursor.com>
- redistribute_budget_pct docstring: ENABLEMENT is excluded from absorbers alongside PRELUDE and CLOSE - _post_prelude_target docstring: called on ENABLEMENT exit too, not only PRELUDE - backfill_langfuse.py comment: add ENABLEMENT to the phase-span list - session-breakdown.md: note that a subset renders in the Markdown report Co-authored-by: Cursor <cursoragent@cursor.com>
Rasterized with resvg-py at 2x zoom (2800x1040) to match the SVG's aspect ratio. The prior PNG (1662x946) predated the ENABLEMENT node addition to the SVG source. Co-authored-by: Cursor <cursoragent@cursor.com>
…p commits The renderer read four field names no producer emits (attempts/opened_at on rounds, status/framework on builds), so half of every table rendered blank. Point the columns at the fields EnablementRoundSummary and TargetedBuildAttemptSummary actually carry. Also drop what was redundant: - unused warnings list threaded into RenderedSection - two near-identical table helpers collapsed into one - per-row isinstance skips, which silently dropped malformed rows that render_section already reports - a 10-row display cap duplicating the collector's _MAX_ROUNDS bound - str()/or-None coercions md_kv_list and _md_cell already handle - duplicated test state builders merged; a disjunctive assertion made exact and its explanatory comment removed Restore the return-value precedence dropped from the _post_prelude_target docstring and trim the session-breakdown note. Co-authored-by: Cursor <cursoragent@cursor.com>
.gitignore:70 lists pr.md; it is a local scratch file for authoring the PR body, not a repo artifact. Force-adding it in an earlier commit was a mistake. Co-authored-by: Cursor <cursoragent@cursor.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Comment on lines
+17
to
+22
| from hyperloom.orchestrator.tests._fixtures import ( # noqa: F401 | ||
| NoLaunchBackendInstalled, | ||
| _isolate_session_layout_env, | ||
| launch_backend, | ||
| virtual_clock, | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description: what and why
Two related changes to the enablement subsystem, landed together because the second depends on the module layout the first establishes.
Part 1 — module relocation (commits 1–8). Enablement code lived in six places across
agents/andorchestrator/. This collapses it to three owners, deletes the repo's only reverse dependency edge, and moves 26 test files next to the code they exercise.The reverse edge is the substantive part:
agents/framework/enablement_ops.pyreached back intoorchestrator.framework.pathsthrough a function-local import wrapped in a bareexcept Exception: passthat existed only to hide the cycle. Asorchestrator/enablement/mandate.pythat becomes an ordinary relative import and the guard disappears. Likewiseclassify_failurehad no consumer inside the agent runtime — all 22 call sites were in the orchestrator — so living inagents/made the orchestrator import agent Python in-process and rendered the subprocess JSON contract decorative. It moves tocommon/failure_signature.py.Two silent failures fixed in passing:
build_lifecycle._driver_commandspawnedpython -magainst a hardcoded module path (now derived fromtargeted_build.__name__), andtest_git_foreign_checkout_callerswas scanning a 33-line shim instead of the module it meant to guard.Part 2 — ENABLEMENT as the sixth phase (commits 9–17). The chain is now:
ENABLEMENT is entered from PRELUDE when the lane is admitted and
baseline_failure_streak >= 1. A run whose baseline boots first try never enters it —streak == 0, so PRELUDE goes straight to_post_prelude_target().compute_next_phasegains two keyword arguments (enablement_enabled,enablement_in_flight) supplied by the already-async_advance_phase_if_needed.machine_state.pygains no new first-party imports; the facts arrive through the same channelkernel_enabledandoptimize_enabledalready use.The normal exit requires three conjuncts:
baseline_tput > 0,not validation_pending, and no enablement work still in flight. The third is what makes the close-guard deletion sound —_maybe_rearm_authored_laneroutes on the result's lane, not the current phase, so a build outliving its round would otherwise reopenvalidation_pendingfrom insideFRAMEWORK_AGENT.This is a net reduction in mechanism.
enablement_close_guard_active(),MAX_SKIP_TO_CLOSE_SUPPRESSIONS, theskip_to_close_suppressionsfield, the intent-router suppression block, and the dead"enablement_stalled"stop reason are all deleted; the phase boundary now expresses what they approximated.Budget: ENABLEMENT 5%, FRAMEWORK_AGENT 40→38%, KERNEL_AGENT 50→47%. Sum stays 1.0.
Follow-ups (commits 13–17) close what a branch-wide sweep turned up: a test left broken by a method rename, zero coverage on the new phase predicate, and
EnablementBreakdownreaching the JSON but never the Markdown report.Linked issue(s): close/fix refs
None.
Tests: added/updated? commands run?
Both. Added: five cases covering the ENABLEMENT entry and exit predicate (neither
enablement_enterednorenablement_donewas exercised by anything, andenablement_enabled=Truehad never been passed);test_enablement_relocation_completeness.py, agit ls-filessweep with a self-validating allowlist, an importability check for all ten canonical module paths, and a spawn-path check for_driver_command.Fixed:
test_environment_fault_round.pybound a lane helper by its pre-rename name, raisingAttributeErrorat collection and silently dropping all 13 of its tests.Updated: constant assertions (
PHASE_NAMES, budget identity/sum,_PHASE_ORIENTATION, CLI redistribution figures,PHASE_GOAL_BLOCKS); three three-strike tests now setenablement_mode="off"to stay honest about the fast-fail path they assert. Six close-guard and suppression-counter tests deleted along with the mechanism.155 passed locally on the touched surfaces; leaving the full suite to CI.
Breaking changes: yes/no (details if yes)
No, for anything outside this repo. Internally, resume across the change is not supported: a session whose
state.jsonrecordsphase: "ENABLEMENT"cannot be resumed on pre-merge code. New sessions are unaffected, andEnablementRound.from_dictfilters unknown keys, so olderstate.jsonfiles load without migration.PR addresses single concern: yes/no (details if no)
No — two. The phase insertion needs the relocated modules to avoid re-creating the import cycle it deletes, so splitting them would mean landing the relocation, then immediately reopening the same files. They are cleanly separated in history (commits 1–8 and 9–17) and can be reviewed in that order.
Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed:
Not upstream — entirely in-repo.