fix(aiter): self-heal the compiled registry so a shipped CSV cannot fail every boot - #1532
Conversation
The fix is one function, not three layersThe failure is real, but all three commits are guessing at the same unknown, and the guess has an exact answer already pinned in this repo. The invariant
The only non-trivial term is "the CSV set this boot will load". Each commit guesses it differently:
It doesn't need guessing.
So the PR description's premise — "aiter merges What to builddef csvs_aiter_will_load(configs_dir: Path, tuned_file_name: str, value: str) -> list[Path]:
"""The CSV set this boot resolves to, by aiter's own two-branch rule."""
if value.strip():
return [Path(p) for p in value.split(":") if p.strip()]
overlays = sorted(
p for p in (configs_dir / "model_configs").glob(f"*{tuned_file_name}*.csv") if "untuned" not in p.name
)
return [configs_dir / tuned_file_name, *overlays]Feed its output to the existing
It also closes a silent hole the current preflight inherits: Concretely
Net effect should be one new function, one call site, and a pile of deletions. Three findings worth keeping regardless
The stated cost omits the remedy. "40s on a healthy tree" is the scan.
Smaller
|
A round that starts sglang boots against whatever get_config_file resolves each AITER_CONFIG_* to, and a kernel the compiled module never registered raises from inside graph capture. #1457 gave the GEMM integrate lane a coverage check before its own boots, but the baseline executor only learned to name the failure aiter_jit_registry_mismatch afterwards, with nothing to undo it -- and PRELUDE's first measurement and every FRAMEWORK variant boot through there. Which CSVs a boot loads is not a guess. aiter/jit/core.py::get_config_file has two branches: set, and it takes the ':'-joined paths verbatim with no shipped default and no overlay discovery; unset, and it merges configs/model_configs/*{tuned_file_name}*.csv (minus untuned) on top of the shipped default. A check keyed on the CSVs a round names therefore cannot see the overlay an unset variable pulls in, which is exactly how fmoe_ck -- setting only AITER_CONFIG_FMOE -- took the unset branch for bpreshuffle, merged the dsv3 overlay and failed both integrate attempts in 20260915T083736Z-36368bab on a kernel it had never tuned. Resolve each variable through that rule and feed the result to the existing coverage check. A round that pins a table gets only what it will read; one that pins nothing gets the overlays checked, which is the only way PRELUDE is ever protected. It also closes the hole where Path("a.csv:b.csv").is_file() is False, so a ':'-joined env was silently treated as covered. csv_jit_kernel_rows now also survives the tables it does not own: the shipped CSVs can carry a BOM (UnicodeDecodeError) or an overlong field (csv.Error), and either escaping would skip the whole check.
aafd95c to
0ff4b9e
Compare
…s at KERNEL entry aiter merges configs/ and configs/model_configs/ at import, so a table shipped for another model -- DeepSeek, GLM -- naming a kernel this install never compiled fails every boot in the session. The three env-keyed checks cannot see it: they compare only the CSVs a round names, and no round names a shipped table. On this install 6 of 142 shipped CSVs outran 5 modules, which is how fmoe_ck failed both integrate attempts in 20260915T083736Z-36368bab on a bpreshuffle kernel it had not tuned, and why the same class of failure kept returning after being fixed twice. Audit once at phase entry, where it is an install-level fact rather than a lane's: resolve every shipped CSV's kernels by name, unlink the modules that do not provide them, and let the next boot rebuild. 40s on a healthy tree, and every lane that follows inherits a consistent one.
…env's A round tuning one CSV still boots against every CSV aiter merges, so the kernel a compiled-registry miss names can belong to a variable the round never set -- and AITER_CONFIG_FMOE maps to no serving module at all. The env-keyed drop then unlinks nothing and the one retry repeats the failure verbatim, which is how fmoe_ck failed both attempts in session 20260915T083736Z-36368bab on a bpreshuffle kernel it had not tuned, leaving GEMM with no e2e verdict for any tuner. Read the module out of the kernel the error names and unlink it alongside the env's own. The error text is the only thing that identifies the module when the env cannot.
haishuok0525
left a comment
haishuok0525
left a comment
There was a problem hiding this comment.
What this PR does
Boots no longer walk into aiter tuned CSVs whose kernelNames are missing from the host compiled module_*.so.
- Resolve CSVs the way
get_config_filedoes (pinned:-list vs unset → shipped default + model overlays) and runprepare_serving_so_for_csvsfrom the baseline executor before materialize. - On integrate
aiter_jit_registry_mismatch, also drop modules named in the error text (registry_mismatch_modules), not only env-mapped ones. - Once at Forge GEMM KERNEL entry, audit shipped
configs/+model_configs/and unlink uncovered GEMM modules.
Checked: _aiter_jit.py resolution vs aiter's set/unset branches, baseline preflight call site, integrate retry also_modules, KERNEL audit placement, new preflight tests. Description matches the final three-commit diff.
Blocking issues
…e-aiter-registry-selfheal # Conflicts: # CHANGELOG.md
What this PR doesStops a boot from loading an aiter tuned CSV whose Checked: Blocking issues
|
The audit globbed every shipped configs/*.csv and configs/model_configs/*.csv, on the premise that aiter merges them all at import. csvs_aiter_will_load, added by the first commit, disproves that: a pinned AITER_CONFIG_* is taken verbatim and an unset one resolves to the shipped default plus only its matching overlays. So the audit checked tables no boot in the session can resolve to, and any one of them being uncovered unlinked modules and moved the whole jit/build aside -- a full aiter recompile for every lane that follows, on a healthy install: 6 of 142 shipped CSVs outrun 5 modules on this image. It could not deliver the stated effect either. The call sat below the `if geak_enabled: ... return` branch in _on_enter_kernel, and GEAK is the default backend, so "audit once at KERNEL entry so every lane inherits a consistent tree" only ever applied to the forge opt-in. Deleted: audit_serving_so_against_aiter_configs, _aiter_config_csvs, _audit_aiter_serving_so, its call site, and the three tests that covered them. The boot-time check the first commit added is where this belongs -- it runs ahead of materialize on every lane, and resolves the CSV set by aiter's own rule rather than by globbing. The integrate-side comment still said a round tuning one CSV "boots against every CSV aiter merges". It now states the rule the code implements.
…e-aiter-registry-selfheal # Conflicts: # CHANGELOG.md
xiaofei-zheng
left a comment
xiaofei-zheng
left a comment
There was a problem hiding this comment.
Re-reviewed at 616d21db1
All five blocking points from the previous round are addressed.
The KERNEL-entry audit is gone (d701c2b1). audit_serving_so_against_aiter_configs, _aiter_config_csvs, _audit_aiter_serving_so, its call site and the three tests that covered it are deleted -- grep confirms no reference survives, and phases/kernel.py has dropped out of the diff entirely. That removes both the destructive behaviour (a shipped table no boot can resolve to moving the whole jit/build aside, on an install where 6 of 142 CSVs outrun 5 modules) and the dead-on-default-backend call site under if geak_enabled: ... return. The boot-time check that remains resolves a reachable CSV set, so an unlink converges instead of recompiling on every entry.
The disproved premise is out of the prose. The PR description is rewritten around get_config_file's two branches and states plainly why the audit was dropped. The integrate-side comment (request_handlers.py:5826-5831) now describes the unset-branch rule the code implements instead of "boots against every CSV aiter merges". The two docstrings carrying the old claim went with the deleted functions.
Fix bullet 1 no longer claims a no-op; the description now says the absent early return is deliberate and names PRELUDE as the case that needs it.
CHANGELOG matches the final diff: two-branch resolution plus the integrate-side drop, no audit.
Checked this round: csvs_aiter_will_load (_aiter_jit.py:659-684) against config_files_and_merge.md:40-66, AITER_ENV_TO_TUNED_FILE vs AITER_ENV_TO_SERVING_MODULES coverage, the _run_once ordering ahead of materialize_config_with_envs, registry_mismatch_modules / also_modules on the integrate retry, and the deletion being clean. CI green (30 success, 2 skipped).
No blocking issues. LGTM.
4 participants
Problem
aiter resolves a tuned table two ways, and
jit/core.py::get_config_filehasexactly these two branches:
:-joined paths. The shipped default is notprepended and model overlays are not discovered.
configs/model_configs/*{tuned_file_name}*.csv(excludinguntuned) merged on top of the shipped default.A round tunes one operator, so it sets only that operator's
AITER_CONFIG_*.Every other variable is therefore unset and takes the second branch, pulling in
overlays cut on another host -- tables whose
kernelNames are absent from thismachine's compiled
module_*.so.fmoe_cksets onlyAITER_CONFIG_FMOE, sobpreshuffle takes the unset branch and merges the dsv3 overlay.
Serving then aborts at load with a registry mismatch. Because the offending
table is shipped rather than produced by the run, every retry hits the same
wall: the session fails at boot with nothing to roll back.
Fix
Resolve the CSV set a boot will actually load, by aiter's own rule.
csvs_aiter_will_loadimplements the two branches above, andprepare_serving_so_for_csvsconsumes its expanded list. It runs from thebaseline executor ahead of materialize, on every lane. An uncovered module is
unlinked so the next boot rebuilds it.
This deliberately has no early return for a round with no
AITER_CONFIG_*set:no env is exactly the case that needs the unset-branch check, and it is the
case PRELUDE boots in. A
:-joined env is also now expanded rather thanPath(...).is_file()-tested, which had silently read a multi-path value ascovered.
On integrate, a registry mismatch also drops the modules the error names.
The env does not always reach the module at fault --
AITER_CONFIG_FMOEmapsto no serving module at all -- so the module is read out of the kernel the
error named.
An earlier revision also audited every shipped CSV once at KERNEL entry. That
is removed: it rested on the premise this PR's own code disproves, it scanned
tables no boot can resolve to, and one uncovered table there moved the whole
jit/buildaside -- a full recompile for every following lane, on a healthyinstall. It also sat below the
if geak_enabled: ... returnbranch, so itnever ran on the default backend.
Test plan
test_baseline_aiter_registry_preflight.py,test_kernel_integrate_and_report.py(123 passed)src/hyperloom/orchestrator/(1295 passed, 1 skipped -- the skip needshypothesis, absent on this box)ruff check .+ruff format --check .on the changed files