KB warm replay does not compare the ROCm/AITER build a config was tuned on

[rpoornac]

What is verified

Every KB row carries a stack_fingerprint recording four components:

# src/hyperloom/common/provenance.py
_STACK_FINGERPRINT_ENVS: dict[str, tuple[str, ...]] = {
    "rocm": ("ROCM_VERSION", "HIP_VERSION"),
    "aiter": ("AITER_COMMIT", "AITER_VERSION"),
    "sglang": ("SGLANG_VERSION", "SGL_VERSION"),
    "vllm": ("VLLM_VERSION",),
}

Two of the four are already protected, though not by the fingerprint: framework_version is one of the seven canonical_id dimensions, so a row recorded under sglang 0.4.5 has a different identity than one under 0.5.11 and cannot be handed to the wrong pod.

rocm and aiter are written on every row and compared nowhere at read time. Every stack_fingerprint reference in orchestrator/knowledge/ is a write path — local_store.py puts it in the row, schema.py serialises and deserialises it, recipe_kb_t0.py accepts it as a parameter and forwards it to put_recipe. The one place it is read reads it for something else:

# src/hyperloom/orchestrator/knowledge/recipe_kb_t0.py:1018
fp: Mapping[str, Any] = stack_fingerprint if isinstance(stack_fingerprint, Mapping) else {}
# framework_version: SharedState > stack_fingerprint > importlib auto-detect.

So when PRELUDE auto-replays a warm best_config, nothing has established that the ROCm or AITER build it was tuned against is the one this pod is running.

--recipe-kb-strict-fingerprint nominally promised this and was declared in the parser while being read nowhere. It was removed in #1495, because the collision that PR addressed (compute-partition mode) is now handled in the canonical_id instead. This issue is the part of that flag's promise that was real and is still open.

Why #1495's fix does not generalise

#1495 put the partition mode into the hardware slug. That was cheap because SPX is the default and gets no suffix, so every historical key stayed byte-identical.

ROCm and AITER cannot go into the identity on those terms: every ROCm patch bump or AITER commit would mint a fresh canonical_id, so essentially no row would ever be reusable and the KB would stop working as a corpus. This wants a soft signal, not an identity split.

The two components need different rules

ROCm is a version and can very likely reuse the existing machinery almost verbatim. _framework_version_is_compatible already encodes a total order, a release line, and a direction:

# src/hyperloom/orchestrator/knowledge/recipe_kb_t0.py:674
"""Accept exact or nearest non-newer PEP 440 version in one release line."""
return bool(
    target_key and candidate_key
    and candidate_key.release[:2] == target_key.release[:2]
    and candidate_key <= target_key
)

AITER is heterogeneous, which is the harder half. stack_fingerprint["aiter"] can hold a commit SHA (when AITER_COMMIT is set), a bare distribution version (from the importlib probe), or the literal "unknown". A SHA has no order, no release line and no direction, so "non-newer" is unsayable; and a SHA is not comparable to a version at all. kernelforge's preflight already states the ranking between the two forms:

# src/kernelforge/gemm_tune/aiter_preflight.py:51
"AITER_COMMIT unset -> tuned-CSV provenance falls back to the installed "
"aiter distribution version (coarser than a commit)"

So AITER's comparison is three-valued rather than binary: equal (two SHAs or two versions that match), coarsely equal (two matching distribution versions, explicitly weaker evidence than a commit), and incomparable (a SHA against a version, or anything against "unknown").

Also worth noting: AITER_VERSION is the second element of the env tuple but nothing in the repo ever sets it, so the env path only ever yields a commit.

Proposal, for discussion

Three candidate actions, increasing in strength:

1. Disclose only — surface the delta into the specialist prompt alongside the row's lessons and let the model weigh it.
2. Reduce replay confidence — keep the row but lower config_confidence below the replay threshold.
3. Gate the auto-replay — allow priors, deny the config.

"unknown" on either side must mean no claim, never a disagreement.

One trap, learned in #1495

Demoting a tier does not deny a replay. This was defect #2 in the review of #1495: setting warm_tier/warm_conf to seed_only/0.0 leaves the config-donor block below it fully reachable, because it is entered on warm_point being truthy. _donor_is_trustworthy then compares only conc/isl/osl and never looks at the demoted dimensions, and _find_config_donor can return a row whose config_confidence clears the 0.7 default. The observed result was a guard that logged a refusal and replayed the config anyway.

Option 2 above is therefore not automatically safe. Any design here has to name which code path stops the config, not merely which field it lowers.

Open question I cannot answer from the code

Do real best_config payloads carry ROCm- or AITER-dependent settings in practice? That decides whether this is a disclosure nicety (option 1) or a correctness gate (option 3), and it is a domain call rather than something the code reveals.

cc @ZhengGong-amd @meinali-566 — flagging for the mechanism decision before any code exists, since #1495 showed a working, fully-tested implementation can still be the wrong mechanism.

[rpoornac]

Prerequisite opened as #1508: it only fixes the recorded value (AITER installed as amd-aiter was degrading to unknown), with no read-path change. Worth landing before any comparison is designed here, for the reason in the trap section — a check over a field that reads unknown on a whole class of hosts fires on every historical row.