feat(keep): --max-latency-ms as a constraint on every promotion

CHANGELOG.md

@@ -766,6 +766,48 @@ for the user-facing summary.
 
 ### Added
 
+- **`--max-latency-ms` makes a latency SLA a constraint on every KEEP.**
+  The optimizer maximized `output_throughput` and nothing else. Latency was
+  measured, reported and fed to the prompts, but no latency number could block a
+  promotion. That is survivable for a lever that raises throughput without
+  touching per-request latency, and unsafe for any lever that raises throughput
+  *by* making each stream slower: against a throughput-only gate such a lever
+  does not merely tolerate a latency regression, it selects for the largest one
+  on offer. Measured on one MI355X, splitting the card eight ways at two streams
+  each bought about 20% aggregate throughput while mean end-to-end latency went
+  from 183 ms to 1211 ms — a 6.6x regression that the session would have signed
+  off as a win.
+
+  The ceiling rides the verdict the gain gates already decide rather than
+  arriving as a second opinion beside them: `graded_comparison` marks an
+  over-budget candidate `REVERT` and names it in a new
+  `GradedComparison.veto_reason`, so every consumer of the verdict — the
+  promotion choke point, explore's round ladder, the kernel stack — honours it
+  without a lane-by-lane check. Explore therefore refuses in the round that
+  measured the variant, which keeps an over-budget config from being folded onto
+  the stack and becoming the anchor the rest of the batch is graded against.
+
+  It fails closed: a candidate that reported no end-to-end latency is refused,
+  since a constraint nobody measured is not one anybody satisfied. That makes
+  latency part of the promotion contract, so each lane copies `e2el_mean_ms`
+  from its `VariantResult` onto the dict it promotes — a lane that does not is
+  that lane's bug, not a spelling the lookup should learn. It fails closed at
+  the boundary too: a baseline already over the ceiling stops the run with
+  `baseline_over_latency_budget` instead of spending the whole `--max-hours`
+  refusing every candidate to discover what was knowable at launch, which is
+  also why no `current_best` can end a session over budget.
+
+  The budget has one home, `SharedState.latency_budget_ms`, written at launch
+  and archived with the session so a resume restores it. Parse failures happen
+  only at the CLI, which exits 2: the switch on a fail-closed gate must not
+  itself fail open, and a value like `--max-latency-ms 200ms` silently resolving
+  to "no budget" would leave an operator believing an SLA was enforced.
+  Refusals are recorded on `SharedState.latency_refusals` and rendered as
+  `=== Latency budget (constraint) ===`, because a constrained session that ends
+  near baseline is otherwise indistinguishable from one that found no headroom,
+  and the two call for opposite responses. Off by default; KEEP behaviour is
+  exactly as it was when unset.
+
 - **Session breakdown exports now include the additive V6 startup contract.**
   The existing V5 payload remains intact while `metadata`, `outcome`,
   `timeline`, and `close` provide the V6 read model. Install and model-gate

docs/conceptual/optimization-loop.md

@@ -181,6 +181,28 @@ authoring specialist's prompt. The rungs, in increasing complexity:
    environment variable (see
    [Targeted builds (Rung 5)](../reference/environment-variables.md#targeted-builds-rung-5)).
 
+### Latency budget (constraint on KEEP)
+
+`--max-latency-ms` sets a ceiling on mean end-to-end latency. It is a
+constraint rather than an objective: it does not decide when the run stops,
+only which winners are admissible, so it composes with whichever `--target-*`
+is in use. It rides the same verdict the gain gates decide — a candidate that
+clears its objective and breaks the ceiling is a REVERT, carrying a
+`veto_reason` that distinguishes it from one that simply did not gain.
+
+The constraint exists because a throughput-only comparison does not merely
+tolerate a latency-for-throughput trade, it selects for the worst one on
+offer: facing a lever that raises aggregate throughput *by* making each stream
+slower, the largest regression is where the most throughput is.
+
+It fails closed. A candidate that reported no end-to-end latency is refused,
+since a constraint nobody measured is not one anybody satisfied — which is why
+every lane copies `e2el_mean_ms` onto the dict it promotes. It fails closed at
+the boundary too: if the baseline itself exceeds the ceiling, the run stops
+with `baseline_over_latency_budget` rather than spending its whole budget
+refusing every candidate to learn what was knowable at launch. Off by default,
+leaving KEEP behaviour unchanged when unset.
+
 ### Runnable gate (earned KEEP)
 
 A verified build does not KEEP on artifact verification alone. After a

src/hyperloom/common/perf_metric.py

@@ -7,6 +7,7 @@
 
 import os
 from dataclasses import dataclass
+from math import isfinite
 from typing import Any, Mapping
 
 from hyperloom.common.env import env_bool, env_str
@@ -205,6 +206,28 @@ def passes_intvty_gate(
     return _within_band(intvty_of(candidate), intvty_of(anchor), band)
 
 
+def latency_veto_reason(observed_ms: Any, budget_ms: float) -> str:
+    """Why the latency budget refuses this candidate, or "" when it does not.
+
+    The budget is a ceiling on mean end-to-end latency, so unlike the gain gates
+    it refuses a candidate whose throughput won: a lever that buys throughput by
+    making each stream slower is exactly the case a throughput-only comparison
+    selects for. Off entirely when *budget_ms* is not positive.
+
+    Fails closed on an unmeasured candidate — a constraint nobody measured is not
+    one anybody satisfied — which is why every lane copies ``e2el_mean_ms`` onto
+    the dict it promotes.
+    """
+    if not budget_ms or budget_ms <= 0:
+        return ""
+    if isinstance(observed_ms, bool) or not isinstance(observed_ms, (int, float)):
+        return "latency_unmeasured"
+    observed = float(observed_ms)
+    if not isfinite(observed) or observed <= 0:
+        return "latency_unmeasured"
+    return "latency_budget_exceeded" if observed > float(budget_ms) else ""
+
+
 def passes_tput_guard(
     candidate: Mapping[str, float],
     anchor: Mapping[str, float],
@@ -224,6 +247,7 @@ class GradedComparison:
 
     ``candidate`` and ``reference`` are both read on ``objective``. ``tput_*`` carry the guard axis and are 0.0 off
     AgentX. ``degrade_reason`` names why the interactivity axis did not apply on a session that asked for it.
+    ``veto_reason`` names a constraint that refused a candidate its throughput would otherwise have kept.
     """
 
     objective: str
@@ -233,6 +257,7 @@ class GradedComparison:
     tput_candidate: float = 0.0
     tput_reference: float = 0.0
     degrade_reason: str = ""
+    veto_reason: str = ""
 
     @property
     def comparable(self) -> bool:
@@ -268,6 +293,7 @@ def graded_on_intvty(self) -> bool:
     "intvty_of",
     "intvty_serving_grading_enabled",
     "is_agentx_mode",
+    "latency_veto_reason",
     "output_tput_of",
     "parse_intvty_noise_pct",
     "passes_intvty_gate",

src/hyperloom/inference_optimizer/SKILL.md

@@ -657,6 +657,7 @@ and the operator's stated value is lost:
 | Expert parallel | `--ep` | Pass the prompt's EP for MoE. Default `1`. |
 | Precision | `--precision` | Match the checkpoint (`bf16` default / `fp8` / ...). Keep consistent with `--quantize`. |
 | Budget | `--max-hours` | Pass the prompt's time budget. Default `2.0`. |
+| Latency SLA | `--max-latency-ms` | Pass any stated ceiling on per-request latency ("must stay under 250 ms", "interactive workload"). A **constraint, not a target**: it composes with `--target-*` rather than competing, and refuses any KEEP whose mean end-to-end latency exceeds it — including one that reported no latency at all. Off when omitted, which does not lose a preference but does remove the SLA from the search. |
 | Max model len | `--max-model-len` | Optional; auto-derived from ISL+OSL+headroom when omitted. |
 | External reference GPU | `--compare-against-gpu` | `target_analysis` writes `target_analysis/target_baseline.json` for query/status metadata and `competitor_target.json` for both advisory and final-report comparisons. Without a target GPU it writes `reason="no_target_gpu_configured"` and clears the competitor target. AgentX reads accepted `current_best.total_throughput / state.tp` and `current_best.e2e_norm_intvty_p90` at `state.conc`; it does not reread raw results or recipes. Missing targets or axes remain unavailable. This is a cross-system advisory, not proof of identical measurement estimators or deployment, and never changes Objective or KEEP/REVERT. |
 | Target advisory | `--no-target-advisory` | Disable external-target hints in prompts without disabling final-report comparison. `primary_gap` uses the existing latency/throughput categories; the interactivity axis is displayed as interactivity. |

src/hyperloom/inference_optimizer/cli/bootstrap.py

@@ -285,6 +285,9 @@ def _resolve_framework_version(args_in: Any) -> str:
         # config.json structural summary, persisted for downstream collectors.
         model_info=summarize_model_config(str(args.model)),
         framework=os.environ.get("FRAMEWORK", "sglang"),
+        # The only copy of the budget. Validated at the CLI, so anything that reaches here is usable, and archived
+        # with the session so a resume restores it without a second source to reconcile.
+        latency_budget_ms=float(getattr(args, "max_latency_ms", None) or 0.0),
         gpu_type=str(getattr(args, "gpu_type", None) or os.environ.get("GPU_TYPE", "")),
         # Workload metadata mirrored from CLI/env.
         tp=_int_arg("tp", DEFAULT_TP),

src/hyperloom/inference_optimizer/cli/parser.py

@@ -7,6 +7,7 @@
 
 import argparse
 import os
+from math import isfinite
 from pathlib import Path
 from typing import NoReturn
 
@@ -106,6 +107,22 @@ def _positive_int_arg(value: str) -> int:
     return parsed
 
 
+def _positive_ms_arg(value: str) -> float:
+    """argparse type for a millisecond ceiling.
+
+    The gate this feeds fails closed, so its switch must not fail open: an
+    unusable value has to stop the launch rather than resolve to "no budget" and
+    leave the operator believing an SLA is enforced.
+    """
+    try:
+        parsed = float(str(value).strip())
+    except (TypeError, ValueError) as exc:
+        raise argparse.ArgumentTypeError(f"expected a positive number of milliseconds, got {value!r}") from exc
+    if not isfinite(parsed) or parsed <= 0:
+        raise argparse.ArgumentTypeError(f"expected a positive number of milliseconds, got {value!r}")
+    return parsed
+
+
 def _default_claude_model_env() -> str:
     """Resolve the default Claude model from env."""
     explicit = (os.environ.get("CLAUDE_MODEL") or "").strip()
@@ -563,6 +580,19 @@ def _build_parser() -> argparse.ArgumentParser:
             "independently of --target-gain / --target-tput / --target-baseline-dir."
         ),
     )
+    # Outside the group as well, and for a stronger reason than --target-roofline: this is a constraint rather than
+    # an objective. It does not say when to stop, it says which winners are admissible, so it composes with whichever
+    # target is in use instead of competing with one.
+    opt.add_argument(
+        "--max-latency-ms",
+        type=_positive_ms_arg,
+        default=None,
+        help=(
+            "Refuse any KEEP whose mean end-to-end latency exceeds N ms. Off by "
+            "default. A candidate that reported no end-to-end latency is refused "
+            "too, since an unmeasured constraint is not a satisfied one."
+        ),
+    )
     opt.add_argument(
         "--resume-from",
         type=str,

src/hyperloom/inference_optimizer/tests/test_explore_executor.py

@@ -2718,3 +2718,73 @@ def fake_kill():
 
     assert result["status"] == "failed"
     assert kill_calls["n"] == 0, "must be a no-op while PYTEST_CURRENT_TEST is set"
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "budget_ms,expected_outcome,expected_reason",
+    [
+        (250.0, "REVERT", "latency_budget_exceeded"),
+        (5000.0, "KEEP", ""),
+        (0.0, "KEEP", ""),
+    ],
+)
+async def test_explore_refuses_an_over_budget_winner_in_the_round_that_measured_it(
+    sub_agent_runner, tmp_path, monkeypatch, budget_ms, expected_outcome, expected_reason
+):
+    """``--max-latency-ms`` rides the verdict explore's ladder already reads.
+
+    The variant gains throughput either way; only the SLA separates the cases.
+    Refusing here rather than at promotion keeps an over-budget variant from
+    being folded onto the stack and becoming the anchor the rest of the batch is
+    graded against, and gives the ledger a latency reason for the REVERT.
+    """
+    _force_cold_decision(monkeypatch)
+    monkeypatch.delenv("HYPERLOOM_AGENTX", raising=False)
+    monkeypatch.delenv("HYPERLOOM_PERF_METRIC", raising=False)
+    sub, tr, _ = sub_agent_runner
+    state = SharedState(framework="sglang")
+    state.baseline_tput = 200.0
+    state.latency_budget_ms = budget_ms
+    sub.shared_state = state
+    base = tmp_path / "base.yaml"
+    _write_baseline_yaml(base)
+
+    def _fake_run(cmd, *args, **kwargs):
+        slot = Path(cmd[cmd.index("--output-dir") + 1])
+        # The harness workspace reports e2el mean 2500 ms, well over the 250 ms budget.
+        _fake_workspace(slot, tput=20000.0)
+        return subprocess.CompletedProcess(args=cmd, returncode=0, stdout="ok", stderr="")
+
+    task = await tr.create(
+        kind="explore",
+        params={
+            "config_path": str(base),
+            "output_dir": str(tmp_path / f"explore-latency-{budget_ms:g}"),
+            "base_tput": 200.0,
+            "grid": [{"name": "v_slow_but_fast", "extra_args": "--split 8"}],
+            "variant_timeout_sec": 10,
+        },
+        idempotency_key=f"ex-latency-{budget_ms:g}",
+    )
+    sub.register_executor("explore", ExploreExecutor(session_dir=tmp_path))
+    with patch(
+        "hyperloom.orchestrator.actions.executors._grid_runner.run_with_session_kill",
+        side_effect=_fake_run,
+    ):
+        res = await sub.run_task(task)
+
+    out = res.result
+    tested = out["explore_search_update"]["tested"][canonical_fingerprint("--split 8", {})]
+    assert tested["status"] == "succeeded"
+    assert tested["outcome"] == expected_outcome
+    if expected_outcome == "REVERT":
+        # Not "gain_below_threshold": the variant gained 100x. Naming the wrong
+        # gate would send the search looking for throughput it already has.
+        assert out["losers"][0]["reason"] == expected_reason
+        assert out["winners"] == []
+        gates = {g["gate"]: g for g in tested["gates"]}
+        assert "latency_budget" in gates
+        assert gates["latency_budget"]["passed"] is False
+    else:
+        assert [w["name"] for w in out["winners"]] == ["v_slow_but_fast"]