feat(sbd-v6): declare the axis a session graded on, and record it where it is resolved

docs/reference/session-breakdown.md

@@ -136,8 +136,8 @@ started, …).
 ## `metadata` — `V6Metadata`
 
 Task identity, recorded as each fact is decided rather than re-derived at
-export. Four blocks: `session`, `task_config`, `versions` and `langfuse`, plus
-the export's own `exported_at_utc` and `warnings`.
+export. Five blocks: `session`, `task_config`, `grading`, `versions` and
+`langfuse`, plus the export's own `exported_at_utc` and `warnings`.
 
 `metadata.session` — identity and lifecycle:
 
@@ -171,6 +171,24 @@ treat the `objective.kind` enum as the canonical optimisation goal. Its
 `architecture` sub-object is the structural model summary parsed from the
 model's own `config.json`, and is empty on non-transformers models.
 
+`metadata.grading` — which axis this session was configured to grade on:
+`benchmark_mode` (`agentx` or `synthetic`), `objective`, and the `tput_guard`
+that rides along with the interactivity objective (`enabled`, `noise_pct`).
+
+An AgentX replay is ranked on the slow-tail interactivity percentile
+(`e2e_norm_intvty_p90`) with total throughput held as a guard; a synthetic run
+is ranked on output throughput alone. Every throughput field elsewhere in this
+document is the output axis by construction, so without this block a consumer
+cannot tell the two kinds of session apart — and on the canonical corpus the
+two axes differ by roughly two orders of magnitude.
+
+This is the session-level *setting*. What the run actually decided a given
+promotion on is `outcome.validation.graded_on`, read off the promotion itself:
+a session configured for interactivity still grades an individual comparison
+on output whenever either side of it cannot supply the axis pair. Neither field
+resolves the other. `tput_guard.noise_pct` is null on a session that predates
+the band being recorded.
+
 `metadata.versions` — the schema version, the Hyperloom revision, the framework
 and its version, and a `tools` map carrying `{tool, root_dir, commit, version}`
 per external tool.
@@ -205,6 +223,14 @@ the exact baseline benchmark.
 `extra_envs` is allowlist-filtered to keep secrets out of the
 breakdown. Do not assume it contains every env var the session ran with.
 
+`baseline.perf` and `final.perf` carry the four AgentX axes the measurement
+reported — `e2e_norm_intvty_p90`, `total_throughput`, `input_throughput`,
+`tpot_p90_ms` — each an explicit `null` where nothing measured it. Absent would
+be indistinguishable from an axis the framework failed to report, and zero
+reads as "measured, and it was zero", so a synthetic run publishes four nulls.
+`final.graded_on` names the axis `final.gain_pct` is on, and always agrees with
+`outcome.validation.graded_on`: they are the same figure read twice.
+
 ---
 
 ## `outcome.final` — `Final` (SaFE contract core)
@@ -237,7 +263,11 @@ downstream consumers:
 
 `outcome` is the terminal result: `status`, `stop_reason`, `stage_reached`,
 the `baseline` and `final` blocks documented above, and the `validation`
-block that reconciles the optimization stack's parts against its total.
+block that reconciles the optimization stack's parts against its total. That
+reconciliation is single-axis and `validation.graded_on` names the axis: an
+attributed figure on one axis against an unattributed figure on another makes
+the gap meaningless. `validation.notes` reports any adoption that fell off
+that axis, because its contribution sits in the same sum as the rest.
 
 `timeline` is the run itself — one event per stage, oldest first. An event
 carries its `type`, its identity (`event_id`, `phase`, `macro_cycle`), its
@@ -369,6 +399,11 @@ The following example shows a complete `session_breakdown.json` for a finished G
       "launch_server_args": "",
       "architecture": { "model_class": "moe_mla_nsa", "model_type": "glm5", "is_moe": true }
     },
+    "grading": {
+      "benchmark_mode": "synthetic",
+      "objective": "output_throughput",
+      "tput_guard": { "enabled": false, "noise_pct": 5.0 }
+    },
     "langfuse": { "enabled": false, "disabled_reason": "no_credentials", "trace_url": null, "counts": {} },
     "warnings": []
   },
@@ -383,6 +418,12 @@ The following example shows a complete `session_breakdown.json` for a finished G
       "accuracy": 0.812,
       "ttft_mean_ms": 0.0,
       "e2el_mean_ms": 0.0,
+      "perf": {
+        "e2e_norm_intvty_p90": null,
+        "total_throughput": null,
+        "input_throughput": null,
+        "tpot_p90_ms": null
+      },
       "ttft_e2el_source": "state_workspace",
       "config_path": "runs/baseline/baseline_config.with_envs.yaml",
       "benchmark_report_path": "runs/baseline/report.json",

src/hyperloom/common/gain_math.py

@@ -8,6 +8,7 @@
 from typing import Any
 
 from hyperloom.common.coerce import to_float
+from hyperloom.common.perf_metric import GRADED_INTVTY, GRADED_TOTAL, graded_axes_of, passes_tput_guard
 
 
 def gain_pct(new: float | None, base: float) -> float | None:
@@ -37,8 +38,18 @@ def conc_pair_comparison(
     optimized_points: list[dict[str, Any]],
     *,
     metric_key: str = "output_throughput",
+    guard_noise_pct: float | None = None,
 ) -> tuple[list[dict[str, Any]], dict[str, Any]]:
-    """Pair curve points by CONC (outer join), compute per-conc speedup, and aggregate."""
+    """Pair curve points by CONC (outer join), compute per-conc speedup on *metric_key*, and aggregate.
+
+    Under the interactivity objective each pair also reports whether throughput held within the noise band the
+    session grades under. It is reported, not enforced: InferenceX publishes a 2-D frontier with no fixed
+    interactivity target, so a rung that traded throughput for interactivity moved along that frontier rather than
+    violating a constraint -- and a sweep exists to draw the frontier. Gating on the guard here would drop half the
+    curve. The KEEP path enforces it because a stack promotion at one concurrency is a different question.
+    """
+    # The guard belongs to the interactivity objective; on the output axis there is no second axis to hold.
+    guard_axis = GRADED_TOTAL if metric_key == GRADED_INTVTY else ""
 
     def _norm_conc(p: dict[str, Any]) -> int | float | str:
         raw = p.get("conc")
@@ -68,23 +79,42 @@ def _norm_conc(p: dict[str, Any]) -> int | float | str:
             successful_pairs += 1
         else:
             failed_pairs += 1
+        # ``graded_axes_of`` is what normalises the sweep's ``total_token_throughput`` onto GRADED_TOTAL, so the
+        # guard cannot read a differently-named axis as an absent one.
+        base_axes = graded_axes_of(b) if guard_axis else {}
+        opt_axes = graded_axes_of(o) if guard_axis else {}
+        guard_holds: bool | None = None
+        if guard_axis and base_axes.get(guard_axis) and opt_axes.get(guard_axis):
+            guard_holds = passes_tput_guard(opt_axes, base_axes, noise_pct=guard_noise_pct)
         rows.append(
             {
                 "conc": c,
-                "baseline_tput": bt,
-                "optimized_tput": ot,
+                # Named for the axis rather than for throughput: under the interactivity objective these hold a
+                # slow-tail percentile, and ``summary.metric`` is what says which.
+                "baseline_value": bt,
+                "optimized_value": ot,
                 "speedup": speedup,
                 "delta_pct": delta_pct,
                 "baseline_status": b.get("status"),
                 "optimized_status": o.get("status"),
+                # The guard axis beside the objective, so the frontier this rung sits on is readable rather than
+                # only the one number it was ranked by. Null off the interactivity objective, and null when a side
+                # did not measure the axis -- which is not the same as a rung that measured it and fell outside.
+                "baseline_guard": base_axes.get(guard_axis) if guard_axis else None,
+                "optimized_guard": opt_axes.get(guard_axis) if guard_axis else None,
+                "guard_holds": guard_holds,
             }
         )
     summary: dict[str, Any] = {
         "metric": metric_key,
+        # The axis held beside the objective, empty off the interactivity objective. Named here so a reader of
+        # ``best_conc_guard_holds`` does not have to infer which axis the verdict is about.
+        "guard_axis": guard_axis,
         "successful_pairs": successful_pairs,
         "failed_pairs": failed_pairs,
         "best_conc": None,
         "best_speedup": None,
+        "best_conc_guard_holds": None,
         "median_speedup": None,
         "mean_speedup": None,
     }
@@ -100,6 +130,9 @@ def _norm_conc(p: dict[str, Any]) -> int | float | str:
             {
                 "best_conc": rows[best_idx]["conc"],
                 "best_speedup": round(best_val, 4),
+                # The headline rung is the best on the objective alone. Whether the session's own KEEP rule would
+                # have accepted it is a second fact, and the two disagreeing is worth seeing rather than resolving.
+                "best_conc_guard_holds": rows[best_idx]["guard_holds"],
                 "median_speedup": round(median, 4),
                 "mean_speedup": round(sum(speedups) / len(speedups), 4),
             }

src/hyperloom/common/perf_metric.py

@@ -26,6 +26,11 @@
 GRADED_TOTAL = "total_throughput"
 GRADED_OUTPUT = "output_throughput"
 
+# The axes ``graded_axes_of`` can carry, for a consumer that must publish all four including the ones a measurement
+# did not supply. Absent and null are not the same fact: a recorder that omits an axis leaves a reader unable to tell
+# an unmeasured axis from one the framework failed to report, and zero reads as "measured, and it was zero".
+GRADED_AXIS_KEYS = (GRADED_INTVTY, GRADED_TOTAL, "input_throughput", "tpot_p90_ms")
+
 # Upstream reports run-to-run noise on this workload as 1-5% depending on the concurrency regime, so the band opens
 # to the top of that range instead of rejecting movement upstream would call noise.
 _DEFAULT_INTVTY_NOISE_PCT = 5.0
@@ -237,6 +242,7 @@ def graded_on_intvty(self) -> bool:
 __all__ = [
     "AGENTX_KEEP_THRESHOLD_FLOOR_PCT",
     "GradedComparison",
+    "GRADED_AXIS_KEYS",
     "GRADED_INTVTY",
     "GRADED_OUTPUT",
     "GRADED_TOTAL",

src/hyperloom/inference_optimizer/breakdown/SKILL.md

@@ -32,8 +32,8 @@ authoritative.
 | `schema_version`    | The wire contract. Gate features on the **major** version, not the exact string.                         |
 | `exported_at_utc`   | When this export was built.                                                                              |
 | `exporter_version`  | Which exporter built it.                                                                                 |
-| `metadata`          | Session identity, launch configuration, component versions, Langfuse receipt, and `warnings` -- how the export itself went, reported once and only here. |
-| `outcome`           | Terminal status, stage reached, stop reason, the `baseline` and `final` measured results, and the `validation` that reconciles the stack's parts against its total. |
+| `metadata`          | Session identity, launch configuration, `grading` -- the axis this session was configured to rank on -- component versions, Langfuse receipt, and `warnings`, how the export itself went, reported once and only here. |
+| `outcome`           | Terminal status, stage reached, stop reason, the `baseline` and `final` measured results with the graded axes each reported, and the `validation` that reconciles the stack's parts against its total on one axis, named by `graded_on`. |
 | `timeline`          | The run itself: one event per stage, oldest first, each carrying its span, status, and an `ext` block of what that kind of stage records. Startup source events live under `reports/sbd_v6/timeline/`. |
 | `close`             | What the session settled at close: the steps the sequencer ran, the artifacts it published, the robustness findings it collected. |
 | `critic`            | The critic agent's own run, iteration by iteration: what it was asked about, how its rulings fell (`verdict_counts`), and the four artifacts each pass left behind. Per-proposal verdicts stay with the proposals, on the timeline. |
@@ -122,7 +122,7 @@ this reference is partial — `breakdown/exporter.py` is authoritative.
 
 | Section     | Read from                                                                                                       |
 |-------------|-----------------------------------------------------------------------------------------------------------------|
-| `metadata`  | `manifest.json` + `state.json`, overlaid by the recorder's own `session` / `task_config` / `versions` fragments  |
+| `metadata`  | `manifest.json` + `state.json`, overlaid by the recorder's own `session` / `task_config` / `grading` / `versions` fragments. `grading` is recorded only: the axis is resolved at seed, where the run can still see its own configuration, and re-deriving it here would read the exporting subprocess's environment |
 | `outcome`   | The recorder's `close` and stack fragments, plus `state.{current_best, cumulative_gain_validated, optimization_stack}` for the sessions that predate them |
 | `timeline`  | The event fragments in the spool, closed and assembled per event; orphans left open by a killed phase are closed on first build |
 | `close`     | The CLOSE sequencer's own `close` / `close_step` fragments                                                        |

src/hyperloom/inference_optimizer/breakdown/collectors/v6.py

@@ -349,6 +349,25 @@ def _stage_reached(
 _ANCHORING_BASELINE_STATUSES = frozenset({"succeeded", "degraded"})
 
 
+def _graded_axes(recorded: Any) -> dict[str, Any]:
+    """Publish the four graded axes a recorder projected, absent ones as explicit nulls.
+
+    The recorder already filled all four, so this only has to hold the shape for a session recorded before it did.
+    All four are always present because absent would be indistinguishable from an axis the framework failed to
+    report, and zero reads as "measured, and it was zero".
+
+    Args:
+        recorded (Any): The recorded ``perf`` block, or ``None`` on a session that has none.
+
+    Returns:
+        dict[str, Any]: The four axes, each ``None`` where nothing measured it.
+    """
+    from hyperloom.common.perf_metric import GRADED_AXIS_KEYS
+
+    source = _mapping(recorded)
+    return {key: _optional_float(source.get(key)) for key in GRADED_AXIS_KEYS}
+
+
 def _baseline_from_timeline(timeline: list[dict[str, Any]]) -> dict[str, Any]:
     """Read the session's anchoring baseline off the ``baseline`` events.
 
@@ -382,10 +401,13 @@ def _baseline_from_timeline(timeline: list[dict[str, Any]]) -> dict[str, Any]:
             # no chronology of its own.
             anchors.append((str(action.get("end_time") or action.get("start_time") or ""), action))
     if not anchors:
-        return dict.fromkeys(_BASELINE_OUTCOME_FIELDS)
+        return {**dict.fromkeys(_BASELINE_OUTCOME_FIELDS), "perf": _graded_axes(None)}
     anchors.sort(key=lambda row: row[0])
     measurement = _mapping(anchors[-1][1].get("measurement"))
-    return {field: _optional_float(measurement.get(field)) for field in _BASELINE_OUTCOME_FIELDS}
+    return {
+        **{field: _optional_float(measurement.get(field)) for field in _BASELINE_OUTCOME_FIELDS},
+        "perf": _graded_axes(measurement.get("perf")),
+    }
 
 
 def _validation_from_timeline(timeline: list[dict[str, Any]]) -> dict[str, Any]:
@@ -432,6 +454,15 @@ def _bucket(*names: str) -> dict[str, Any]:
     validations = _mapping(ledger.get("validations"))
     settled = _mapping(validations.get("settled"))
     return {
+        # The axis every percentage below shares. The reconciliation has to be single-axis: an attributed figure on
+        # one axis against an unattributed figure on another makes the gap meaningless. Read off the validation row
+        # that produced the settled figure, falling back to the axis the adoptions were graded on for a session that
+        # adopted but never validated -- and never from the session's configured axis, which says what was asked
+        # for rather than what this figure was measured on.
+        "graded_on": str(settled.get("graded_objective") or ledger.get("objective") or "") or None,
+        # The settled measurement's own axes, carried beside the gain they produced rather than read off
+        # ``current_best``: a revalidation moves the cumulative figure without re-promoting the recipe.
+        "perf": _graded_axes(settled.get("perf")),
         "attributed_gain_pct": _optional_float(ledger.get("attributed_gain_pct")) or 0.0,
         "unattributed_gain_pct": _optional_float(ledger.get("unattributed_gain_pct")) or 0.0,
         "reconciliation_gap_pct": _optional_float(ledger.get("reconciliation_gap_pct")),
@@ -501,6 +532,24 @@ def _bucket_of(backends: dict[str, Any], name: str, available: bool) -> dict[str
 _RECONCILIATION_NOISE_PP = 0.01
 
 
+def _degraded_adoptions(ledger: dict[str, Any]) -> dict[str, int]:
+    """Count the ledger's adoptions per reason their comparison fell off the configured axis.
+
+    Args:
+        ledger (dict[str, Any]): The stack event's ``ext``.
+
+    Returns:
+        dict[str, int]: Adoption count per ``degrade_reason``, empty when every adoption graded on the axis the
+            session asked for.
+    """
+    counts: dict[str, int] = {}
+    for row in _dict_rows(_mapping(ledger.get("adoptions")).get("rows")):
+        reason = str(row.get("degrade_reason") or "").strip()
+        if reason:
+            counts[reason] = counts.get(reason, 0) + 1
+    return counts
+
+
 def _validation_notes(ledger: dict[str, Any]) -> list[str]:
     """Name what the ledger's own figures say is wrong with it.
 
@@ -536,6 +585,16 @@ def _validation_notes(ledger: dict[str, Any]) -> list[str]:
             "either an adoption is missing from the ledger or its recorded throughputs disagree "
             "with the end-to-end measurement"
         )
+    degraded = _degraded_adoptions(ledger)
+    if degraded:
+        # A session configured for the interactivity axis still grades an individual adoption on output whenever
+        # either side of that comparison cannot supply the axis pair, and its contribution then sits in the same
+        # sum as the axis-graded ones. Naming the count is what keeps the total from reading as single-axis.
+        reasons = ", ".join(sorted(degraded))
+        notes.append(
+            f"{sum(degraded.values())} adoption(s) were graded on the output axis rather than the axis the "
+            f"session was configured for ({reasons}); their contributions are not on the same axis as the rest"
+        )
     validations = _mapping(ledger.get("validations"))
     if not _optional_int(validations.get("count")):
         notes.append("no whole-stack validation was measured, so the ledger has nothing to reconcile against")
@@ -590,6 +649,12 @@ def collect_v6_outcome(
             # session's total means, and asking two sources the same question
             # is how the export came to publish an answer nothing measured.
             "gain_pct": validation.get("validated_total_gain_pct") or 0.0,
+            # The same axis and the same measurement as the gain above, from the one row that produced both. A
+            # consumer sorting sessions has to be able to tell an interactivity-graded AgentX result from an
+            # output-graded synthetic one: on the canonical corpus the two axes differ by two orders of magnitude,
+            # and every other throughput field in this document is the output axis by construction.
+            "graded_on": validation.get("graded_on"),
+            "perf": validation.get("perf"),
             "action_path": [str(step) for step in recipe.get("action_path") or []],
             "extra_envs": dict(_mapping(recipe.get("extra_envs"))),
             "extra_server_args": str(recipe.get("extra_server_args") or ""),