Skip to content

feat(conc-sweep): grade the ladder on the axis the session keeps on, and report the guard beside it - #1490

Open
haishuok0525 wants to merge 2 commits into
feat/sbd-v6-agentx-grading-axisfrom
feat/conc-sweep-agentx-graded-axis
Open

feat(conc-sweep): grade the ladder on the axis the session keeps on, and report the guard beside it#1490
haishuok0525 wants to merge 2 commits into
feat/sbd-v6-agentx-grading-axisfrom
feat/conc-sweep-agentx-graded-axis

Conversation

@haishuok0525

Copy link
Copy Markdown
Collaborator

Stacked on #1480 (base is feat/sbd-v6-agentx-grading-axis; retarget to main once that merges).

The divergence this closes

#1480 made the KEEP path read the graded axis recorded at seed, through resolved_grading. The concurrency sweep kept resolving its own axis through perf_metric.graded_metric_key, which is the environment-derived form and diverges two ways:

  • A resume is a new process. graded_metric_key never sees state.grading, so a session whose shell lost HYPERLOOM_PERF_METRIC would redraw the curve on output throughput while the promotions in that same session were decided on interactivity. One session_breakdown.json would carry metadata.grading.objective = e2e_norm_intvty_p90 next to a sweep event reporting result.metric = output_throughput.
  • No scriptable carve-out. graded_metric_key calls intvty_grading_enabled, not intvty_serving_grading_enabled. An image framework reports no interactivity axis at all, so under AgentX every rung would be compared on a field it never measures, every pair would fail, and the whole sweep would report failed where it should have drawn an output-axis curve.

Both call sites now go through resolved_grading, which is the one place the answer is settled. The recorded noise band travels with the axis, so the guard below grades under the band the session started with rather than whatever the current shell holds.

The guard is reported, not enforced

A KEEP is two-dimensional: an interactivity gain that clears the threshold and throughput holding inside the noise band. The sweep's pair was one-dimensional, so a rung could show speedup 1.4 on the chart while the session's own rule would have reverted it.

Each pair now carries the throughput axis beside the objective, plus guard_holds. Deliberately not a gate: InferenceX publishes a 2-D frontier with interactivity on x and per-chip throughput on y and no fixed interactivity target, so a rung that traded throughput for interactivity moved along that frontier rather than violating a constraint — and drawing that frontier is precisely what a sweep is for. Gating here would drop half the curve.

So successful_pairs, best_conc and best_speedup keep their existing meaning (consumers in report.py and record_conc_sweep already read them, and silently changing a headline number is worse than not changing it). best_conc_guard_holds says whether the session's own KEEP rule would have taken the headline rung, and the two disagreeing is worth seeing rather than resolving.

Guard columns are null off the interactivity objective, and null when a side did not measure the axis — which is not the same fact as a rung that measured it and fell outside.

Renames

baseline_tput / optimized_tput held an interactivity percentile whenever the session graded on one, and the V6 timeline renamed them further into baseline_throughput / optimized_throughput. They are now baseline_value / optimized_value, with result.metric naming the axis. Contained: the only non-test readers were the producer in gain_math and the mapping in conc_sweep_event.

Two drive-by fixes in the same area:

  • V6ConcSweepPoint declared intvty_p90 where the recorded curve has always written e2e_norm_intvty_p90.
  • bool_or_none moves into the shared event_fields leaf instead of leaving a second copy in warm_start_event.

Test plan

  • test_conc_sweep.py, test_sbd_v6_conc_sweep_timeline.py, test_sbd_v6_conc_sweep_wiring.py, test_conc_sweep_ceiling.py, test_promote_shared_state_lock.py — 157 passed
  • New TestTheSweepGradesOnTheAxisTheSessionKeepsOn: recorded axis beats the environment, the band travels with it, fallback to mode, scriptable carve-out
  • New TestTheGuardAxisIsReportedNotEnforced: a rung that bought interactivity with throughput still pairs and can be best_conc with guard_holds false; the recorded band decides the verdict; the output objective has no second axis; an unmeasured guard axis is null rather than false
  • Regression over conc_sweep / sbd / breakdown / sweep / gain / warm_start / perf_metric / shared_state / report — 1983 passed
  • Full inference_optimizer suite — 12777 passed, 18 failed; all 18 reproduce identically on a clean tree (LLM credential and multi-node environment tests) and are untouched by this change
  • ruff check + ruff format --check over 1232 files clean

Not in scope

prompts/prompt_builder.py:165 renders graded_axis from the same environment-derived graded_metric_key, so it can tell the agent the wrong axis on a resume. It is a pure rendering function with no access to state, so fixing it means threading the resolved axis in from its callers — a separate change.

…and report the guard beside it

The sweep resolved its own axis through the environment-derived
graded_metric_key, which diverges from the KEEP path two ways now that
KEEP reads the axis recorded at seed. A resume whose shell lost
HYPERLOOM_PERF_METRIC would redraw the curve on output throughput while
the promotions in the same session were decided on interactivity, and
graded_metric_key has no scriptable carve-out, so an image framework --
which reports no interactivity axis at all -- would rank every rung on a
field it never measures and report the whole sweep as failed. Both call
sites now go through resolved_grading, which is the one place the answer
is settled.

Each pair also carries the throughput axis beside the objective, and
whether it held within the band the session grades under. Reported, not
enforced: InferenceX publishes a 2-D frontier with no fixed
interactivity target, so a rung that traded throughput for interactivity
moved along that frontier rather than violating a constraint -- and
drawing that frontier is what a sweep is for. Gating here would drop
half the curve. best_conc stays the best rung on the objective alone,
with best_conc_guard_holds saying whether the session's own KEEP rule
would have taken it.

The pair columns are renamed off throughput accordingly: baseline_tput /
optimized_tput held an interactivity percentile whenever the session
graded on one, and the V6 timeline renamed them further to
baseline_throughput / optimized_throughput. They are now *_value, with
result.metric naming the axis. Also fixes V6ConcSweepPoint declaring
intvty_p90 where the recorded curve has always written
e2e_norm_intvty_p90, and lifts bool_or_none into the shared event_fields
leaf rather than leaving a second copy in warm_start_event.

Co-authored-by: Cursor <cursoragent@cursor.com>
@haishuok0525
haishuok0525 requested a review from a team as a code owner September 11, 2026 08:36
…rites it

The fixture called itself a flattened rung "shaped the way the sweep's
own report writes it" while keying interactivity as intvty_p90, which is
the VariantResult dataclass field and not what _point_from_variant
emits. That is why V6ConcSweepPoint could declare the same stale name
for as long as it did: the only test reading the key agreed with the
schema instead of with the producer.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant