feat(conc-sweep): grade the ladder on the axis the session keeps on, and report the guard beside it - #1490
Open
haishuok0525 wants to merge 2 commits into
Open
Conversation
…and report the guard beside it The sweep resolved its own axis through the environment-derived graded_metric_key, which diverges from the KEEP path two ways now that KEEP reads the axis recorded at seed. A resume whose shell lost HYPERLOOM_PERF_METRIC would redraw the curve on output throughput while the promotions in the same session were decided on interactivity, and graded_metric_key has no scriptable carve-out, so an image framework -- which reports no interactivity axis at all -- would rank every rung on a field it never measures and report the whole sweep as failed. Both call sites now go through resolved_grading, which is the one place the answer is settled. Each pair also carries the throughput axis beside the objective, and whether it held within the band the session grades under. Reported, not enforced: InferenceX publishes a 2-D frontier with no fixed interactivity target, so a rung that traded throughput for interactivity moved along that frontier rather than violating a constraint -- and drawing that frontier is what a sweep is for. Gating here would drop half the curve. best_conc stays the best rung on the objective alone, with best_conc_guard_holds saying whether the session's own KEEP rule would have taken it. The pair columns are renamed off throughput accordingly: baseline_tput / optimized_tput held an interactivity percentile whenever the session graded on one, and the V6 timeline renamed them further to baseline_throughput / optimized_throughput. They are now *_value, with result.metric naming the axis. Also fixes V6ConcSweepPoint declaring intvty_p90 where the recorded curve has always written e2e_norm_intvty_p90, and lifts bool_or_none into the shared event_fields leaf rather than leaving a second copy in warm_start_event. Co-authored-by: Cursor <cursoragent@cursor.com>
…rites it The fixture called itself a flattened rung "shaped the way the sweep's own report writes it" while keying interactivity as intvty_p90, which is the VariantResult dataclass field and not what _point_from_variant emits. That is why V6ConcSweepPoint could declare the same stale name for as long as it did: the only test reading the key agreed with the schema instead of with the producer. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1480 (base is
feat/sbd-v6-agentx-grading-axis; retarget tomainonce that merges).The divergence this closes
#1480 made the KEEP path read the graded axis recorded at seed, through
resolved_grading. The concurrency sweep kept resolving its own axis throughperf_metric.graded_metric_key, which is the environment-derived form and diverges two ways:graded_metric_keynever seesstate.grading, so a session whose shell lostHYPERLOOM_PERF_METRICwould redraw the curve on output throughput while the promotions in that same session were decided on interactivity. Onesession_breakdown.jsonwould carrymetadata.grading.objective = e2e_norm_intvty_p90next to a sweep event reportingresult.metric = output_throughput.graded_metric_keycallsintvty_grading_enabled, notintvty_serving_grading_enabled. An image framework reports no interactivity axis at all, so under AgentX every rung would be compared on a field it never measures, every pair would fail, and the whole sweep would reportfailedwhere it should have drawn an output-axis curve.Both call sites now go through
resolved_grading, which is the one place the answer is settled. The recorded noise band travels with the axis, so the guard below grades under the band the session started with rather than whatever the current shell holds.The guard is reported, not enforced
A KEEP is two-dimensional: an interactivity gain that clears the threshold and throughput holding inside the noise band. The sweep's pair was one-dimensional, so a rung could show
speedup 1.4on the chart while the session's own rule would have reverted it.Each pair now carries the throughput axis beside the objective, plus
guard_holds. Deliberately not a gate: InferenceX publishes a 2-D frontier with interactivity on x and per-chip throughput on y and no fixed interactivity target, so a rung that traded throughput for interactivity moved along that frontier rather than violating a constraint — and drawing that frontier is precisely what a sweep is for. Gating here would drop half the curve.So
successful_pairs,best_concandbest_speedupkeep their existing meaning (consumers inreport.pyandrecord_conc_sweepalready read them, and silently changing a headline number is worse than not changing it).best_conc_guard_holdssays whether the session's own KEEP rule would have taken the headline rung, and the two disagreeing is worth seeing rather than resolving.Guard columns are null off the interactivity objective, and null when a side did not measure the axis — which is not the same fact as a rung that measured it and fell outside.
Renames
baseline_tput/optimized_tputheld an interactivity percentile whenever the session graded on one, and the V6 timeline renamed them further intobaseline_throughput/optimized_throughput. They are nowbaseline_value/optimized_value, withresult.metricnaming the axis. Contained: the only non-test readers were the producer ingain_mathand the mapping inconc_sweep_event.Two drive-by fixes in the same area:
V6ConcSweepPointdeclaredintvty_p90where the recorded curve has always writtene2e_norm_intvty_p90.bool_or_nonemoves into the sharedevent_fieldsleaf instead of leaving a second copy inwarm_start_event.Test plan
test_conc_sweep.py,test_sbd_v6_conc_sweep_timeline.py,test_sbd_v6_conc_sweep_wiring.py,test_conc_sweep_ceiling.py,test_promote_shared_state_lock.py— 157 passedTestTheSweepGradesOnTheAxisTheSessionKeepsOn: recorded axis beats the environment, the band travels with it, fallback to mode, scriptable carve-outTestTheGuardAxisIsReportedNotEnforced: a rung that bought interactivity with throughput still pairs and can bebest_concwithguard_holdsfalse; the recorded band decides the verdict; the output objective has no second axis; an unmeasured guard axis is null rather than falseinference_optimizersuite — 12777 passed, 18 failed; all 18 reproduce identically on a clean tree (LLM credential and multi-node environment tests) and are untouched by this changeruff check+ruff format --checkover 1232 files cleanNot in scope
prompts/prompt_builder.py:165rendersgraded_axisfrom the same environment-derivedgraded_metric_key, so it can tell the agent the wrong axis on a resume. It is a pure rendering function with no access tostate, so fixing it means threading the resolved axis in from its callers — a separate change.