fix(ORACLE-PIN-SOURCE-BUILD): replace the stale pin inequality with the guarantee it stood for, and unred main - #2941
Open
localai-org-maint-bot wants to merge 13 commits into
Open
fix(ORACLE-PIN-SOURCE-BUILD): replace the stale pin inequality with the guarantee it stood for, and unred main#2941localai-org-maint-bot wants to merge 13 commits into
localai-org-maint-bot wants to merge 13 commits into
Conversation
… ENGINE, not a GPU Three places in this spec and issue #2714 record that the MoE placement seam's placed branch cannot be gated without a GPU or a Vulkan/lavapipe build, because the CPU is the only legal placement TARGET and a CPU engine therefore always takes the short circuit. That reasoning constrains the wrong end. `PlacementQueue` limits the destination; `RunMoePlaced` reads the engine device from `engine.q.device.type`, which the caller supplies. The branch fires whenever the ENGINE identifies as something other than the CPU, and this repository already registers a fake backend on `kXPU` in five test files for exactly this kind of question. W3i records the mechanism, what a byte-for-byte host-to-host comparison proves, and what it deliberately does not -- the GB10 NMSE run stays the evidence that a real cross-bus transfer works. It also corrects the three bullets that carried the false dichotomy rather than deleting them, because the wrong conclusion is the part worth being able to find again. FOLLOWING_AGENTS_PROTOCOL Refs #2714 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…t apply a chat template Every number this harness has published was taken with `sp.ignore_eos = true` hardcoded and no chat template, and the report never said so. The comparator for this row posts to `/v1/chat/completions` with a template and no `ignore_eos`, and across its 25 published rows not one generation reaches the 512 cap, so every figure it publishes ended on EOS. Suppressing EOS decodes into lower-entropy continuation, which is easier for a drafter to predict, and the bias lands on our side of the comparison. The spec takes the flag names from vLLM at the current pin `e126687a9` rather than inventing a second vocabulary: `--ignore-eos` from `serve.py:1757-1762`, `--skip-chat-template` from `datasets/datasets.py:1654-1658`, `--chat-template` from `launchers/cli_args.py:80-82`. Both boolean defaults keep today's behaviour, because flipping them would change what every landed figure means. Section 7 records what the gate cannot see. The synthetic fixture declares no EOS id, so no CPU test watches a generation stop; the gate proves the setting reaches the SamplingParams the admission path builds, and the stop itself stays owed to the first checkpoint run. FOLLOWING_AGENTS_PROTOCOL Refs #2759 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
… with no GPU The MoE placement seam's placed branch has had no gate since `6416aab85` deleted `test_placed_moe_roundtrip.cpp`, which was calling a symbol `866075b2f` had removed and was stopping `main` building. Nothing replaced it, and eight architectures route through that seam. It was not rewritten because the seam short-circuits on `placed_on == engine_device` and `kCPU` is the only legal placement target, so the branch looked reachable only from a GPU or a Vulkan build. That reasoning constrains the destination. The seam reads the engine from `engine.q.device.type`, which the caller supplies, so the branch fires whenever the ENGINE is not the CPU -- and the engine does not have to be an accelerator to say so. Five test files here already register a fake backend on `kXPU` for questions of this shape. So the engine is a loopback backend on `kXPU` whose allocation and transfers delegate to the real CPU backend. Both sides being host memory is what lets the two arms be compared byte-for-byte instead of within a tolerance, which is the correct bar for a seam whose whole contract is that placement never changes a value. The loopback COUNTS its transfers, because the defect this branch actually shipped (#2383) was a byte count and not a value: it hardcoded bf16 and copied half of an f32 block, which a value comparison can pass over. Four mutations, each compiled at rc=0 before its result was believed, each RED, and the header restored byte-for-byte after each. Removing the crossing does not merely disagree, it throws `no kernel for op Matmul on device xpu` -- the loopback carries transfers but no kernels, so this test cannot silently fall back into the short circuit and pass. This does not replace the GB10 run at NMSE 5.239e-06; that stays the evidence that a real cross-bus transfer works. It adds the half that one cannot give: a gate that runs on every merge with no GPU and no checkpoint. FOLLOWING_AGENTS_PROTOCOL Closes #2714 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…hat template, and say which it did `MakeSampling` set `sp.ignore_eos = true` with no field, no flag and no environment variable, so no request this harness ever admitted could terminate naturally, and the report never said so. It also sent the raw prompt string, so no chat template was ever applied. The comparator for this row posts to `/v1/chat/completions` with a template and no `ignore_eos`, and across its 25 published rows not one generation reaches the 512 cap, so every figure it publishes ended on EOS. That is a measurement problem: suppressing EOS decodes into lower-entropy continuation, which is easier for a drafter to predict, and the bias lands on our side of the comparison. Four settings, and every flag name is taken rather than invented. `--ignore-eos` is `vllm/benchmarks/serve.py:1757-1762`, `--skip-chat-template` is `vllm/benchmarks/datasets/datasets.py:1654-1658`, `--chat-template` is `vllm/entrypoints/launchers/cli_args.py:80-82`, all at the current pin `e126687a9`, and `--enable-thinking` is `server_main.cpp`'s, resolved by the same `DefaultChatTemplateKwargs` rule. The `--no-` negations are this tree's own convention, needed because both boolean defaults keep TODAY's behaviour: a `vllm-bench` invocation with no new flag is byte-for-byte the one every landed figure was measured on, and flipping a default would change what those numbers mean without re-measuring one of them. The render is one call site, before tokenization, so the pretokenized default and the `VT_BENCH_PRETOKENIZE=0` timed-string path carry the same prompt. Two settings that would otherwise be silently ignored are refused instead: `--chat-template` while the template is skipped, and `--no-skip-chat-template` with no template to load. The report carries the EOS setting as a requested and a resolved pair, the resolved one read back out of the `SamplingParams` `MakeSampling` builds, because a line that echoes the request cannot detect a flag that parses and is then dropped. `tests/examples/test_bench_eos_chat_template.cpp` execs the built binary, since for a benchmark harness the production entry point is `argv` and the existing `test_bench` drives `RunBench` from a hand-built `BenchConfig`. Red before at 9/9 cases failing; green after at 9/9, 73 assertions. `test_bench` 11/11 and `test_bench_kv_cache_dtype` 9/9 unchanged. FOLLOWING_AGENTS_PROTOCOL Refs #2759 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…mutation it cannot catch The mutation table belongs with the spec that promised it. Eight of nine rows are red, and the ninth is the finding: reading the EOS value back out of MakeSampling is the only reason deleting the pass-through is detectable, so replacing that read-back with an echo of the config disarms the proof while changing no value. M1 alone reds three cases, M4 alone reds none, and M1 and M4 together are green with the flag dead. Closing that needs an observation that the setting changed a generation, and the CPU fixture declares no EOS id to emit. The two ways to manufacture one are both worse than the hole -- pinning the toy model's argmax is a tautology that reds on unrelated work, and a synthetic-eos knob is production surface built for a test -- so the spec names it and the first checkpoint run closes it. FOLLOWING_AGENTS_PROTOCOL Refs #2759 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
main moved nine commits under this branch while it was gated. The only file both sides write is tests/CMakeLists.txt, where main registers the token-gate build requirements suite and this branch registers test_bench_eos_chat_template; the two additions are independent target blocks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…l now, and what the test that forbade it was really for `main` is red on `tools suites` at `c796fea41`, which every preflight runs, so every row that merges `main` inherits it. #2931 named two possible repairs and said only the owning row can choose between them. This spec chooses, and states the evidence for each link so the choice is checkable rather than asserted. The invariant is stale and the landed sync is correct. `5d97007c2` is a developer instruction (#2896) whose value was MEASURED: a source build at this revision produced a wheel with no local suffix (`.agents/sync/2026-09-03-e126687-runhalf.md` lines 52 and 56), and the previous `.precompiled` value named the only aarch64 build mode that cannot execute a kernel, so `online_gate.py:3531-3534` was unsatisfiable in both directions. What #520 built the assertion to catch is a DIFFERENT thing than what the assertion asserted. `git show 356fa77^:tools/bench/online_gate.py:3509-3510` is the shape it replaced -- one constant serving both fields. Strict inequality of the two recorded strings was the symptom that made that shape unsatisfiable at the pin of the day, never the guarantee. A source build erases the symptom and leaves the guarantee untouched. So the replacement is behavioural rather than a claim about data, and it is strictly wider than what it replaces: the old assertion could not observe `online_gate.py` at all. The spec also records why NO existing test covers the guarantee -- while the two recorded strings are equal, no real input separates the correct two-field shape from the one-constant shape, because the correct code's two comparisons are against the same value. Spec before code. No implementation is in this commit. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…he guarantee it stood for, and unred main `main` was red on `tools suites` at `c796fea41`, which every preflight runs, so every row that merged `main` inherited it and could not show a green gate (#2931). `tests/tools/test_oracle_pin.py:86` asserted that the pin's distribution and runtime version strings must DIFFER; `5d97007c2` (#2896) made them equal. THE ASSERTION WAS NOT THE GUARANTEE. #520's own body says what it built the check for: the shape it replaced was `metadata == runtime == CONST`, one constant serving both fields, and `git show 356fa77^:tools/bench/online_gate.py` lines 3509-3510 is exactly that. Strict inequality was the SYMPTOM that made the one-constant shape unsatisfiable at the pin of the day, because `VLLM_USE_PRECOMPILED=1` appended `.precompiled` to the metadata string only. A source build at this revision appends nothing -- measured, not inferred: `.agents/sync/2026-09-03-e126687-runhalf.md` lines 52 and 56 record `VLLM_USE_PRECOMPILED=0 VLLM_TARGET_DEVICE=cuda` producing `vllm-0.28.1rc1.dev132+ge126687a9-cp312-cp312-linux_aarch64.whl`. So the two strings coincide and the guarantee is untouched. THE REPLACEMENT IS WIDER, NOT NARROWER. The deleted `assertNotEqual` asserted over DATA and could never observe `online_gate.py` at all. Two new cases execute the call site. `test_record_oracle_compares_each_string_against_its_own_constant` is red against the pre-#520 one-constant shape; the second case is red if the runtime term is deleted or re-pointed. Neither could exist unpatched: while the two recorded strings are equal, no real input separates the two shapes, because the correct code's comparisons are against the same value. The patched value is derived as `VLLM_ORACLE_VERSION + ".precompiled"`, not written as a literal, so it follows the pin. The surviving record-level half is the prefix relation, which holds under both build modes and is a property of the design. MUTATION-VERIFIED on this tree. Restoring `metadata_version != VLLM_ORACLE_VERSION or runtime_version != VLLM_ORACLE_VERSION` turns exactly ONE case red -- the new one -- and leaves the two pre-existing rollback cases green, which is the demonstration that they never covered this. Deleting the runtime term turns the second new case red. The tree was restored byte-for-byte after each, verified by sha256. Three comments the sync falsified are corrected in the same change rather than left contradicting the tree: `online_gate.py` and `serve_low_common.py` both stated in the present tense that the two strings differ on the pin. tests/tools: 735 -> 737, `OK`, and the module is 25 -> 27. Fixes #2931 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Nine commits, no conflicts. Taken before the gate run rather than after, so preflight measures the tree that will actually land: `main` moved through `c796fea41` while this branch was building, and a verdict against the base I started from would have been a verdict about a tree nobody will have. Nothing here touches the placement seam. The overlap is `tests/CMakeLists.txt`, where both sides append registrations in different places, and `src/vt/ops.cpp`, which the new suite links but does not exercise.
… apply a chat template `MakeSampling` set `sp.ignore_eos = true` with no field, no flag and no environment variable, so no request this harness admitted could terminate naturally, and the report never said so. It also sent the raw prompt string, so no chat template was ever applied. The comparator this row measures against posts to `/v1/chat/completions` with a template and no `ignore_eos`, so the two protocols could not be matched. Four settings, every flag name taken from vLLM rather than invented, cited at the parity pin `e126687a9` in the branch's own commit body. Both boolean defaults keep today's behaviour, so an invocation with no new flag is the one every landed figure was measured on. Re-gated on the merged tree rather than on the branch report. `test_bench_eos_chat_template` 9/9 cases and 73 assertions, `test_bench` 11/11, `test_bench_kv_cache_dtype` 9/9. The keyed record `tests/CMakeLists.txt` took both additions: this row's target and main's token-gate build-requirements suite are independent blocks, and both are present. Preflight on the merged tree reports one failed gate, `tools suites`, and it is not this row's. The failure is `test_oracle_pin.ParityPinRecordTests.test_metadata_and_runtime_strings_differ_on_the_pin`, which asserts `VLLM_DISTRIBUTION_VERSION != VLLM_ORACLE_VERSION`. Commit 5d97007 set the distribution version to the source-build value and made the two equal. Verified inherited by running that test on clean `origin/main` at c796fea, where it fails identically; nothing in this row's diff reaches `.agents/upstream-sync.md` or `tests/tools/`. Tracked as #2931. FOLLOWING_AGENTS_PROTOCOL Closes #2759 Refs #2770 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Five more commits while the gate ran. Taken before the push so the suites below were run against the tree that lands, not the one this branch started from. No overlap with the placement seam; `tests/CMakeLists.txt` is the only shared file and both sides append in different places.
…mmand that cannot run, and an anchor this row inherited Fresh review returned PASS WITH REPAIRS on `b0fdf3840`. It confirmed no gate was weakened, and it found four things to correct. None of them moves a gate: every edit here is comment or document text. `tools/bench/gdn_packed_component.py:1596` carried the same present-tense falsehood the row already corrected in `online_gate.py` and `serve_low_common.py`, above the same two-field comparison. The sweep missed the file, so a reader was still told that the distribution string "differs from the runtime one on the pin" when `5d97007c2` (#2896) made them equal. It is rewritten in the wording the other two use, and `## Scope` now lists it, which makes the count five rather than the three the section claimed while listing four. The spec's `## Gates` block invoked `check-commit-style.py --base ... --head ...`. That script takes `--range`, so the documented command exits 2 on an argparse usage error, and AGENTS.md requires a `## Gates` command to run. The PR body already had the correct form, so the spec was the stale copy of the two. Every other command in the block was then run: all exit as documented. The preflight anchor is `scripts/agent-preflight.sh:561`, not `:522`. This one is not the row's mistake to have made. The wrong number came from the body of #2931, and the row copied it verbatim into its spec, its commit body and its pull request body. The spec opens its ruling by saying that "each link was verified against the tree rather than inherited", and this link was the exception. The number here was read at HEAD before it was written, and it agrees with `origin/main`. The spec and the pull request body are corrected; `b0fdf3840` is not rewritten, so its body keeps the wrong number and this commit is the correction of record. A fifth finding is filed, not fixed. `tools/bench/serve_low_common.py:135` still says the oracle constant "already CONTAINS `+g555967922`", which `e8467758e` falsified on 2026-09-03 by advancing the pin to `e126687a9a`. That is an earlier pin advance rather than the sync this row answers, so it is outside the declared scope; it is #2949, it names no owning row, and `## Owed` lists it so it is not orphaned. `## Owed` also no longer leans on #2931 staying open, because this row closes #2931 and that bullet has to survive the merge as the only owner. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…E-BUILD `origin/main` advanced nine commits while the repairs were written. The row's final gate runs `tree-compiles` over the two-dot `origin/main..HEAD` range, so the merge has to happen before the gate rather than after it, and the claims in the spec and the commit body are re-verified against the merged tree. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Collaborator
Author
NEEDS_DECISION — this pull request was partly superseded on
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mainwas red ontools suitesatc796fea41, whichscripts/agent-preflight.sh:561runs in every preflight, so every row that merged
maininherited it and couldnot show a green gate (#2931).
tests/tools/test_oracle_pin.py:86asserted thatthe pin's distribution and runtime version strings must DIFFER;
5d97007c2(#2896) made them equal.
#2931 named two possible repairs and said only the owning row can choose. It is
the second one: the invariant is stale and the landed sync is correct. Every
link was verified against the tree rather than inherited.
The assertion was not the guarantee
#520's own body says what it built the check for: the shape it replaced was
metadata == runtime == CONST, one constant serving both fields.git show 356fa7750^:tools/bench/online_gate.pylines 3509-3510 is literallythat:
Strict inequality of the two recorded strings was the SYMPTOM that made the
one-constant shape unsatisfiable at the pin of the day, because
VLLM_USE_PRECOMPILED=1appended.precompiledto the metadata string only. Itwas never the guarantee. The guarantee is two separately sourced fields, each
observed string compared against its OWN recorded constant.
A source build at this revision appends nothing, and that is measured rather
than inferred:
.agents/sync/2026-09-03-e126687-runhalf.mdline 52 recordsBUILD_ENV VLLM_USE_PRECOMPILED=0 VLLM_TARGET_DEVICE=cudaand line 56 recordsWHEEL=…/vllm-0.28.1rc1.dev132+ge126687a9-cp312-cp312-linux_aarch64.whl. Theold
.precompiledvalue named the only aarch64 build mode that leaves aninstall with no compiled extensions, so
online_gate.py:3531-3534, whichcompares that field for EQUALITY, was unsatisfiable in both directions: the only
string it accepted named a build that cannot execute a kernel.
The pin value is not touched by this change. It is a developer instruction
backed by a recorded measurement, and this row reads it.
The replacement is wider, not narrower
The deleted
assertNotEqualasserted over DATA and could never observeonline_gate.pyat all. Two new cases execute the call site:test_record_oracle_compares_each_string_against_its_own_constantis redagainst the pre-The benchmark harness ENFORCES the 0.25.0 rollback oracle and would REJECT the recorded pin — nobody could have measured correctly even deliberately #520 one-constant shape;
test_record_oracle_compares_the_runtime_string_against_the_runtime_constantis red if the runtime term is deleted or re-pointed at the distribution
constant.
Neither could exist unpatched. While the two recorded strings are equal, no real
input separates the two shapes, because the correct code's two comparisons are
against the same value — the same situation
OracleIdentityIsWiredIntoEveryEntryPointTestsalready documents for the committerm. The patched value is derived as
VLLM_ORACLE_VERSION + ".precompiled",not written as a literal, so it follows any future pin advance. Both cases match
the exact refusal string rather than bare
HarnessError, which the firstmutation vindicated: the mutated code sailed past the version check and died
later on a missing FlashInfer, and only the anchored regex caught it.
The surviving record-level half is the prefix relation
VLLM_DISTRIBUTION_VERSION.startswith(VLLM_ORACLE_VERSION), which holds underboth build modes and is a property of the design rather than of one build mode.
Mutation-verified on this tree
Restoring the pre-#520 one-constant shape turns exactly ONE case red — the new
one — and leaves the two pre-existing rollback cases green, which is the
demonstration that they never covered this. Deleting the
runtime_version != VLLM_ORACLE_VERSIONterm turns the second new case red. Thetree was restored byte-for-byte after each, verified by sha256 and an empty
git status.Records
Five comments the sync falsified are corrected in the same change rather than
left contradicting the tree:
tools/bench/online_gate.py,tools/bench/serve_low_common.pyandtools/bench/gdn_packed_component.pyallstated in the present tense that the two strings differ on the pin.
.agents/specs/oracle-pin-source-build.mdis committed in0afa3f744, beforethe implementation in
b0fdf3840.Fresh-review repairs
The review returned PASS WITH REPAIRS and confirmed no gate was weakened: eight
mutations, each red for the claimed reason. Its four findings are applied in
fd86a7476, and none of them moves a gate.tools/bench/gdn_packed_component.py:1596was missed by the sweep above andcarried the same falsehood over the same two-field comparison. It is rewritten
in the wording the other files use, and
## Scopenow lists it, which makesthe count five rather than the three the section claimed while listing four.
## Gatesblock calledcheck-commit-style.pywith--baseand--head. The script takes--range, so the documented command exited 2 on anargparse usage error, which reads as a gate failure to the next runner. This
body already had the correct form, so the spec was the stale copy. Every other
command in the block was then run, and each exits as documented.
of main is red: an ORACLE-VLLM sync made the distribution and oracle versions equal, which test_oracle_pin pins as impossible #2931 and was copied verbatim into the spec,
b0fdf3840's commit body andthis body, while the spec opens by saying each link was verified against the
tree rather than inherited. It was re-read at HEAD before being written here
and agrees with
origin/main.b0fdf3840is not rewritten, sofd86a7476isthe correction of record for its body.
One finding is filed rather than fixed.
tools/bench/serve_low_common.py:135still says the oracle constant "already CONTAINS
+g555967922", whiche8467758efalsified on 2026-09-03 by advancing the pin toe126687a9a— anearlier pin advance, not the sync this row answers, so it is outside the declared
scope. It is filed as #2949, which names no owning row, and the spec's
## Owedlists it so it is not orphaned. That section also no longer leans on #2931
staying open, since this pull request retires #2931 and the bullet has to survive
as the only owner.
Gates
Re-run on the repaired tree after merging
origin/main:tools suitesisRan 737 tests,OK;check-agent-record.pyis rc 0 with the refreshedsnapshot present, which is what makes it read the newly filed issue; and
check-pr-size.pyandcheck-commit-style.py --rangeare both rc 0.scripts/agent-preflight.shexits 1 on ONE gate,test_cpu_x86_llamacpp_floor,and it is not this row's. Its failing case is
test_a_contended_leg_is_discarded_and_never_summarisedand its own output namesthe reason:
load=108.76,waiting for quiet: 15s busy=128%. Three othersessions were compiling on this box.
git diff --name-only origin/main..HEADlists four files, none of them that test or
scripts/cpu-x86-llamacpp-floor.shwhich it drives, and the test references nothing this change touches.
Re-run alone on this exact tree once the box quietened:
Ran 10 tests in 83.254s,OK, rc 0, atload 23.37. Same binary, same commit, different contention — sothe red was the contention and not the tree. That is the documented behaviour of
this test, whose failing case is named for the condition it could not establish.
Fixes #2931
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]