From 7bc2546e9561a0db21741f4e98753417abd6c732 Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Wed, 2 Sep 2026 22:51:24 -0700 Subject: [PATCH 1/8] spec(BACKEND-ROCM): define Qwen3.5 gfx1100 numerical gate Issue #1588 still lacks active cache-state evidence and a three-mode ROCm correctness gate. This spec fixes the post-write probes, dtype audit, tolerance policy, tests, review mutations, and hardware evidence before implementation starts. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-5.6-sol [codex] --- .../rocm-qwen35-08b-cpu-gfx1100-numerics.md | 685 ++++++++++++++++++ 1 file changed, 685 insertions(+) create mode 100644 .agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md diff --git a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md new file mode 100644 index 0000000000..c6ae9756e1 --- /dev/null +++ b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md @@ -0,0 +1,685 @@ +# Qwen3.5-0.8B CPU and gfx1100 numerical characterization + +Row: `BACKEND-ROCM`. + +Issue: [#1588](https://github.com/mudler/vllm.cpp/issues/1588). + +Base: `4d10c8acc527c34a6a58a309d52ea5f8fbd1d47b`. + +Branch: `row/BACKEND-ROCM-NUMERICS-1588`. + +Primary oracle: vLLM at `5559679229bc961848b121ccdeaa8fa5d79bec98`. + +Target: AMD Radeon RX 7900 XTX, `gfx1100`. + +## Now + +The issue is open and the implementation has not started. This spec is the +committed prerequisite for the implementation. + +Current `main` runs the Qwen3.5-0.8B paged engine on ROCm. The backend matrix +still names its CPU and ROCm numerical characterization as open. + +The current gate runs one default cache configuration. It does not characterize +`auto`, `bfloat16`, and `fp8_e4m3` separately. + +The current dump records the residual halves and named layer stages. It does not +record full-attention cache writes or persistent Gated Delta Net state writes. + +ROCm static graph mode remains false in `src/vllm/platforms/rocm.cpp:91-98`. +This work characterizes the current eager path before issue #332 activates model +graph replay. + +## Live gap + +The spec implementer checked the pinned base before writing this spec. + +- `HEAD`, `origin/main`, and the merge base all resolve to the pinned base. +- `git log --all --grep=1588` finds no landed commit for issue #1588. +- No spec with this scope exists on the pinned base. +- GitHub reports issue #1588 as open. +- No open pull request owns issue #1588. +- The issue comment describes an older Q4_K_M experiment at `e2a9e035d`. +- That experiment predates the current checkpoint gate and the later W1 fix. +- That experiment does not compare persistent state or cache modes. + +The current production gate is +`tests/parity/test_qwen35_paged_engine.cpp:159-484`. It loads the model through +`LoadedEngine::FromModelDir` and exercises the paged engine. + +The gate checks 16 prompts against a pinned ROCm vLLM oracle. It reports strict +token agreement and applies the ratified 500 milli-nat near-tie rule. + +The gate constructs `EngineParams{}` at +`tests/parity/test_qwen35_paged_engine.cpp:231-234`. It therefore covers only +the resolved default cache dtype. + +The gate checks native selection and zero declines for 15 required operators at +`tests/parity/test_qwen35_paged_engine.cpp:254-275` and `:422-446`. It does not +run the required three-mode matrix. + +The current dump writer lives in +`include/vllm/model_executor/models/act_dump.h:55-227`. Its manifest key is +`(step, layer, stage)` and its schema is: + +```text +step layer stage dtype rows cols bytes file +``` + +`DenseForwardLayers` records `hidden` and `res` before layer 0 and after every +layer at `src/vllm/model_executor/models/qwen3_5.cpp:9352-9403`. + +`RunDenseLayerPaged` records `block_out` and `mlp_out` at +`src/vllm/model_executor/models/qwen3_5.cpp:7700-7772`. On a full-attention +layer, `block_out` is the complete attention output after its output projection. + +These existing rows answer the residual-stream, attention-output, and +multilayer perceptron output questions. The comparator must reconstruct the +residual stream as `hidden + res` in FP32. + +The full-attention store calls `dense_attn::WriteKvCache` at +`src/vllm/model_executor/models/qwen3_5.cpp:5894-5905`. No dump reads the +destinations after that call. + +GDN prefill scatters working state at +`src/vllm/model_executor/models/qwen3_5.cpp:5280-5304` and `:5454-5495`. +GDN decode can update persistent state in place at `:5319-5334` and +`:5431-5445`. + +No existing row records the state after all four write paths. Dumping an FP32 +working buffer would not answer what the persistent cache stores. + +## Scope + +### In scope + +1. Keep the existing `VT_DUMP_ACT` writer and manifest as the dump surface. +2. Keep `VT_DUMP_ACT_SUB` as the switch for named sub-stage rows. +3. Add no state-specific environment variable. +4. Record active full-attention K and V destinations after the production store. +5. Record active GDN convolution and SSM rows after the production update. +6. Compare CPU and ROCm on identical model inputs. +7. Report `max_abs`, root mean square error, and relative L2 error for each key. +8. Audit every relevant local dtype against the pinned upstream executing chain. +9. Report the byte cost of each local dtype and each upstream difference. +10. Gate the ROCm paged engine in `auto`, `bfloat16`, and `fp8_e4m3` modes. +11. Prove that each ROCm mode has zero reference-tier hits. +12. Run the final acceptance on one RX 7900 XTX with architecture `gfx1100`. + +### Out of scope + +- Enabling ROCm static graph mode. Issue #332 owns that change. +- Changing a model dtype before the characterization identifies a defect. +- Changing a kernel reduction order to make raw state bytes equal. +- Adding a new activation-dump environment variable. +- Dumping unused KV blocks or unused recurrent-state slots. +- Throughput, latency, and power measurements. +- A CUDA comparison. +- A GGUF or quantized-weight model arm. +- A public benchmark claim. +- A fix for a newly found numerical defect without its required issue flow. + +## Artifact and oracle pins + +Use this exact model snapshot for every local and oracle run: + +```text +repository: Qwen/Qwen3.5-0.8B +revision: 2fc06364715b967f1860aea9cf38778875588b17 +path: /home/vikash/models/Qwen3.5-0.8B +``` + +The local checkpoint inspection found these hashes: + +```text +model.safetensors 04b1c301231dd422b8860db31311ab2721511346a32cb1e079c4c4e5f1fe4696 +config.json b90b86f35c8e6925ef74ee04d0e758f0a845c83a42089ad82bbaa948de9b4204 +``` + +The model file is 1,746,942,600 bytes. Its safetensors header declares 452 BF16 +tensors and 36 FP32 tensors. + +`tests/parity/hf_snapshot.h` pins the same revision and exposes +`parity::Qwen35_08BSnapshot()`. An explicit snapshot override checks existence, +not revision identity. + +The characterization must hash the two files before every run. A hash mismatch +is `ARTIFACT_MISMATCH`, not a skipped comparison. + +Use the vLLM source checkout at `/home/vikash/oracle/vllm-src`. Its detached +`HEAD` must equal `5559679229bc961848b121ccdeaa8fa5d79bec98`. + +Run the pinned vLLM oracle on the identical model, prompt set, cache mode, +sampling configuration, and token count. Use production configuration without +`--enforce-eager` for the end-to-end denominator. + +The internal CPU and ROCm comparison remains eager on both local arms. The dump +synchronizes the queue and cannot run inside graph capture. + +## Model geometry + +The pinned checkpoint declares this text-model geometry: + +| Field | Value | +|---|---:| +| Model dtype | `bfloat16` | +| Hidden size | 1,024 | +| Decoder layers | 24 | +| Full-attention interval | 4 | +| Full-attention layers | 6 | +| GDN layers | 18 | +| Query heads | 8 | +| KV heads | 2 | +| Attention head dimension | 256 | +| GDN key heads | 16 | +| GDN value heads | 16 | +| GDN key dimension | 128 | +| GDN value dimension | 128 | +| GDN convolution kernel | 4 | +| GDN SSM dtype | `float32` | + +The full-attention layers are 3, 7, 11, 15, 19, and 23. Every other layer is a +GDN layer. + +## Upstream executing chain + +The implementation must preserve these pinned upstream decisions. + +- `vllm/model_executor/models/qwen3_5.py:471-531` routes the text model and + publishes its recurrent state dtype. +- `vllm/model_executor/models/qwen3_next.py:389-400` projects Q, K, and V, then + calls attention before the output projection. +- `vllm/model_executor/models/qwen3_next.py:492-550` keeps the residual pair and + routes each layer to full attention or GDN. +- `vllm/model_executor/models/config.py:744-768` copies checkpoint + `mamba_ssm_dtype` into the default SSM cache dtype. +- `vllm/model_executor/layers/mamba/mamba_utils.py:96-128` resolves the + convolution and temporal-state dtypes independently. +- `vllm/model_executor/layers/mamba/mamba_utils.py:180-199` defines the two GDN + state shapes. +- `vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py:1228-1308` + identifies the persistent convolution state and its indexed write paths. +- `vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py:1372-1478` + identifies the persistent SSM updates and the prefill scatter. +- `vllm/v1/attention/ops/paged_attn.py:31-50` calls the paged KV store with the + slot mapping and cache scales. + +The local mirror resolves the cache layout in +`src/vllm/model_executor/models/qwen3_5_common.cpp:37-90`. The runner consumes +that layout at `src/vllm/v1/worker/gpu/runner.cpp:930-999` and `:1747-1793`. + +The local full-attention store routes through +`include/vllm/model_executor/models/kv_cache_route.h:48-67`. The float and FP8 +arms reach different store operators through this shared seam. + +The implementation must inspect the complete executed kernel chain for each +mode. A dispatch wrapper alone does not establish a compute dtype. + +## Dtype and byte contract + +The checkpoint and pinned upstream source establish these storage types before +any measurement: + +| Surface | Shape per active unit | Required storage | Bytes per unit | +|---|---|---|---:| +| Hidden state | `[T, 1024]` | BF16 | `2048 * T` | +| Residual half | `[T, 1024]` | BF16 | `2048 * T` | +| Full-attention K | `[T, 2, 256]` per layer | BF16 or FP8 E4M3 | `1024 * T` or `512 * T` | +| Full-attention V | `[T, 2, 256]` per layer | BF16 or FP8 E4M3 | `1024 * T` or `512 * T` | +| GDN convolution state | `[1, 6144, 3]` per layer | BF16 | 36,864 | +| GDN SSM state | `[1, 16, 128, 128]` per layer | FP32 | 1,048,576 | + +`auto` and `bfloat16` both store full-attention K and V in BF16. Across six +full-attention layers, they use 12,288 bytes per cached token. + +`fp8_e4m3` stores full-attention K and V in one byte per element. Across six +layers, it uses 6,144 bytes per cached token. + +FP8 therefore removes 6,144 bytes per cached token from this model's +full-attention cache. The report must confirm the allocated bytes from the +resolved `FullAttentionSpec` instead of trusting this arithmetic alone. + +Across 18 GDN layers, convolution state uses 663,552 bytes per active state +slot. SSM state uses 18,874,368 bytes per active state slot. + +The required GDN total is 19,537,920 bytes per active state slot. Storing the +SSM state in BF16 would remove 9,437,184 bytes but would violate this checkpoint. + +The dtype audit must include each model-path buffer and each GEMM output that +feeds a compared boundary. For each item, record: + +1. The local symbol and runtime dtype. +2. The pinned upstream symbol and runtime dtype. +3. The tensor shape and active element count. +4. The local bytes and upstream bytes. +5. The byte difference at the measured token count. +6. The source annotation for every local FP32 exception. + +The audit includes the attention query, K and V before storage, attention +output, GDN projections, GDN working state, MLP output, residual halves, stored +KV, and stored GDN state. + +A wider local dtype is not accepted because tokens agree. Classify it as +`DTYPE_WIDTH_MISMATCH` unless a committed source annotation names the upstream +reason for the FP32 exception. + +## State dump design + +Enable state rows only when `VT_DUMP_ACT` and `VT_DUMP_ACT_SUB` name the same +writable directory. Use the existing writer and the existing manifest. + +The existing `hidden`, `res`, `block_out`, and `mlp_out` keys do not change. +Add these four stage keys: + +| Stage key | Production source | On-disk dtype | Manifest shape | +|---|---|---|---| +| `state_fa_k` | K destination after `WriteKvCache` | FP32 | `[active_tokens, 512]` | +| `state_fa_v` | V destination after `WriteKvCache` | FP32 | `[active_tokens, 512]` | +| `state_gdn_conv` | Persistent convolution cache after update | FP32 | `[active_requests, 18432]` | +| `state_gdn_ssm` | Persistent SSM cache after update | FP32 | `[active_requests, 262144]` | + +FP32 is the canonical comparison format on disk. It does not change the cache +storage type. + +For BF16 state, read the persistent destination and widen it to FP32. For FP8 +KV, read the stored byte and dequantize it with that layer's recorded scale. + +The run log must state the physical storage dtype, K scale, and V scale. The +dtype audit records the physical byte width independently of the FP32 dump. + +Gather full-attention rows from nonnegative `slot_mapping` entries only. Preserve +the input token order instead of sorting by physical slot. + +Gather GDN rows from the active state indices for the current step. Preserve the +request order used by the scheduler metadata. + +Deduplicate no row silently. Refuse duplicate destination indices unless the +executing metadata defines their update order. + +Do not download an entire KV block pool. Do not download an entire GDN state +pool. A single-request run must write one GDN row per layer per step. + +Place the full-attention probe immediately after the shared production store. +The queue ordering must make the probe read the stored destination. + +Place the GDN probe at one shared tail after all prefill and decode update arms. +The probe must gather from `state.conv_state` and `state.ssm_state`. + +Do not dump `dcs`, `dss`, `kw`, or `vw` as substitutes. Those tensors precede a +downcast, quantization, scatter, or indexed in-place update. + +Extend the per-step narrative with separate stream, stage, and state counts. A +24-layer step must report 48 state blobs, with two blobs from each layer. + +An enabled state dump that writes fewer than 48 required blobs must refuse the +run. The manifest row count and active-row shapes remain the stronger capacity +checks. + +## Comparison workload + +Use one binary and one checkpoint copy for both local arms. Set +`EngineParams::device` to `kCPU` for the CPU arm. Use the detected ROCm platform +for the device arm. + +Use one request at concurrency 1. Use the first standard gate prompt: + +```text +The capital of France is +``` + +Use greedy sampling, a fixed seed, MTP disabled, and eight output tokens. Keep +the block size, block count, and scheduler token budget equal on both arms. + +Run each cache mode in a separate process and a separate empty dump directory. +This avoids static environment parsing and file-name collisions. + +Run each local arm twice with dumps enabled. The two repeats establish the +run-to-run floor. + +Run an additional dump-disabled control for each arm and mode. Its token IDs and +logits must equal the corresponding enabled run. + +Compare prefill and decode steps only when both arms consume the same token IDs. +The CPU denominator and ROCm treatment must share the complete input prefix. + +If greedy outputs diverge, include the first divergent output step. Exclude every +later step because those steps consume different inputs. + +Record the excluded step count. Do not compare equal step ordinals after the +input prefixes diverge. + +If no decode step has an identical input prefix, classify the end-to-end failure +first. Do not use a re-prefilled prefix as evidence for incremental state. + +## Metrics and advance policy + +Decode every compared blob to FP32. Let `A` be ROCm and let `B` be CPU. + +For `N` elements, report these metrics for every joined key: + +```text +max_abs = max_i(abs(A_i - B_i)) +rms = sqrt(sum_i((A_i - B_i)^2) / N) +rel_l2 = sqrt(sum_i((A_i - B_i)^2)) / sqrt(sum_i(B_i^2)) +``` + +If both vectors have zero L2 norm, define `rel_l2` as zero. If only `B` has zero +L2 norm, define `rel_l2` as infinity. + +The report must name directory A and directory B in words. It must report both +manifest row counts, joined rows, rejected rows, and excluded post-divergence +steps. + +The comparator must refuse missing keys, duplicate keys, dtype mismatches, +shape mismatches, byte-count mismatches, nonfinite values, and partial joins. + +Raw byte equality is not the cross-device acceptance rule. CPU and ROCm kernels +can use different valid reduction orders. + +Use these unit roundoffs: + +```text +u(FP32) = 2^-24 +u(BF16) = 2^-8 +u(FP8 E4M3) = 2^-4 +``` + +Before reading cross-device results, finish the dtype audit. For each comparison +key `k`, record `n(k,d)`, the executed round-to-nearest terms at dtype `d`. + +Count a destination store as one term. Count each reduction term at its actual +accumulator dtype. Cite the executing kernel for every count. + +Freeze the audit file hash before the comparator reads either cross-device dump. +Compute this three-root-mean-square envelope: + +```text +E(k) = 3 * sqrt((2 / 3) * sum_d(n(k,d) * u(d)^2)) +``` + +The `sqrt(2)` term models two independent rounding paths. The `1 / sqrt(3)` term +is the root mean square error of round-to-nearest. + +A key is inside the pre-registered dtype envelope only when both conditions hold: + +```text +rel_l2 <= E(k) +rms <= E(k) * sqrt(sum_i(B_i^2) / N) +``` + +Report `max_abs` and the worst index for diagnosis. Do not invent a fixed +absolute threshold after the values are visible. + +For ordered layer boundaries, compute the nonnegative squared-error increment: + +```text +delta(k) = max(0, rel_l2(k)^2 - rel_l2(input(k))^2) +``` + +Classify a discontinuity when `delta(k)` exceeds `E(k)^2` and four times the +median increment for the same stage and layer family. Freeze this factor before +the hardware run. + +The envelope classifies a result. It does not replace the end-to-end correctness +gate. + +## Mismatch classification + +Classify every run as exactly one primary result: + +- `ARTIFACT_MISMATCH`: a source revision, model hash, or run configuration differs. +- `INSTRUMENTATION_FAIL`: an enabled dump is incomplete or changes model output. +- `NONDETERMINISTIC`: a same-arm repeat has a nonzero metric. +- `STRUCTURE_MISMATCH`: key sets, active indices, shapes, or physical dtypes differ. +- `DTYPE_WIDTH_MISMATCH`: local storage is wider than upstream without an accepted annotation. +- `NONFINITE`: one arm creates a NaN or infinity that the other arm does not create. +- `WITHIN_DTYPE_ENVELOPE`: every key meets the frozen envelope and has no discontinuity. +- `ORDERING_DRIFT`: tokens pass, but one numerical key exceeds the frozen envelope without a structural error. +- `NUMERICAL_DEFECT_CANDIDATE`: a state or layer discontinuity identifies a bounded production region. +- `CORRECTNESS_FAIL`: the applicable end-to-end token gate fails. + +Apply the list in this precedence order: + +1. `ARTIFACT_MISMATCH`. +2. `INSTRUMENTATION_FAIL`. +3. `NONDETERMINISTIC`. +4. `STRUCTURE_MISMATCH`. +5. `DTYPE_WIDTH_MISMATCH`. +6. `CORRECTNESS_FAIL`. +7. `NONFINITE`. +8. `NUMERICAL_DEFECT_CANDIDATE`. +9. `ORDERING_DRIFT`. +10. `WITHIN_DTYPE_ENVELOPE`. + +`CORRECTNESS_FAIL` therefore outranks every interpretable numerical label. +`INSTRUMENTATION_FAIL` and `NONDETERMINISTIC` stop the cross-device reading. + +`WITHIN_DTYPE_ENVELOPE` does not promise equal state bytes. It means the measured +difference fits the dtype-derived rounding model and the token gate passes. + +`ORDERING_DRIFT` does not authorize a wider tolerance. Record the exact stage, +kernel chain, and token margin before any change. + +## End-to-end correctness gate + +Extend the current Qwen3.5 paged-engine gate with named cases for: + +1. `auto`. +2. `bfloat16`. +3. `fp8_e4m3`. + +Each case must set `EngineParams::kv_cache_dtype` explicitly. The test name and +log must print the requested and resolved cache dtype. + +`auto` and `bfloat16` resolve to the same physical BF16 cache. Require their +local token streams to match each other exactly. + +Capture a pinned vLLM oracle pair for each distinct physical cache mode. Do not +reuse the BF16 teacher-forced gaps for FP8. + +For each mode, run the existing 16-prompt battery at its current token count. +Keep the current strict-token report and ratified 500 milli-nat near-tie rule. + +The pinned vLLM oracle must use the same cache mode. Capture at least 10 +per-prompt greedy repeats and require determinism before creating a golden. + +Every ROCm case must prove all 15 required operators have nonzero native +selections and zero declines. It must also assert `reference_tier_hits=0`. + +The state-dump characterization does not need to run the 16-prompt battery with +dumps enabled. The narrow workload supplies internal evidence without dumping +inactive capacity. + +## Tests and red-first evidence + +The implementation starts with tests that fail because the four state rows are +absent. Capture the red output before editing production code. + +Add focused tests for these contracts: + +1. The state writer emits all four keys with the declared shape and FP32 dtype. +2. A nonmonotonic slot mapping preserves input token order. +3. Negative slot mappings produce no KV row. +4. Inactive KV blocks and GDN slots produce no bytes. +5. A BF16 state row is widened from the stored destination. +6. An FP8 row is read after quantization and dequantized with its scale. +7. GDN prefill and decode both read persistent post-write state. +8. A missing required state row makes the per-step floor refuse. +9. The comparator refuses duplicate, missing, partial, short, and nonfinite data. +10. The comparator implements the zero-denominator rules exactly. +11. The dtype audit rejects an unannotated wider local dtype. +12. Each cache-mode gate prints its requested and resolved dtype. +13. The ROCm gate rejects one injected reference-tier hit. + +The checkpoint-backed focused test must enter through +`LoadedEngine::FromModelDir`. A hand-built `PagedKvCache` test can test the +gather helper but cannot prove production reachability. + +Run the focused CPU tests before hardware work. Run the ROCm cases only inside +the required GPU lease and file mutex. + +`IMP-TEST-FIRST` and `IMP-MUTATE` do not apply to this spec-only commit. This +commit changes no behavior that a test or mutation can falsify. + +They apply to the later implementation. Preserve each red result and each +mutation result in that implementation's evidence. + +## Review mutations + +A fresh reviewer reviews one immutable implementation head. The reviewer uses a +scratch copy and restores every file byte for byte after each mutation. + +Run these mutations separately: + +1. Delete the production probe after `dense_attn::WriteKvCache`. +2. Delete the shared GDN post-write probe. +3. Replace the post-write KV read with `kw` and `vw`. +4. Replace the post-write GDN read with `dcs` and `dss`. +5. Remove negative-slot filtering. +6. Sort active rows by physical slot. +7. Dump the full allocated cache capacity. +8. Report the FP8 physical dtype as BF16. +9. Remove one cache-mode test case. +10. Force one ROCm operator to the reference tier. +11. Delete the `LoadedEngine::FromModelDir` production call from the focused test. + +Each mutation must make its named test fail for the intended reason. A green +reachability mutation is a review finding. + +Record the immutable head, scratch path, command, exit status, and restored tree +hash for each mutation. + +## Gates + +The later implementation must pass these gates in order: + +1. Confirm the checkpoint and oracle hashes. +2. Capture the red focused tests. +3. Pass the dump-writer and comparator tests on CPU. +4. Pass the checkpoint-backed CPU trace case. +5. Run the full controlled preflight. +6. Get a fresh immutable review with all mutations detected. +7. Run two CPU and two ROCm trace repeats for each cache mode. +8. Pass the dump-disabled identity control for every arm and mode. +9. Pass all three ROCm paged-engine end-to-end cases. +10. Prove zero ROCm reference-tier hits in every case. +11. Have the operator rerun the focused, full, and hardware gates. + +Use this controlled environment for local repository gates: + +```sh +PATH=/usr/local/bin:/usr/bin:/bin \ +PYTHONPATH=$PWD \ +GIT_CONFIG_GLOBAL=/dev/null \ +scripts/agent-preflight.sh +``` + +An exit status of zero with skipped checks is not green. List each skip and its +reason. + +The hardware run must use the repository's lease procedure. Use the file mutex +inside the lease because the target is a non-fleet local GPU unless `rc devices` +reports it as a fleet device. + +## Evidence required + +Record this evidence for the implementation and hardware run: + +- The implementation commit and tree hashes. +- The clean source status before and after each gate. +- The exact checkpoint hashes and upstream revision. +- The compiler, build type, ROCm version, kernel driver, board name, and `gfx1100` architecture. +- The GPU lease identity or the non-fleet mutex evidence. +- The exact commands and exit statuses. +- Every preflight skip. +- The requested and resolved cache dtype for each run. +- The physical cache dtype and allocated bytes for each state surface. +- The operator-provider selection, decline, and reference-tier counts. +- Both enabled repeats and the disabled identity control. +- Both manifests and all raw dump paths. +- The frozen dtype-audit file and its hash. +- The full metric table for every joined key. +- The first token divergence and excluded step count. +- The selected mismatch classification. +- The red-first output and every review mutation result. +- The fresh reviewer report on the immutable head. +- The operator's independent gate rerun. + +Keep raw evidence under an issue-specific evidence path. Do not publish a result +from temporary files that the reviewer cannot inspect. + +The spec-only preflight ran at the pinned base with the controlled environment. +It returned zero and skipped 10 checks. + +Five LTX-2.5 checks skipped because NumPy was unavailable. Three ISA checks +required compile commands. The pull-request size check required base and head +arguments. The Triton AOT check required a vendored source root. + +This result is `PENDING`, not green. No source build, checkpoint inference, or +GPU gate applies to this spec-only commit. + +## Risks + +- The dump can synchronize the queue and perturb scheduling. The disabled + identity controls detect output changes. +- FP8 K or V near a quantization boundary can change one stored code. Numeric + comparison occurs after dequantization and never assumes byte equality. +- GDN update kernels use different reduction orders on CPU and ROCm. The frozen + dtype envelope classifies that difference without widening after measurement. +- A greedy divergence makes later steps incomparable. The comparator excludes + those steps and reports the exclusion. +- The FP32 canonical dump can hide a physical-width error. The independent byte + audit and run narrative cover the storage width. +- A probe before the write can look plausible. The pre-write substitutions are + required review mutations. +- Static environment parsing can mix output directories. Separate processes and + empty directories prevent that collision. +- The current gate goldens describe the BF16 default. FP8 needs its own pinned + oracle capture and teacher-forced gaps. +- Issue #1588 currently lacks the required first-line `Row:` metadata. An + authorized operator must reconcile it before the pull request lands. +- Graph activation changes execution order and invalidates this characterization. + Issue #332 requires a new graph-on measurement. + +## Stop conditions + +- Stop with `NEEDS_CONTEXT` if any required revision, hash, model file, or board + identity is unavailable. +- Stop if the issue, spec, and pull request do not name `BACKEND-ROCM` together. +- Stop if the dump changes tokens or logits on either arm. +- Stop if a same-arm repeat is not deterministic. +- Stop if a manifest omits an active row or includes inactive capacity. +- Stop if the dtype audit is not frozen before cross-device results are read. +- Stop if the local chain cannot name an upstream dtype for a compared surface. +- Stop if any ROCm operator declines or reaches the reference tier. +- Stop if the running device is not an RX 7900 XTX with architecture `gfx1100`. +- Stop if ROCm static graph mode becomes true before the measurement. +- Stop before changing a tolerance after result values are visible. +- Stop before fixing an unexpected numerical defect without the required issue + and fresh implementation flow. +- Do not report a hardware pass from a skipped test. + +## Git integration + +Use one pull request for the spec and implementation. This is the repository +policy default for `BACKEND-ROCM` and the selected shape for this row. + +Commit this spec before any implementation. A fresh implementer starts from the +committed spec. + +Keep the work on `row/BACKEND-ROCM-NUMERICS-1588`. The spec implementer does not +push or open the pull request. + +The pull request body must link issue #1588 and close it when the complete work +lands. The issue's `Row:` line, this spec, and the pull request must agree. + +## Owed + +- Issue #1588 owes its first-line `Row: BACKEND-ROCM` metadata before landing. +- A fresh implementer owes the state probes, tests, comparator changes, and + three-mode end-to-end gate. +- A fresh reviewer owes static review and every mutation in this spec. +- The operator owes the controlled build and the RX 7900 XTX hardware run. +- The operator owes the pinned vLLM production runs for all cache modes. +- The final implementation change owes an `## Outcome` in this spec. +- Issue #332 owns the graph-enabled repetition after ROCm graph activation. From 389f9707f8cad3a627bd94135cdc334bebbdd3b8 Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Fri, 4 Sep 2026 13:57:29 -0700 Subject: [PATCH 2/8] spec(BACKEND-ROCM): repair Qwen3.5 numerical plan The rejected plan treated a red local gate as usable, invented a numerical envelope, and described oracle and provider paths that could not run. Bind the work to #2773, keep both correctness prerequisites pending, and make the future evidence recipe executable without claiming unavailable results. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6 [codex] --- .../rocm-qwen35-08b-cpu-gfx1100-numerics.md | 981 +++++++----------- 1 file changed, 365 insertions(+), 616 deletions(-) diff --git a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md index c6ae9756e1..f0303ad3c2 100644 --- a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md +++ b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md @@ -2,360 +2,205 @@ Row: `BACKEND-ROCM`. -Issue: [#1588](https://github.com/mudler/vllm.cpp/issues/1588). - -Base: `4d10c8acc527c34a6a58a309d52ea5f8fbd1d47b`. +Issue: [#2773](https://github.com/mudler/vllm.cpp/issues/2773), which supersedes +issue #1588 for implementation traceability. Branch: `row/BACKEND-ROCM-NUMERICS-1588`. -Primary oracle: vLLM at `5559679229bc961848b121ccdeaa8fa5d79bec98`. - -Target: AMD Radeon RX 7900 XTX, `gfx1100`. +Active primary oracle pin: vLLM +`e126687a9a828d513c01a07cd69f025f27d63280`. ## Now -The issue is open and the implementation has not started. This spec is the -committed prerequisite for the implementation. - -Current `main` runs the Qwen3.5-0.8B paged engine on ROCm. The backend matrix -still names its CPU and ROCm numerical characterization as open. - -The current gate runs one default cache configuration. It does not characterize -`auto`, `bfloat16`, and `fp8_e4m3` separately. - -The current dump records the residual halves and named layer stages. It does not -record full-attention cache writes or persistent Gated Delta Net state writes. - -ROCm static graph mode remains false in `src/vllm/platforms/rocm.cpp:91-98`. -This work characterizes the current eager path before issue #332 activates model -graph replay. - -## Live gap - -The spec implementer checked the pinned base before writing this spec. - -- `HEAD`, `origin/main`, and the merge base all resolve to the pinned base. -- `git log --all --grep=1588` finds no landed commit for issue #1588. -- No spec with this scope exists on the pinned base. -- GitHub reports issue #1588 as open. -- No open pull request owns issue #1588. -- The issue comment describes an older Q4_K_M experiment at `e2a9e035d`. -- That experiment predates the current checkpoint gate and the later W1 fix. -- That experiment does not compare persistent state or cache modes. - -The current production gate is -`tests/parity/test_qwen35_paged_engine.cpp:159-484`. It loads the model through -`LoadedEngine::FromModelDir` and exercises the paged engine. - -The gate checks 16 prompts against a pinned ROCm vLLM oracle. It reports strict -token agreement and applies the ratified 500 milli-nat near-tie rule. - -The gate constructs `EngineParams{}` at -`tests/parity/test_qwen35_paged_engine.cpp:231-234`. It therefore covers only -the resolved default cache dtype. - -The gate checks native selection and zero declines for 15 required operators at -`tests/parity/test_qwen35_paged_engine.cpp:254-275` and `:422-446`. It does not -run the required three-mode matrix. - -The current dump writer lives in -`include/vllm/model_executor/models/act_dump.h:55-227`. Its manifest key is -`(step, layer, stage)` and its schema is: - -```text -step layer stage dtype rows cols bytes file -``` - -`DenseForwardLayers` records `hidden` and `res` before layer 0 and after every -layer at `src/vllm/model_executor/models/qwen3_5.cpp:9352-9403`. - -`RunDenseLayerPaged` records `block_out` and `mlp_out` at -`src/vllm/model_executor/models/qwen3_5.cpp:7700-7772`. On a full-attention -layer, `block_out` is the complete attention output after its output projection. - -These existing rows answer the residual-stream, attention-output, and -multilayer perceptron output questions. The comparator must reconstruct the -residual stream as `hidden + res` in FP32. - -The full-attention store calls `dense_attn::WriteKvCache` at -`src/vllm/model_executor/models/qwen3_5.cpp:5894-5905`. No dump reads the -destinations after that call. - -GDN prefill scatters working state at -`src/vllm/model_executor/models/qwen3_5.cpp:5280-5304` and `:5454-5495`. -GDN decode can update persistent state in place at `:5319-5334` and -`:5431-5445`. - -No existing row records the state after all four write paths. Dumping an FP32 -working buffer would not answer what the persistent cache stores. +`PENDING`. This repaired spec is the only completed deliverable in this commit. +Implementation cannot start until a fresh reviewer passes this immutable spec. +Results cannot be accepted until both correctness prerequisites pass: + +1. Issue #2772 and PR #2856 own the default-on ROCm `wvSplitK` sacred-anchor + repair. The reviewed old base failed prompt 10/token 10 (`369` instead of + `488`), while `VT_ROCM_SKINNY=0` passed. The operator reran PR #2856 at + `f06619e4c213e3de28359ee10995e682e8c06932`: CPU mode 2/2, prerequisites + 77/77/77/86, `wvSplitK` 79796/79796, and sacred 137/137 with 15 strict, one + tied, maximum gap zero, and zero divergence. PR #2856 is still open and the + contributor lacks merge authority. This prerequisite remains `PENDING` + until a maintainer lands it and the unchanged default gate passes on the + implementation base. Disabling skinny GEMM is diagnostic only. +2. The configured source and runnable ROCm wrapper are still at historical + vLLM `5559679229bc961848b121ccdeaa8fa5d79bec98`. The active source object is + available for read-only inspection, but no runnable active-pin ROCm runtime + has been supplied or proved. Issue #2794 records repository pin-validation + context; #2773 itself owns the cache-matched active-pin Qwen3.5-0.8B captures. + Active-pin capture and token revalidation remain `PENDING` until that runtime + exists and runs the model under the GPU mutex. + +Do not create characterization goldens from a known-regressed local default or +from the historical oracle revision. ## Scope ### In scope -1. Keep the existing `VT_DUMP_ACT` writer and manifest as the dump surface. -2. Keep `VT_DUMP_ACT_SUB` as the switch for named sub-stage rows. -3. Add no state-specific environment variable. -4. Record active full-attention K and V destinations after the production store. -5. Record active GDN convolution and SSM rows after the production update. -6. Compare CPU and ROCm on identical model inputs. -7. Report `max_abs`, root mean square error, and relative L2 error for each key. -8. Audit every relevant local dtype against the pinned upstream executing chain. -9. Report the byte cost of each local dtype and each upstream difference. -10. Gate the ROCm paged engine in `auto`, `bfloat16`, and `fp8_e4m3` modes. -11. Prove that each ROCm mode has zero reference-tier hits. -12. Run the final acceptance on one RX 7900 XTX with architecture `gfx1100`. +- Add opt-in production-boundary dumps for the residual stream, attention + output, MLP output, stored K/V cache, and persistent GDN convolution/SSM + state. +- Compare CPU and gfx1100 on identical weights and identical token prefixes. +- Report descriptive `max_abs`, RMS, and relative-L2 deltas. +- Port applicable active-pin upstream operation tests with their parameters, + modes, fixtures, failure cases, and exact tolerances. +- Audit runtime dtype, byte width, requested/resolved cache dtype, selected + attention backend, and provider counts for `auto`, `bfloat16`, and + `fp8_e4m3`. +- Run cache-matched active-pin vLLM captures and the established end-to-end + oracle-backed token/near-tie gate after the prerequisites pass. ### Out of scope -- Enabling ROCm static graph mode. Issue #332 owns that change. -- Changing a model dtype before the characterization identifies a defect. -- Changing a kernel reduction order to make raw state bytes equal. -- Adding a new activation-dump environment variable. -- Dumping unused KV blocks or unused recurrent-state slots. -- Throughput, latency, and power measurements. -- A CUDA comparison. -- A GGUF or quantized-weight model arm. -- A public benchmark claim. -- A fix for a newly found numerical defect without its required issue flow. +- Changing a numerical tolerance after observing results. +- Treating CPU as the conformance oracle for FP8 physical-cache behavior. +- Permanently expanding the sacred 16-prompt gate merely because this + characterization measures three cache modes. A new permanent case or golden + may land only when #2773's active-pin capture proves it is necessary and the + final implementation records the evidence. Broader gate policy needs its own + issue. +- Enabling ROCm static graph mode; issue #332 owns that work. +- Performance claims or tuning. ## Artifact and oracle pins -Use this exact model snapshot for every local and oracle run: +Use `Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17` +from `${CHECKPOINT_ROOT}/Qwen3.5-0.8B`. Before every model run, require: -```text -repository: Qwen/Qwen3.5-0.8B -revision: 2fc06364715b967f1860aea9cf38778875588b17 -path: /home/vikash/models/Qwen3.5-0.8B +```sh +sha256sum \ + "${CHECKPOINT_ROOT}/Qwen3.5-0.8B/model.safetensors-00001-of-00001.safetensors" \ + "${CHECKPOINT_ROOT}/Qwen3.5-0.8B/model.safetensors.index.json" \ + "${CHECKPOINT_ROOT}/Qwen3.5-0.8B/config.json" ``` -The local checkpoint inspection found these hashes: +Expected hashes are: ```text -model.safetensors 04b1c301231dd422b8860db31311ab2721511346a32cb1e079c4c4e5f1fe4696 -config.json b90b86f35c8e6925ef74ee04d0e758f0a845c83a42089ad82bbaa948de9b4204 +04b1c301231dd422b8860db31311ab2721511346a32cb1e079c4c4e5f1fe4696 model.safetensors-00001-of-00001.safetensors +d8a08838a613b025eb7952ed9db11696213e57e76a375661ef5c12f9dd5dcf4e model.safetensors.index.json +b90b86f35c8e6925ef74ee04d0e758f0a845c83a42089ad82bbaa948de9b4204 config.json ``` -The model file is 1,746,942,600 bytes. Its safetensors header declares 452 BF16 -tensors and 36 FP32 tensors. - -`tests/parity/hf_snapshot.h` pins the same revision and exposes -`parity::Qwen35_08BSnapshot()`. An explicit snapshot override checks existence, -not revision identity. - -The characterization must hash the two files before every run. A hash mismatch -is `ARTIFACT_MISMATCH`, not a skipped comparison. - -Use the vLLM source checkout at `/home/vikash/oracle/vllm-src`. Its detached -`HEAD` must equal `5559679229bc961848b121ccdeaa8fa5d79bec98`. - -Run the pinned vLLM oracle on the identical model, prompt set, cache mode, -sampling configuration, and token count. Use production configuration without -`--enforce-eager` for the end-to-end denominator. - -The internal CPU and ROCm comparison remains eager on both local arms. The dump -synchronizes the queue and cannot run inside graph capture. - -## Model geometry +Resolve the active oracle source with: -The pinned checkpoint declares this text-model geometry: - -| Field | Value | -|---|---:| -| Model dtype | `bfloat16` | -| Hidden size | 1,024 | -| Decoder layers | 24 | -| Full-attention interval | 4 | -| Full-attention layers | 6 | -| GDN layers | 18 | -| Query heads | 8 | -| KV heads | 2 | -| Attention head dimension | 256 | -| GDN key heads | 16 | -| GDN value heads | 16 | -| GDN key dimension | 128 | -| GDN value dimension | 128 | -| GDN convolution kernel | 4 | -| GDN SSM dtype | `float32` | +```sh +git -C "${VLLM_SOURCE}" rev-parse e126687a9a828d513c01a07cd69f025f27d63280^{commit} +``` -The full-attention layers are 3, 7, 11, 15, 19, and 23. Every other layer is a -GDN layer. +The configured checkout may remain detached at the historical revision. Read +active-pin files with `git -C "${VLLM_SOURCE}" show :`; do not move +that checkout. Source availability does not prove runtime gateability. ## Upstream executing chain -The implementation must preserve these pinned upstream decisions. - -- `vllm/model_executor/models/qwen3_5.py:471-531` routes the text model and - publishes its recurrent state dtype. -- `vllm/model_executor/models/qwen3_next.py:389-400` projects Q, K, and V, then - calls attention before the output projection. -- `vllm/model_executor/models/qwen3_next.py:492-550` keeps the residual pair and - routes each layer to full attention or GDN. -- `vllm/model_executor/models/config.py:744-768` copies checkpoint - `mamba_ssm_dtype` into the default SSM cache dtype. -- `vllm/model_executor/layers/mamba/mamba_utils.py:96-128` resolves the - convolution and temporal-state dtypes independently. -- `vllm/model_executor/layers/mamba/mamba_utils.py:180-199` defines the two GDN - state shapes. -- `vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py:1228-1308` - identifies the persistent convolution state and its indexed write paths. -- `vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py:1372-1478` - identifies the persistent SSM updates and the prefill scatter. -- `vllm/v1/attention/ops/paged_attn.py:31-50` calls the paged KV store with the - slot mapping and cache scales. - -The local mirror resolves the cache layout in -`src/vllm/model_executor/models/qwen3_5_common.cpp:37-90`. The runner consumes -that layout at `src/vllm/v1/worker/gpu/runner.cpp:930-999` and `:1747-1793`. - -The local full-attention store routes through -`include/vllm/model_executor/models/kv_cache_route.h:48-67`. The float and FP8 -arms reach different store operators through this shared seam. - -The implementation must inspect the complete executed kernel chain for each -mode. A dispatch wrapper alone does not establish a compute dtype. - -## Dtype and byte contract - -The checkpoint and pinned upstream source establish these storage types before -any measurement: - -| Surface | Shape per active unit | Required storage | Bytes per unit | -|---|---|---|---:| -| Hidden state | `[T, 1024]` | BF16 | `2048 * T` | -| Residual half | `[T, 1024]` | BF16 | `2048 * T` | -| Full-attention K | `[T, 2, 256]` per layer | BF16 or FP8 E4M3 | `1024 * T` or `512 * T` | -| Full-attention V | `[T, 2, 256]` per layer | BF16 or FP8 E4M3 | `1024 * T` or `512 * T` | -| GDN convolution state | `[1, 6144, 3]` per layer | BF16 | 36,864 | -| GDN SSM state | `[1, 16, 128, 128]` per layer | FP32 | 1,048,576 | - -`auto` and `bfloat16` both store full-attention K and V in BF16. Across six -full-attention layers, they use 12,288 bytes per cached token. - -`fp8_e4m3` stores full-attention K and V in one byte per element. Across six -layers, it uses 6,144 bytes per cached token. - -FP8 therefore removes 6,144 bytes per cached token from this model's -full-attention cache. The report must confirm the allocated bytes from the -resolved `FullAttentionSpec` instead of trusting this arithmetic alone. - -Across 18 GDN layers, convolution state uses 663,552 bytes per active state -slot. SSM state uses 18,874,368 bytes per active state slot. - -The required GDN total is 19,537,920 bytes per active state slot. Storing the -SSM state in BF16 would remove 9,437,184 bytes but would violate this checkpoint. - -The dtype audit must include each model-path buffer and each GEMM output that -feeds a compared boundary. For each item, record: - -1. The local symbol and runtime dtype. -2. The pinned upstream symbol and runtime dtype. -3. The tensor shape and active element count. -4. The local bytes and upstream bytes. -5. The byte difference at the measured token count. -6. The source annotation for every local FP32 exception. - -The audit includes the attention query, K and V before storage, attention -output, GDN projections, GDN working state, MLP output, residual halves, stored -KV, and stored GDN state. - -A wider local dtype is not accepted because tokens agree. Classify it as -`DTYPE_WIDTH_MISMATCH` unless a committed source annotation names the upstream -reason for the FP32 exception. +All cross-file anchors name symbols and refer to active pin `e126687a9` unless +explicitly labeled historical: + +- `vllm/model_executor/models/qwen3_5.py::Qwen3_5ForCausalLM` defines the text + model and layer composition. +- `vllm/model_executor/layers/mamba/mamba_utils.py::MambaStateShapeCalculator.gated_delta_net_state_shape` + defines the GDN convolution and SSM state shapes. It starts at line 258 at + the active pin. `MambaStateShapeCalculator.mamba2_state_shape` is a different + architecture and is not the anchor. +- `tests/kernels/mamba/cpu/test_cpu_gdn_ops.py::test_fused_sigmoid_gating_delta_rule_update_cpu` + and `test_chunk_gated_delta_rule_cpu` define CPU operation tolerances. +- `tests/kernels/mamba/test_gdn_forward_core_split.py::test_forward_core_split_matches_unified` + defines split/unified state and output comparisons. +- `tests/kernels/attention/test_cache.py::test_reshape_and_cache` defines + cache-store dtype behavior and FP8 comparison tolerances. +- `tests/kernels/attention/test_cpu_attn.py::_run_varlen_with_paged_kv` + defines cache-mode-aware CPU attention tolerances; its `_FP8_ATOL` and + `_FP8_RTOL` constants select the FP8 bounds. + +The earlier spec cited these surfaces from historical `555967922`; those +citations remain diagnostic history but cannot satisfy the active-pin gate. The +implementer must refresh line evidence from the active object and run the active +runtime before acceptance. + +The local chain is +`ModelRegistry::Forward` -> `dense_attn::AttnBlock` -> +`dense_attn::WriteKvCache` -> `vt::ReshapeAndCache` or +`vt::ReshapeAndCacheFp8`, plus the Qwen3.5 GDN prefill/decode update arms. Cite +the executing local and active-pin upstream symbols in the evidence. + +## Dtype and state contract + +The checkpoint resolves model dtype BF16 and explicitly sets +`mamba_ssm_dtype=float32`. The measured surfaces are: + +| Surface | Shape per active unit | Physical storage | +|---|---|---| +| Hidden/residual | `[T, 1024]` | BF16 | +| Full-attention K/V | `[T, 2, 256]` per layer | BF16 or FP8 E4M3 | +| GDN convolution state | `[1, 6144, 3]` per layer | BF16 | +| GDN SSM state | `[1, 16, 128, 128]` per layer | FP32 | + +Across six full-attention layers, BF16 K/V use 12,288 bytes per cached token +and FP8 K/V use 6,144 bytes. Across 18 GDN layers, convolution state uses +663,552 bytes and SSM state uses 18,874,368 bytes per active slot. Confirm +allocated bytes from runtime specs; arithmetic alone is not evidence. + +For every compared buffer and GEMM output, log the local symbol, active-pin +upstream symbol, runtime dtype, shape, active elements, bytes on each side, and +the reason beside every FP32 exception. An unjustified wider local dtype is a +`DTYPE_WIDTH_MISMATCH` even when tokens agree. ## State dump design -Enable state rows only when `VT_DUMP_ACT` and `VT_DUMP_ACT_SUB` name the same -writable directory. Use the existing writer and the existing manifest. - -The existing `hidden`, `res`, `block_out`, and `mlp_out` keys do not change. -Add these four stage keys: +Enable dumps only when `VT_DUMP_ACT` and `VT_DUMP_ACT_SUB` name the same empty, +writable directory. Preserve existing keys and add: -| Stage key | Production source | On-disk dtype | Manifest shape | +| Key | Production source | Dump dtype | Shape | |---|---|---|---| -| `state_fa_k` | K destination after `WriteKvCache` | FP32 | `[active_tokens, 512]` | -| `state_fa_v` | V destination after `WriteKvCache` | FP32 | `[active_tokens, 512]` | -| `state_gdn_conv` | Persistent convolution cache after update | FP32 | `[active_requests, 18432]` | -| `state_gdn_ssm` | Persistent SSM cache after update | FP32 | `[active_requests, 262144]` | - -FP32 is the canonical comparison format on disk. It does not change the cache -storage type. - -For BF16 state, read the persistent destination and widen it to FP32. For FP8 -KV, read the stored byte and dequantize it with that layer's recorded scale. - -The run log must state the physical storage dtype, K scale, and V scale. The -dtype audit records the physical byte width independently of the FP32 dump. - -Gather full-attention rows from nonnegative `slot_mapping` entries only. Preserve -the input token order instead of sorting by physical slot. - -Gather GDN rows from the active state indices for the current step. Preserve the -request order used by the scheduler metadata. - -Deduplicate no row silently. Refuse duplicate destination indices unless the -executing metadata defines their update order. - -Do not download an entire KV block pool. Do not download an entire GDN state -pool. A single-request run must write one GDN row per layer per step. - -Place the full-attention probe immediately after the shared production store. -The queue ordering must make the probe read the stored destination. - -Place the GDN probe at one shared tail after all prefill and decode update arms. -The probe must gather from `state.conv_state` and `state.ssm_state`. - -Do not dump `dcs`, `dss`, `kw`, or `vw` as substitutes. Those tensors precede a -downcast, quantization, scatter, or indexed in-place update. - -Extend the per-step narrative with separate stream, stage, and state counts. A -24-layer step must report 48 state blobs, with two blobs from each layer. - -An enabled state dump that writes fewer than 48 required blobs must refuse the -run. The manifest row count and active-row shapes remain the stronger capacity -checks. - -## Comparison workload - -Use one binary and one checkpoint copy for both local arms. Set -`EngineParams::device` to `kCPU` for the CPU arm. Use the detected ROCm platform -for the device arm. - -Use one request at concurrency 1. Use the first standard gate prompt: - -```text -The capital of France is -``` - -Use greedy sampling, a fixed seed, MTP disabled, and eight output tokens. Keep -the block size, block count, and scheduler token budget equal on both arms. - -Run each cache mode in a separate process and a separate empty dump directory. -This avoids static environment parsing and file-name collisions. - -Run each local arm twice with dumps enabled. The two repeats establish the -run-to-run floor. - -Run an additional dump-disabled control for each arm and mode. Its token IDs and -logits must equal the corresponding enabled run. - -Compare prefill and decode steps only when both arms consume the same token IDs. -The CPU denominator and ROCm treatment must share the complete input prefix. - -If greedy outputs diverge, include the first divergent output step. Exclude every -later step because those steps consume different inputs. - -Record the excluded step count. Do not compare equal step ordinals after the -input prefixes diverge. - -If no decode step has an identical input prefix, classify the end-to-end failure -first. Do not use a re-prefilled prefix as evidence for incremental state. - -## Metrics and advance policy - -Decode every compared blob to FP32. Let `A` be ROCm and let `B` be CPU. - -For `N` elements, report these metrics for every joined key: +| `state_fa_k` | stored K after `dense_attn::WriteKvCache` | FP32 | `[active_tokens, 512]` | +| `state_fa_v` | stored V after `dense_attn::WriteKvCache` | FP32 | `[active_tokens, 512]` | +| `state_gdn_conv` | persistent convolution cache after update | FP32 | `[active_requests, 18432]` | +| `state_gdn_ssm` | persistent SSM cache after update | FP32 | `[active_requests, 262144]` | + +Widen stored BF16 to FP32. Dequantize stored FP8 with the recorded per-layer +scale. The FP32 file format never conceals separately logged physical dtype and +allocation. + +Gather nonnegative slot mappings in input-token order and active GDN indices in +scheduler request order. Refuse unexplained duplicate destinations, missing +rows, partial joins, or capacity-wide dumps. A 24-layer step emits 48 state +blobs, two per layer. Place each probe after its production write and preserve +queue ordering. + +## Workload and backend identity + +Use one binary and checkpoint copy. Run one request at concurrency one with +`The capital of France is`, greedy sampling, seed 0, MTP disabled, and eight +output tokens. Keep block size, block count, and scheduler budget equal. Run +each arm and cache mode in a separate process and empty directory. Take two +enabled repeats and one dump-disabled identity control. + +Log and assert requested cache dtype, resolved cache dtype, physical cache +dtype, selected attention backend, and selected cache-store operator for every +arm and mode. Do not assume backend identity from device alone. The current CPU +provider accepts `auto` and `fp8_e4m3` in `CPU_ATTN` but omits explicit +`bfloat16`; that mode falls through to `FLASH_ATTN`, which accepts it. Assert +`CPU_ATTN` for `auto` and `fp8_e4m3` and `FLASH_ATTN` for explicit `bfloat16` +on this implementation base. If main changes the selector before implementation, +re-read and cite the source, update this spec before code, and freshly review +the changed contract. + +CPU is a diagnostic comparison arm for CPU-versus-ROCm deltas, including FP8 +state after dequantization. Active-pin vLLM and its upstream tests are the +conformance authority. CPU results cannot accept or reject FP8 physical-cache +semantics. + +Join only steps with identical input token prefixes. Include the first divergent +output step and exclude all later steps. Record the first divergence and number +of excluded steps. + +## Descriptive metrics and acceptance + +Decode joined blobs to FP32. Let `A` be ROCm and `B` be CPU. Report: ```text max_abs = max_i(abs(A_i - B_i)) @@ -363,323 +208,227 @@ rms = sqrt(sum_i((A_i - B_i)^2) / N) rel_l2 = sqrt(sum_i((A_i - B_i)^2)) / sqrt(sum_i(B_i^2)) ``` -If both vectors have zero L2 norm, define `rel_l2` as zero. If only `B` has zero -L2 norm, define `rel_l2` as infinity. - -The report must name directory A and directory B in words. It must report both -manifest row counts, joined rows, rejected rows, and excluded post-divergence -steps. - -The comparator must refuse missing keys, duplicate keys, dtype mismatches, -shape mismatches, byte-count mismatches, nonfinite values, and partial joins. - -Raw byte equality is not the cross-device acceptance rule. CPU and ROCm kernels -can use different valid reduction orders. - -Use these unit roundoffs: - -```text -u(FP32) = 2^-24 -u(BF16) = 2^-8 -u(FP8 E4M3) = 2^-4 -``` - -Before reading cross-device results, finish the dtype audit. For each comparison -key `k`, record `n(k,d)`, the executed round-to-nearest terms at dtype `d`. - -Count a destination store as one term. Count each reduction term at its actual -accumulator dtype. Cite the executing kernel for every count. - -Freeze the audit file hash before the comparator reads either cross-device dump. -Compute this three-root-mean-square envelope: +Define relative L2 as zero when both norms are zero and infinity when only `B` +has zero norm. Report the worst index, both manifests, joined/rejected rows, and +excluded steps. Refuse missing or duplicate keys, dtype/shape/byte mismatches, +nonfinite values, and partial joins. + +These layer metrics are descriptive. They have no propagated numerical +acceptance envelope, discontinuity multiplier, or post-measurement threshold. +The rejected `E(k)` construction is removed because operation counts do not +bound cancellation, conditioning, nonlinear sensitivity, correlated +reductions, FP8 scaling/saturation, or subnormals. + +Acceptance consists of both: + +1. Ported active-pin operation tests with their exact upstream per-surface + tolerances. Examples include GDN `atol=rtol=1e-2` where + `test_cpu_gdn_ops.py` specifies it, exact convolution-state equality in the + split-core test, its dtype-specific output/SSM tolerances, and cache/CPU + attention FP8 tolerances selected by the upstream fixtures. Do not collapse + them into one project-wide tolerance. +2. The established end-to-end 16-prompt strict-token and ratified 500 + milli-nat near-tie gate against deterministic, cache-matched, active-pin + vLLM captures. A new physical cache mode needs its own capture and + teacher-forced gaps before it can be accepted. + +If no upstream analogue exists for a layer boundary, report its metrics only. +A future stage-level acceptance rule requires an independently justified bound +or explicit developer ratification before values are inspected. + +## Provider and end-to-end contract + +For `auto` and `bfloat16`, require the established native operator set, +including nonzero `kReshapeAndCache`, zero `kReshapeAndCacheFp8`, zero declines, +and zero reference-tier hits. The two modes resolve to physical BF16 and must +produce identical local token streams. + +For `fp8_e4m3`, replace the store requirement: require nonzero +`kReshapeAndCacheFp8`, zero `kReshapeAndCache`, nonzero native selections for +all other applicable operators, zero declines, and zero reference-tier hits. +Provider sets are mode-specific; never require a BF16 store from a correct FP8 +run. + +The existing gate remains the permanent gate until #2773 produces reviewed +active-pin evidence for an additional case. Characterization must run all three +modes, but this spec alone does not authorize new permanent goldens. + +## Required capture-tool implementation + +The current `scripts/qwen3-oracle-capture.py` and +`scripts/qwen3-neartie-gap.py` hard-code `enforce_eager=True` and expose no +cache-dtype argument. Before active-pin capture, add reviewed options equivalent +to: ```text -E(k) = 3 * sqrt((2 / 3) * sum_d(n(k,d) * u(d)^2)) +--kv-cache-dtype {auto,bfloat16,fp8_e4m3} +--execution-mode {production,eager} +--seed INT +--max-tokens INT +--repetitions INT +--model-revision REV +--vllm-revision REV +--provenance-out PATH ``` -The `sqrt(2)` term models two independent rounding paths. The `1 / sqrt(3)` term -is the root mean square error of round-to-nearest. - -A key is inside the pre-registered dtype envelope only when both conditions hold: - -```text -rel_l2 <= E(k) -rms <= E(k) * sqrt(sum_i(B_i^2) / N) -``` - -Report `max_abs` and the worst index for diagnosis. Do not invent a fixed -absolute threshold after the values are visible. - -For ordered layer boundaries, compute the nonnegative squared-error increment: - -```text -delta(k) = max(0, rel_l2(k)^2 - rel_l2(input(k))^2) -``` - -Classify a discontinuity when `delta(k)` exceeds `E(k)^2` and four times the -median increment for the same stage and layer family. Freeze this factor before -the hardware run. - -The envelope classifies a result. It does not replace the end-to-end correctness -gate. - -## Mismatch classification - -Classify every run as exactly one primary result: - -- `ARTIFACT_MISMATCH`: a source revision, model hash, or run configuration differs. -- `INSTRUMENTATION_FAIL`: an enabled dump is incomplete or changes model output. -- `NONDETERMINISTIC`: a same-arm repeat has a nonzero metric. -- `STRUCTURE_MISMATCH`: key sets, active indices, shapes, or physical dtypes differ. -- `DTYPE_WIDTH_MISMATCH`: local storage is wider than upstream without an accepted annotation. -- `NONFINITE`: one arm creates a NaN or infinity that the other arm does not create. -- `WITHIN_DTYPE_ENVELOPE`: every key meets the frozen envelope and has no discontinuity. -- `ORDERING_DRIFT`: tokens pass, but one numerical key exceeds the frozen envelope without a structural error. -- `NUMERICAL_DEFECT_CANDIDATE`: a state or layer discontinuity identifies a bounded production region. -- `CORRECTNESS_FAIL`: the applicable end-to-end token gate fails. - -Apply the list in this precedence order: - -1. `ARTIFACT_MISMATCH`. -2. `INSTRUMENTATION_FAIL`. -3. `NONDETERMINISTIC`. -4. `STRUCTURE_MISMATCH`. -5. `DTYPE_WIDTH_MISMATCH`. -6. `CORRECTNESS_FAIL`. -7. `NONFINITE`. -8. `NUMERICAL_DEFECT_CANDIDATE`. -9. `ORDERING_DRIFT`. -10. `WITHIN_DTYPE_ENVELOPE`. - -`CORRECTNESS_FAIL` therefore outranks every interpretable numerical label. -`INSTRUMENTATION_FAIL` and `NONDETERMINISTIC` stop the cross-device reading. - -`WITHIN_DTYPE_ENVELOPE` does not promise equal state bytes. It means the measured -difference fits the dtype-derived rounding model and the token gate passes. - -`ORDERING_DRIFT` does not authorize a wider tolerance. Record the exact stage, -kernel chain, and token margin before any change. +`production` must instantiate vLLM without `enforce_eager=True`; eager is a +diagnostic arm and never the denominator. The output must record source, wheel, +image and artifact hashes, complete arguments, a hash of the scripts' shared +16-entry `PROMPTS` list, batching, concurrency, sampling, seed, token count, +repetitions, cache mode, resolved +cache dtype, execution mode, and output hash. -## End-to-end correctness gate - -Extend the current Qwen3.5 paged-engine gate with named cases for: - -1. `auto`. -2. `bfloat16`. -3. `fp8_e4m3`. - -Each case must set `EngineParams::kv_cache_dtype` explicitly. The test name and -log must print the requested and resolved cache dtype. - -`auto` and `bfloat16` resolve to the same physical BF16 cache. Require their -local token streams to match each other exactly. - -Capture a pinned vLLM oracle pair for each distinct physical cache mode. Do not -reuse the BF16 teacher-forced gaps for FP8. - -For each mode, run the existing 16-prompt battery at its current token count. -Keep the current strict-token report and ratified 500 milli-nat near-tie rule. - -The pinned vLLM oracle must use the same cache mode. Capture at least 10 -per-prompt greedy repeats and require determinism before creating a golden. - -Every ROCm case must prove all 15 required operators have nonzero native -selections and zero declines. It must also assert `reference_tier_hits=0`. - -The state-dump characterization does not need to run the 16-prompt battery with -dumps enabled. The narrow workload supplies internal evidence without dumping -inactive capacity. - -## Tests and red-first evidence - -The implementation starts with tests that fail because the four state rows are -absent. Capture the red output before editing production code. - -Add focused tests for these contracts: - -1. The state writer emits all four keys with the declared shape and FP32 dtype. -2. A nonmonotonic slot mapping preserves input token order. -3. Negative slot mappings produce no KV row. -4. Inactive KV blocks and GDN slots produce no bytes. -5. A BF16 state row is widened from the stored destination. -6. An FP8 row is read after quantization and dequantized with its scale. -7. GDN prefill and decode both read persistent post-write state. -8. A missing required state row makes the per-step floor refuse. -9. The comparator refuses duplicate, missing, partial, short, and nonfinite data. -10. The comparator implements the zero-denominator rules exactly. -11. The dtype audit rejects an unannotated wider local dtype. -12. Each cache-mode gate prints its requested and resolved dtype. -13. The ROCm gate rejects one injected reference-tier hit. - -The checkpoint-backed focused test must enter through -`LoadedEngine::FromModelDir`. A hand-built `PagedKvCache` test can test the -gather helper but cannot prove production reachability. - -Run the focused CPU tests before hardware work. Run the ROCm cases only inside -the required GPU lease and file mutex. - -`IMP-TEST-FIRST` and `IMP-MUTATE` do not apply to this spec-only commit. This -commit changes no behavior that a test or mutation can falsify. - -They apply to the later implementation. Preserve each red result and each -mutation result in that implementation's evidence. - -## Review mutations - -A fresh reviewer reviews one immutable implementation head. The reviewer uses a -scratch copy and restores every file byte for byte after each mutation. - -Run these mutations separately: - -1. Delete the production probe after `dense_attn::WriteKvCache`. -2. Delete the shared GDN post-write probe. -3. Replace the post-write KV read with `kw` and `vw`. -4. Replace the post-write GDN read with `dcs` and `dss`. -5. Remove negative-slot filtering. -6. Sort active rows by physical slot. -7. Dump the full allocated cache capacity. -8. Report the FP8 physical dtype as BF16. -9. Remove one cache-mode test case. -10. Force one ROCm operator to the reference tier. -11. Delete the `LoadedEngine::FromModelDir` production call from the focused test. - -Each mutation must make its named test fail for the intended reason. A green -reachability mutation is a review finding. - -Record the immutable head, scratch path, command, exit status, and restored tree -hash for each mutation. - -## Gates - -The later implementation must pass these gates in order: - -1. Confirm the checkpoint and oracle hashes. -2. Capture the red focused tests. -3. Pass the dump-writer and comparator tests on CPU. -4. Pass the checkpoint-backed CPU trace case. -5. Run the full controlled preflight. -6. Get a fresh immutable review with all mutations detected. -7. Run two CPU and two ROCm trace repeats for each cache mode. -8. Pass the dump-disabled identity control for every arm and mode. -9. Pass all three ROCm paged-engine end-to-end cases. -10. Prove zero ROCm reference-tier hits in every case. -11. Have the operator rerun the focused, full, and hardware gates. - -Use this controlled environment for local repository gates: +Extend `test_qwen35_paged_engine` with future environment inputs +`VT_QWEN35_GATE_DIR` and `VT_QWEN35_KV_CACHE_DTYPE`. The first selects an empty +issue-evidence directory instead of the committed golden directory. The second +sets `EngineParams::kv_cache_dtype` and prints requested, resolved, and physical +dtype. These commands describe the intended interface and data flow after +those changes. They are future commands and will fail today: ```sh -PATH=/usr/local/bin:/usr/bin:/bin \ -PYTHONPATH=$PWD \ -GIT_CONFIG_GLOBAL=/dev/null \ -scripts/agent-preflight.sh +VLLM_PIN=e126687a9a828d513c01a07cd69f025f27d63280 +MODEL_REV=2fc06364715b967f1860aea9cf38778875588b17 +MODEL="${CHECKPOINT_ROOT}/Qwen3.5-0.8B" +for mode in auto bfloat16 fp8_e4m3; do + GOLDEN_DIR="evidence/2773/sacred-${mode}" + test ! -e "${GOLDEN_DIR}" + mkdir -p "${GOLDEN_DIR}" + "${VLLM_ORACLE}" scripts/qwen3-oracle-capture.py \ + --model "${MODEL}" --model-revision "${MODEL_REV}" \ + --vllm-revision "${VLLM_PIN}" \ + --kv-cache-dtype "${mode}" --execution-mode production --seed 0 \ + --max-tokens 16 --runs 10 --per-prompt --out-dir "${GOLDEN_DIR}" \ + --provenance-out "evidence/2773/oracle-${mode}.json" + VT_QWEN35_GATE_DIR="${GOLDEN_DIR}" \ + VT_QWEN35_KV_CACHE_DTYPE="${mode}" \ + VT_DUMP_IDS=1 build-rocm/tests/test_qwen35_paged_engine + test -s "${GOLDEN_DIR}/our_ids.i32" + "${VLLM_ORACLE}" scripts/qwen3-neartie-gap.py \ + --model "${MODEL}" --model-revision "${MODEL_REV}" \ + --vllm-revision "${VLLM_PIN}" \ + --kv-cache-dtype "${mode}" --execution-mode production --seed 0 \ + --max-tokens 16 --topk 20 --golden-dir "${GOLDEN_DIR}" \ + --provenance-out "evidence/2773/neartie-${mode}.json" + VT_QWEN35_GATE_DIR="${GOLDEN_DIR}" \ + VT_QWEN35_KV_CACHE_DTYPE="${mode}" \ + build-rocm/tests/test_qwen35_paged_engine +done ``` -An exit status of zero with skipped checks is not green. List each skip and its -reason. - -The hardware run must use the repository's lease procedure. Use the file mutex -inside the lease because the target is a non-fleet local GPU unless `rc devices` -reports it as a fleet device. +The tools must refuse when their `PROMPTS` lists differ from each other or from +`tests/parity/test_qwen35_paged_engine.cpp::Prompts`. Record exact as-run +commands. Do not create a golden unless all 10 repeats are deterministic. + +The internal CPU/ROCm state characterization remains eight output tokens as +specified under `Workload and backend identity`. The permanent sacred-gate +candidate uses the existing 16-prompt, 16-output-token regime above. Never use +the eight-token characterization files as sacred-gate goldens. + +## Tests and review mutations + +The later implementation starts with focused tests that fail because the four +state rows and capture options are absent. It must test row shape/dtype, +nonmonotonic and negative slot mappings, inactive-capacity exclusion, stored +BF16 widening, post-quantization FP8 dequantization, persistent GDN prefill and +decode state, incomplete-step refusal, comparator structural refusals and +zero-denominator rules, dtype-width refusal, backend/dtype logging, and +mode-appropriate provider sets. The checkpoint-backed case must enter through +`LoadedEngine::FromModelDir`. + +A fresh reviewer mutates each guarantee in a scratch copy: remove each +production probe, substitute pre-write tensors, remove negative-slot filtering, +sort by physical slot, dump capacity, misreport FP8 as BF16, swap the mode's +cache-store operator, inject a reference-tier hit, remove the production entry +point, and force eager oracle mode. Each focused test must fail for the intended +reason and the reviewer must restore the tree byte for byte. + +This spec repair changes no runtime behavior. `IMP-TEST-FIRST` and +`IMP-MUTATE` are future implementation/review gates, not evidence claimed by +this commit. + +## Gate order + +The later implementation must satisfy, in order: + +1. #2772 lands and the unchanged default local gate passes on the chosen base. +2. A runnable active-pin ROCm vLLM runtime is identified and proves this model. +3. Artifact, source, wrapper, wheel, image, prompt, and output hashes are saved. +4. The capture tools fail first for missing options, then pass focused tests. +5. Ported active-pin operation tests pass with unchanged upstream tolerances. +6. Dump/comparator CPU tests and checkpoint-backed CPU trace pass. +7. The controlled full preflight runs; every skip remains `PENDING`. +8. Fresh immutable review detects every required mutation. +9. Two CPU and two gfx1100 trace repeats plus disabled controls pass per mode. +10. Cache-matched active-pin production captures are deterministic and the + established end-to-end gate passes per measured mode. +11. Mode-appropriate native selections are nonzero, with zero declines and + zero reference-tier hits. +12. The operator independently reruns focused, full, oracle, and hardware gates. + +No GPU result, oracle result, model execution, runtime mutation, or +implementation test is claimed by this spec-only repair. ## Evidence required -Record this evidence for the implementation and hardware run: - -- The implementation commit and tree hashes. -- The clean source status before and after each gate. -- The exact checkpoint hashes and upstream revision. -- The compiler, build type, ROCm version, kernel driver, board name, and `gfx1100` architecture. -- The GPU lease identity or the non-fleet mutex evidence. -- The exact commands and exit statuses. -- Every preflight skip. -- The requested and resolved cache dtype for each run. -- The physical cache dtype and allocated bytes for each state surface. -- The operator-provider selection, decline, and reference-tier counts. -- Both enabled repeats and the disabled identity control. -- Both manifests and all raw dump paths. -- The frozen dtype-audit file and its hash. -- The full metric table for every joined key. -- The first token divergence and excluded step count. -- The selected mismatch classification. -- The red-first output and every review mutation result. -- The fresh reviewer report on the immutable head. -- The operator's independent gate rerun. - -Keep raw evidence under an issue-specific evidence path. Do not publish a result -from temporary files that the reviewer cannot inspect. - -The spec-only preflight ran at the pinned base with the controlled environment. -It returned zero and skipped 10 checks. - -Five LTX-2.5 checks skipped because NumPy was unavailable. Three ISA checks -required compile commands. The pull-request size check required base and head -arguments. The Triton AOT check required a vendored source root. - -This result is `PENDING`, not green. No source build, checkpoint inference, or -GPU gate applies to this spec-only commit. +Store evidence under an issue-specific durable path and record commit/tree +hashes, clean status, exact commands/statuses, every skip, active oracle and +artifact hashes, compiler/build/ROCm/driver/board identity, GPU mutex evidence, +requested/resolved/physical cache dtype, selected CPU and ROCm attention +backends, provider counts, enabled repeats, disabled controls, manifests, raw +dumps, metric tables, first divergence, excluded steps, upstream tolerance +anchors, red-first output, mutations, fresh review, and operator reruns. + +Classify structural and execution failures before interpreting metrics: +`ARTIFACT_MISMATCH`, `INSTRUMENTATION_FAIL`, `NONDETERMINISTIC`, +`STRUCTURE_MISMATCH`, `DTYPE_WIDTH_MISMATCH`, `CORRECTNESS_FAIL`, or +`NONFINITE`. When none applies, report descriptive metrics and the separate +operation/end-to-end acceptance results. Do not invent `WITHIN_DTYPE_ENVELOPE` +or `ORDERING_DRIFT` labels. ## Risks -- The dump can synchronize the queue and perturb scheduling. The disabled - identity controls detect output changes. -- FP8 K or V near a quantization boundary can change one stored code. Numeric - comparison occurs after dequantization and never assumes byte equality. -- GDN update kernels use different reduction orders on CPU and ROCm. The frozen - dtype envelope classifies that difference without widening after measurement. -- A greedy divergence makes later steps incomparable. The comparator excludes - those steps and reports the exclusion. -- The FP32 canonical dump can hide a physical-width error. The independent byte - audit and run narrative cover the storage width. -- A probe before the write can look plausible. The pre-write substitutions are - required review mutations. -- Static environment parsing can mix output directories. Separate processes and - empty directories prevent that collision. -- The current gate goldens describe the BF16 default. FP8 needs its own pinned - oracle capture and teacher-forced gaps. -- Issue #1588 currently lacks the required first-line `Row:` metadata. An - authorized operator must reconcile it before the pull request lands. -- Graph activation changes execution order and invalidates this characterization. - Issue #332 requires a new graph-on measurement. +- Probes can synchronize queues; disabled controls detect output perturbation. +- FP8 boundaries can change codes; compare dequantized values and record bytes. +- Different valid reduction orders can produce descriptive CPU/ROCm deltas. +- Backend fallback can change the diagnostic denominator; explicit backend + logging exposes it. +- Greedy divergence makes later states incomparable; exclude them. +- The active source object can be read while the active runtime remains absent; + never convert source availability into a gate pass. ## Stop conditions -- Stop with `NEEDS_CONTEXT` if any required revision, hash, model file, or board - identity is unavailable. -- Stop if the issue, spec, and pull request do not name `BACKEND-ROCM` together. -- Stop if the dump changes tokens or logits on either arm. -- Stop if a same-arm repeat is not deterministic. -- Stop if a manifest omits an active row or includes inactive capacity. -- Stop if the dtype audit is not frozen before cross-device results are read. -- Stop if the local chain cannot name an upstream dtype for a compared surface. -- Stop if any ROCm operator declines or reaches the reference tier. -- Stop if the running device is not an RX 7900 XTX with architecture `gfx1100`. -- Stop if ROCm static graph mode becomes true before the measurement. -- Stop before changing a tolerance after result values are visible. -- Stop before fixing an unexpected numerical defect without the required issue - and fresh implementation flow. -- Do not report a hardware pass from a skipped test. +- Stop before implementation until this spec receives a fresh `PASS`. +- Keep correctness `PENDING` until #2772 lands and the unchanged default gate + passes on the implementation base. +- Stop active-pin capture until a runnable active-pin ROCm runtime is proved. +- Stop on a revision/hash/configuration mismatch, dump perturbation, + nondeterminism, incomplete manifest, unexplained backend, provider decline, + reference-tier hit, wrong board, missing GPU mutex, or divergent input prefix. +- Stop before changing tolerances or permanent sacred-gate scope after seeing + results. +- File and assign a new issue before fixing any unexpected defect outside + #2773. ## Git integration -Use one pull request for the spec and implementation. This is the repository -policy default for `BACKEND-ROCM` and the selected shape for this row. - -Commit this spec before any implementation. A fresh implementer starts from the -committed spec. - -Keep the work on `row/BACKEND-ROCM-NUMERICS-1588`. The spec implementer does not -push or open the pull request. - -The pull request body must link issue #1588 and close it when the complete work -lands. The issue's `Row:` line, this spec, and the pull request must agree. +Use one pull request for the committed spec and later implementation, following +the recorded repository default. Preserve rejected spec commit `7bc2546e9` in +history. The eventual pull request body must name row `BACKEND-ROCM`, link and +close #2773, and carry the required trailers. The spec implementer does not +push, open, or merge that pull request. ## Owed -- Issue #1588 owes its first-line `Row: BACKEND-ROCM` metadata before landing. -- A fresh implementer owes the state probes, tests, comparator changes, and - three-mode end-to-end gate. -- A fresh reviewer owes static review and every mutation in this spec. -- The operator owes the controlled build and the RX 7900 XTX hardware run. -- The operator owes the pinned vLLM production runs for all cache modes. -- The final implementation change owes an `## Outcome` in this spec. -- Issue #332 owns the graph-enabled repetition after ROCm graph activation. +- A maintainer owes the merge decision for reviewed PR #2856; #2772 remains + pending until it lands. +- #2773 owes the runnable active-pin Qwen3.5-0.8B ROCm captures, even though + #2794 supplies repository sync context. +- A fresh reviewer owes this repaired spec a verdict. +- A fresh implementer owes the capture options, state probes, tests, comparator, + and provider checks after the prerequisites pass. +- A fresh implementation reviewer owes static review and every mutation. +- The operator owes the independent controlled and gfx1100 gates. +- The final implementation adds `## Outcome` with measured results, rejected + alternatives, and reasons for defaults. +- Issue #332 owns the graph-enabled repetition. From 9b7484f817be26c7ec55bb25fb9c7e733b5e0f37 Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Fri, 4 Sep 2026 15:53:15 -0700 Subject: [PATCH 3/8] spec(BACKEND-ROCM): correct Qwen3.5 capture contracts The #2773 plan must describe the production caller and active oracle layout before instrumentation starts. Correct BF16 selector normalization, name the existing Qwen3.5 path, and record its shared-seam debt in #2923. Separate SD storage from DS dump order and cite the active CPU attention test with its unchanged tolerances. Runtime acceptance remains pending. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [codex] --- .../rocm-qwen35-08b-cpu-gfx1100-numerics.md | 98 +++++++++++++++---- 1 file changed, 80 insertions(+), 18 deletions(-) diff --git a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md index f0303ad3c2..c4b7ddff86 100644 --- a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md +++ b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md @@ -112,20 +112,31 @@ explicitly labeled historical: defines split/unified state and output comparisons. - `tests/kernels/attention/test_cache.py::test_reshape_and_cache` defines cache-store dtype behavior and FP8 comparison tolerances. -- `tests/kernels/attention/test_cpu_attn.py::_run_varlen_with_paged_kv` - defines cache-mode-aware CPU attention tolerances; its `_FP8_ATOL` and - `_FP8_RTOL` constants select the FP8 bounds. +- `tests/kernels/attention/test_cpu_attn.py::varlen_with_paged_kv` starts at + line 415. Its `_FP8_ATOL` and `_FP8_RTOL` constants at lines 45 and 46 + set `atol` to `0.2` for E4M3 and `0.3` for E5M2, with `rtol=0.1`. + The comparisons at lines 629 to 653 select these bounds for FP8 and + `atol=1.5e-2`, `rtol=1e-2` otherwise. Preserve both split and unsplit checks. The earlier spec cited these surfaces from historical `555967922`; those citations remain diagnostic history but cannot satisfy the active-pin gate. The implementer must refresh line evidence from the active object and run the active runtime before acceptance. -The local chain is -`ModelRegistry::Forward` -> `dense_attn::AttnBlock` -> -`dense_attn::WriteKvCache` -> `vt::ReshapeAndCache` or -`vt::ReshapeAndCacheFp8`, plus the Qwen3.5 GDN prefill/decode update arms. Cite -the executing local and active-pin upstream symbols in the evidence. +The local production entry is `ModelRegistry::Forward`, which dispatches the +registered `ForwardQwen3_5Dense` factory. Its paged forward reaches +`DenseForwardBody` -> `DenseForwardLayers` -> `RunDenseLayerPaged` -> +`FullAttnBlockPaged` -> `dense_attn::WriteKvCache` -> `vt::ReshapeAndCache` or +`vt::ReshapeAndCacheFp8`. The full-attention caller is +`src/vllm/model_executor/models/qwen3_5.cpp:7758`. `FullAttnBlockPaged` starts +at line 5726 and writes through `dense_attn::WriteKvCache` at +`src/vllm/model_executor/models/qwen3_5.cpp:5909`, before `vt::PagedAttention` +at line 5931. Place the stored-K/V probes after that write. Place GDN probes +after the persistent writes in `GdnBlockPaged` for prefill and decode. + +This Qwen3.5 path does not call `dense_attn::AttnBlock`. Issue #2923 under +`Owed` records this existing shared-seam debt. Characterize the actual path +and cite the executing local and active-pin upstream symbols in the evidence. ## Dtype and state contract @@ -136,9 +147,23 @@ The checkpoint resolves model dtype BF16 and explicitly sets |---|---|---| | Hidden/residual | `[T, 1024]` | BF16 | | Full-attention K/V | `[T, 2, 256]` per layer | BF16 or FP8 E4M3 | -| GDN convolution state | `[1, 6144, 3]` per layer | BF16 | +| GDN convolution state | Per layer and slot: oracle SD `[3, 6144]`, local DS `[6144, 3]` | BF16 | | GDN SSM state | `[1, 16, 128, 128]` per layer | FP32 | +The active oracle defaults to SD physical convolution storage, with axes +`(state_len, dim)`. DS reverses these axes to `(dim, state_len)`. +`vllm/model_executor/layers/mamba/mamba_utils.py::get_conv_state_layout`, lines +28 to 44, returns `SD` without an override. `_orient_conv_shape`, lines 162 +to 166, and `gated_delta_net_state_shape`, lines 268 to 272, produce +`[3, 6144]` per slot for this workload. The kernel consumes a DS view +`[6144, 3]` after the transpose in +`vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py:1296` to line 1303. +Local `MakeQwen3_5KVCacheSpec` stores DS directly in +`src/vllm/model_executor/models/qwen3_5_common.cpp:85`. Leave +`VLLM_SSM_CONV_STATE_LAYOUT` unset for the production oracle. Record physical +layout, kernel-view layout, and actual strides on both sides. A transposed +view does not change the underlying physical layout or byte count. + Across six full-attention layers, BF16 K/V use 12,288 bytes per cached token and FP8 K/V use 6,144 bytes. Across 18 GDN layers, convolution state uses 663,552 bytes and SSM state uses 18,874,368 bytes per active slot. Confirm @@ -165,6 +190,19 @@ Widen stored BF16 to FP32. Dequantize stored FP8 with the recorded per-layer scale. The FP32 file format never conceals separately logged physical dtype and allocation. +Use canonical DS order for `state_gdn_conv`, with `0 <= d < 6144` and +`0 <= s < 3`: + +```text +local_dump[r, d * 3 + s] = widen(local[slot_local(r), d, s]) +oracle_dump[r, d * 3 + s] = widen(oracle[slot_oracle(r), s, d]) +``` + +Here `r` follows scheduler request order, and each arm resolves its own slot. +Use the recorded strides and storage offsets for each lookup. Record this +mapping in the manifest. The canonical DS dump order does not prescribe +physical storage or make the oracle's transposed view contiguous. + Gather nonnegative slot mappings in input-token order and active GDN indices in scheduler request order. Refuse unexplained duplicate destinations, missing rows, partial joins, or capacity-wide dumps. A 24-layer step emits 48 state @@ -179,15 +217,34 @@ output tokens. Keep block size, block count, and scheduler budget equal. Run each arm and cache mode in a separate process and empty directory. Take two enabled repeats and one dump-disabled identity control. -Log and assert requested cache dtype, resolved cache dtype, physical cache -dtype, selected attention backend, and selected cache-store operator for every -arm and mode. Do not assume backend identity from device alone. The current CPU -provider accepts `auto` and `fp8_e4m3` in `CPU_ATTN` but omits explicit -`bfloat16`; that mode falls through to `FLASH_ATTN`, which accepts it. Assert -`CPU_ATTN` for `auto` and `fp8_e4m3` and `FLASH_ATTN` for explicit `bfloat16` -on this implementation base. If main changes the selector before implementation, -re-read and cite the source, update this spec before code, and freshly review -the changed contract. +Log and assert requested cache dtype, resolved storage and FP8 interpretation, +normalized selector input, physical cache dtype, selected attention backend, +and actual provider execution for every arm and mode. +`LoadedEngine::ApplyResolvedCacheDType` in +`src/vllm/entrypoints/model_loader.cpp:1835` applies `ParseCacheDType` from +`include/vllm/v1/kv_cache_dtype.h:58` to the cache spec. The runner constructs +`cfg.kv_cache_dtype` from resolved storage and interpretation at +`src/vllm/v1/worker/gpu/runner.cpp:1590`. `KvCacheDTypeName` maps `kBF16` to +`auto` in `src/vllm/v1/attention/backend.cpp:91` to line 96. Therefore the +production CPU selector receives these values for this BF16 workload: + +| Requested cache dtype | Resolved physical storage | Normalized selector input | Selected CPU backend | +|---|---|---|---| +| `auto` | BF16 (`kBF16`) | `auto` | `CPU_ATTN` | +| `bfloat16` | BF16 (`kBF16`) | `auto` | `CPU_ATTN` | +| `fp8_e4m3` | E4M3 bytes (`kI8` plus `kFp8E4M3`) | `fp8_e4m3` | `CPU_ATTN` | + +CPU priority starts with `CPU_ATTN` in `src/vllm/platforms/cpu.cpp:51`. +Its accepted selector values include `auto` and `fp8_e4m3` in +`include/vllm/v1/attention/backends/cpu_attn.h:108` to line 110. +`SelectAttentionBackendName` returns the first valid candidate in +`src/vllm/v1/attention/registry.cpp:124` to line 128. +Selection alone does not prove execution. Independently record the executing +attention and cache-store operator providers, selection counts, declines, and +reference-tier hits. The runner's selected name validates cache configuration +but does not dispatch the model's attention call. If main changes this chain +before implementation, update the cited contract before code and obtain fresh +review. CPU is a diagnostic comparison arm for CPU-versus-ROCm deltas, including FP8 state after dequantization. Active-pin vLLM and its upstream tests are the @@ -424,6 +481,11 @@ push, open, or merge that pull request. pending until it lands. - #2773 owes the runnable active-pin Qwen3.5-0.8B ROCm captures, even though #2794 supplies repository sync context. +- [#2923](https://github.com/mudler/vllm.cpp/issues/2923), owned by + `BACKEND-ROCM`, owes routing Qwen3.5 paged attention through + `dense_attn::AttnBlock`. This tracked exception records existing debt. It + does not waive or satisfy the shared-seam requirement. Wiring needs its own + reviewed spec and implementation, outside #2773's instrumentation scope. - A fresh reviewer owes this repaired spec a verdict. - A fresh implementer owes the capture options, state probes, tests, comparator, and provider checks after the prerequisites pass. From 6b19d40e50dda89b81937859e56390261897ee2b Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Fri, 4 Sep 2026 17:44:25 -0700 Subject: [PATCH 4/8] spec(BACKEND-ROCM): distinguish strict capture provenance The #2773 capture tools also serve the ratified Qwen3 distributional gate. Keep that legacy contract and bind Qwen3.5 publication to verified model identity, ten deterministic repeats, and matched runtime artifacts. Require wheel bytes and launcher image attestation without claiming runtime proof. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [codex] --- .../rocm-qwen35-08b-cpu-gfx1100-numerics.md | 33 ++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md index c4b7ddff86..3e9b7a4e87 100644 --- a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md +++ b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md @@ -326,6 +326,8 @@ to: --model-revision REV --vllm-revision REV --provenance-out PATH +--vllm-wheel PATH +--runtime-manifest PATH ``` `production` must instantiate vLLM without `enforce_eager=True`; eager is a @@ -341,6 +343,8 @@ issue-evidence directory instead of the committed golden directory. The second sets `EngineParams::kv_cache_dtype` and prints requested, resolved, and physical dtype. These commands describe the intended interface and data flow after those changes. They are future commands and will fail today: +The launcher must first supply the inspected `VLLM_WHEEL` and +`ORACLE_RUNTIME_MANIFEST` paths. Missing values keep capture pending. ```sh VLLM_PIN=e126687a9a828d513c01a07cd69f025f27d63280 @@ -353,6 +357,7 @@ for mode in auto bfloat16 fp8_e4m3; do "${VLLM_ORACLE}" scripts/qwen3-oracle-capture.py \ --model "${MODEL}" --model-revision "${MODEL_REV}" \ --vllm-revision "${VLLM_PIN}" \ + --vllm-wheel "${VLLM_WHEEL}" --runtime-manifest "${ORACLE_RUNTIME_MANIFEST}" \ --kv-cache-dtype "${mode}" --execution-mode production --seed 0 \ --max-tokens 16 --runs 10 --per-prompt --out-dir "${GOLDEN_DIR}" \ --provenance-out "evidence/2773/oracle-${mode}.json" @@ -363,6 +368,7 @@ for mode in auto bfloat16 fp8_e4m3; do "${VLLM_ORACLE}" scripts/qwen3-neartie-gap.py \ --model "${MODEL}" --model-revision "${MODEL_REV}" \ --vllm-revision "${VLLM_PIN}" \ + --vllm-wheel "${VLLM_WHEEL}" --runtime-manifest "${ORACLE_RUNTIME_MANIFEST}" \ --kv-cache-dtype "${mode}" --execution-mode production --seed 0 \ --max-tokens 16 --topk 20 --golden-dir "${GOLDEN_DIR}" \ --provenance-out "evidence/2773/neartie-${mode}.json" @@ -374,7 +380,32 @@ done The tools must refuse when their `PROMPTS` lists differ from each other or from `tests/parity/test_qwen35_paged_engine.cpp::Prompts`. Record exact as-run -commands. Do not create a golden unless all 10 repeats are deterministic. +commands. Do not publish a Qwen3.5 sacred candidate unless at least 10 repeats +are deterministic. Select this regime from the artifact's verified +`config.json`, including its architecture, model type, and nested text model +type. Any Qwen3.5 indicator selects the strict regime. Require the runtime's +resolved model identity to agree. A missing or inconsistent identity refuses +publication. No command-line option can downgrade this regime. + +Legacy distributional captures retain their existing caller contracts outside +Qwen3.5. Their manifests identify the legacy regime, including observed +nondeterminism and missing provenance. They cannot supply a Qwen3.5 near-tie +capture. The near-tie tool validates regime, model and oracle identity, cache +mode, execution mode, prompts, sampling, token count, and input hashes before +using a Qwen3.5 capture. Existing legacy captures without manifests remain +usable only outside the strict regime. + +Strict capture requires `--vllm-wheel` and `--runtime-manifest`. The launcher +manifest supplies `vllm_revision`, `wheel_sha256`, and `image_digest`, whose +value is an immutable `sha256:` digest. Hash the wheel archive and compare its +package members with the imported package bytes. Independently verify the +imported source or installed VCS revision against the requested revision. +Record the actual source/package hashes and every model artifact hash. +Reject any revision or hash disagreement. Record image identity as an external +launcher attestation with its verification limit. Do not claim that the Python +process independently measured its container image. Missing wheel, image, +source, or artifact identity prevents strict publication. These metadata checks +do not prove runtime gateability, model correctness, or GPU execution. The internal CPU/ROCm state characterization remains eight output tokens as specified under `Workload and backend identity`. The permanent sacred-gate From 1eb7975d202021727ff3066314dc96e5437dbfc7 Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Fri, 4 Sep 2026 18:47:23 -0700 Subject: [PATCH 5/8] feat(BACKEND-ROCM): verify Qwen capture provenance Qwen3.5 candidates need cache-matched provenance before #2773 can run the active-pin gate. Verify model, package, wheel, and launcher identities, then refuse publication unless ten repeats agree. Keep legacy Qwen3 distributional captures and the PR #2856 execution-mode interface usable. The CPU fixtures pass 36 tests, and the exact PR #2856 suite passes two. All 73 adverse mutations fail their focused tests and restore cleanly. The full staged preflight exits zero with five prerequisite skips. The existing shellcheck-dependent test remains unavailable. Those obligations remain PENDING. The C++ probes, the PR #2856 landing and default gate, and active-pin runtime acceptance remain PENDING under BACKEND-ROCM and #2773. This tooling slice claims no GPU or numerical result. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [codex] --- .../rocm-qwen35-08b-cpu-gfx1100-numerics.md | 32 +- .github/workflows/ci.yml | 2 + docs/USAGE.md | 44 ++ scripts/agent-preflight.sh | 11 + scripts/qwen3-neartie-gap.py | 188 +++++--- scripts/qwen3-oracle-capture.py | 133 +++--- scripts/qwen3_oracle_common.py | 408 ++++++++++++++++++ tests/scripts/test_qwen3_capture_outputs.py | 287 ++++++++++++ tests/scripts/test_qwen3_capture_tools.py | 358 +++++++++++++++ 9 files changed, 1324 insertions(+), 139 deletions(-) create mode 100644 scripts/qwen3_oracle_common.py create mode 100644 tests/scripts/test_qwen3_capture_outputs.py create mode 100644 tests/scripts/test_qwen3_capture_tools.py diff --git a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md index 3e9b7a4e87..b609b615d9 100644 --- a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md +++ b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md @@ -12,8 +12,10 @@ Active primary oracle pin: vLLM ## Now -`PENDING`. This repaired spec is the only completed deliverable in this commit. -Implementation cannot start until a fresh reviewer passes this immutable spec. +`PENDING`. The amended spec passed fresh review before capture-tool implementation. +The Python capture tools now implement the provenance and deterministic publication +contract. Fresh implementation review and operator verification remain required. +The C++ characterization probes and real-runtime acceptance remain `PENDING`. Results cannot be accepted until both correctness prerequisites pass: 1. Issue #2772 and PR #2856 own the default-on ROCm `wvSplitK` sacred-anchor @@ -312,10 +314,9 @@ modes, but this spec alone does not authorize new permanent goldens. ## Required capture-tool implementation -The current `scripts/qwen3-oracle-capture.py` and -`scripts/qwen3-neartie-gap.py` hard-code `enforce_eager=True` and expose no -cache-dtype argument. Before active-pin capture, add reviewed options equivalent -to: +`scripts/qwen3-oracle-capture.py` and `scripts/qwen3-neartie-gap.py` expose the +capture interface below. Their CPU fixtures exercise provenance, refusal, and +NumPy serialization. Active-pin ROCm capture and acceptance remain pending. ```text --kv-cache-dtype {auto,bfloat16,fp8_e4m3} @@ -414,8 +415,9 @@ the eight-token characterization files as sacred-gate goldens. ## Tests and review mutations -The later implementation starts with focused tests that fail because the four -state rows and capture options are absent. It must test row shape/dtype, +The capture-tool slice used failing CLI and publication tests before implementation. +The later state implementation starts with focused tests that fail because the four +state rows are absent. It must test row shape/dtype, nonmonotonic and negative slot mappings, inactive-capacity exclusion, stored BF16 widening, post-quantization FP8 dequantization, persistent GDN prefill and decode state, incomplete-step refusal, comparator structural refusals and @@ -430,9 +432,9 @@ cache-store operator, inject a reference-tier hit, remove the production entry point, and force eager oracle mode. Each focused test must fail for the intended reason and the reviewer must restore the tree byte for byte. -This spec repair changes no runtime behavior. `IMP-TEST-FIRST` and -`IMP-MUTATE` are future implementation/review gates, not evidence claimed by -this commit. +The spec-only repair changed no runtime behavior. The capture-tool slice exercises +`IMP-TEST-FIRST` and `IMP-MUTATE` with CPU fixtures. State probes and real-runtime +gates remain future work. ## Gate order @@ -453,8 +455,8 @@ The later implementation must satisfy, in order: zero reference-tier hits. 12. The operator independently reruns focused, full, oracle, and hardware gates. -No GPU result, oracle result, model execution, runtime mutation, or -implementation test is claimed by this spec-only repair. +The capture-tool slice claims CPU fixture tests and mutations only. It claims no +GPU result, oracle result, model execution, or numerical runtime mutation. ## Evidence required @@ -517,8 +519,8 @@ push, open, or merge that pull request. `dense_attn::AttnBlock`. This tracked exception records existing debt. It does not waive or satisfy the shared-seam requirement. Wiring needs its own reviewed spec and implementation, outside #2773's instrumentation scope. -- A fresh reviewer owes this repaired spec a verdict. -- A fresh implementer owes the capture options, state probes, tests, comparator, +- The amended spec passed fresh review before capture-tool implementation. +- A fresh implementer owes the state probes, tests, comparator, and provider checks after the prerequisites pass. - A fresh implementation reviewer owes static review and every mutation. - The operator owes the independent controlled and gfx1100 gates. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0f8a0b075c..d8d2dd8f76 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -541,6 +541,8 @@ jobs: run: | sudo apt-get update -qq sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-numpy + python3 tests/scripts/test_qwen3_capture_tools.py + python3 tests/scripts/test_qwen3_capture_outputs.py python3 tests/scripts/test_ltx25_render_compare.py && python3 tests/scripts/test_ltx25_absolute_reference.py # The prompt-adherence half of the same tool (#2295, owning #1854's # first sub-question). 42 of its 47 cases need numpy only; the five diff --git a/docs/USAGE.md b/docs/USAGE.md index ca92c355f4..beba5515ad 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1220,6 +1220,50 @@ checkout at exactly the recorded byte counts, so those two sizes are locally confirmed. The distilled bf16 DiT is absent from the shared checkout, so its 42,018,190,584 bytes rest on the tree listing and a range request alone. +## Capture Qwen oracle evidence + +Use `scripts/qwen3-oracle-capture.py` for greedy continuations and +`scripts/qwen3-neartie-gap.py` for gaps under the local engine's exact token +prefix. Both accept `--kv-cache-dtype auto|bfloat16|fp8_e4m3`, `--seed`, +`--max-tokens`, and `--runs`. `--repetitions` is an alias for `--runs`. +Production execution is the default. `--execution-mode eager` and its +`--enforce-eager` alias select diagnostics, which cannot be a denominator. + +Qwen3.5 captures require verified local model artifacts and at least 10 +deterministic repeats. Supply full model and vLLM revisions, the installed +vLLM wheel archive, and a launcher JSON manifest. That manifest contains +`vllm_revision`, `wheel_sha256`, and an immutable `sha256:` `image_digest`. +Model files need matching HuggingFace download metadata or immutable cache +snapshot paths. The tools hash the files and compare wheel members with the +imported package. An installed `+g` version suffix verifies only its recorded +VCS prefix. Image identity remains a launcher attestation. + +After supplying these paths and revisions, use an empty capture directory: + +```sh +python3 scripts/qwen3-oracle-capture.py \ + --model "$MODEL_DIR" --model-revision "$MODEL_REV" \ + --vllm-revision "$VLLM_REV" --vllm-wheel "$VLLM_WHEEL" \ + --runtime-manifest "$RUNTIME_MANIFEST" --kv-cache-dtype auto \ + --execution-mode production --seed 0 --max-tokens 16 --runs 10 \ + --per-prompt --out-dir "$CAPTURE_DIR" +``` + +Place the matching local gate's `our_ids.i32` dump in that directory before +running `qwen3-neartie-gap.py`. Pass the same model, revisions, runtime inputs, +cache mode, seed, and token count, replacing `--out-dir` with `--golden-dir`. +The near-tie tool uses one request at a time, so its strict input capture must +use `--per-prompt`. Each tool writes a provenance JSON file beside its NumPy +outputs. `--provenance-out PATH` writes an identical additional copy. Strict +captures refuse existing outputs and mismatched or incomplete evidence. + +Legacy Qwen3 distributional calls remain usable, including captures without +complete provenance and near-tie inputs without manifests. Their manifests +record missing provenance and observed nondeterminism. A legacy capture cannot +supply a Qwen3.5 near-tie run. These tool checks do not establish GPU execution +or accept new permanent goldens. Active-pin acceptance remains pending in +[#2773](https://github.com/mudler/vllm.cpp/issues/2773). + ## Look up interface details [Reference pages](reference/README.md) collect dense lookup material such as diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index c91e426115..f8e6c0d5d3 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -438,6 +438,17 @@ echo "Mutation suites:" for suite in "${SUITES[@]}"; do run "$suite" python3 "tests/scripts/$suite.py" done +# QWEN3-CAPTURE-TOOLS: begin +if python3 -c 'import numpy' >/dev/null 2>&1; then + run "test_qwen3_capture_tools" python3 tests/scripts/test_qwen3_capture_tools.py + run "test_qwen3_capture_outputs" python3 tests/scripts/test_qwen3_capture_outputs.py +else + for suite in test_qwen3_capture_tools test_qwen3_capture_outputs; do + skip "$suite" "PENDING: numpy is not importable. CI installs python3-numpy for these capture suites." + done +fi +# QWEN3-CAPTURE-TOOLS: end + # THE ONE SUITE HERE WITH A THIRD-PARTY DEPENDENCY (#1612). It exercises # `scripts/ltx25-render-compare.py`, whose only import beyond the standard # library is numpy -- the tool reads PPM and WAV by hand precisely so that a diff --git a/scripts/qwen3-neartie-gap.py b/scripts/qwen3-neartie-gap.py index 692a70e27a..549b6f6783 100644 --- a/scripts/qwen3-neartie-gap.py +++ b/scripts/qwen3-neartie-gap.py @@ -23,9 +23,19 @@ # PATH="${VLLM_ORACLE}/bin:$PATH" "${VLLM_ORACLE}/bin/python" \ # scripts/qwen3-neartie-gap.py --model Qwen/Qwen3-4B \ # --golden-dir tests/parity/goldens/qwen3_greedy_4b +# +# Qwen3.5 additionally requires a matching strict capture manifest and ten +# identical raw-logprob repeats. See docs/USAGE.md for its provenance inputs. import argparse, os, sys +import hashlib +import io +import math +from pathlib import Path import numpy as np +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import qwen3_oracle_common as common + PROMPTS = [ "The capital of France is", "Once upon a time,", "In the beginning God created", "The quick brown fox jumps over", "def fibonacci(n):", @@ -49,80 +59,142 @@ def _parse_args(argv=None): # over-commits and REBOOTS the box. 0.40 is the safe ceiling (does not affect the # teacher-forced logprobs — only KV-cache capacity, unused here at max_tokens=1). ap.add_argument("--gpu-mem-util", type=float, default=0.40) - ap.add_argument("--enforce-eager", action="store_true", - help="use eager execution for diagnostics instead of the " - "production execution mode") - return ap.parse_args(argv) + ap.add_argument("--runs", "--repetitions", type=int, default=None) + common.add_options(ap) + return common.finish_args(ap, argv) def _llm_kwargs(args): - return { - "model": args.model, - "dtype": "bfloat16", - "enforce_eager": args.enforce_eager, - "gpu_memory_utilization": args.gpu_mem_util, - } + return common.llm_kwargs(args, args.gpu_mem_util) def _mode_narration(args): - eager = args.enforce_eager - mode = "eager diagnostic" if eager else "production" - return f"oracle execution mode: {mode} (enforce_eager={eager})" + return common.mode_narration(args) + + +def validate_capture(context, capture, inputs): + common.require(capture.get("tool") == "qwen3-oracle-capture" + and capture.get("regime") == "qwen3_5_strict" + and capture.get("deterministic") is True + and isinstance(capture.get("repetitions"), int) and capture["repetitions"] >= 10, + "near-tie input is not a deterministic strict capture") + for key in ("prompts_sha256", "sampling", "sampling_resolved", "execution_mode", "batching"): + common.require(capture.get(key) == context[key], f"capture {key} differs") + for key in ("requested", "resolved"): + common.require(capture.get("cache", {}).get(key) == context["cache"][key], f"capture cache {key} differs") + for key in ("requested_revision", "identity", "files"): + common.require(capture.get("model", {}).get(key) == context["model"][key], f"capture model {key} differs") + for key in ("requested_revision", "package_files"): + common.require(capture.get("runtime", {}).get(key) == context["runtime"][key], f"capture oracle {key} differs") + for key, field in (("wheel", "sha256"), ("image", "digest")): + common.require(capture.get("runtime", {}).get(key, {}).get(field) == context["runtime"][key][field], + f"capture oracle {key} differs") + expected = {"greedy_ids.npy", "greedy_dist.npy", *(f"p{i}_prompt.i32" for i in range(len(PROMPTS)))} + outputs = capture.get("outputs", {}) + common.require(set(outputs) == expected, "capture output manifest is incomplete") + common.require(capture.get("output_sha256") == hashlib.sha256(common.json_bytes(outputs)).hexdigest(), + "capture output manifest hash differs") + for name in expected: + common.require(name in inputs and outputs[name] == { + "sha256": hashlib.sha256(inputs[name]).hexdigest(), "size": len(inputs[name])}, + f"capture input hash differs: {name}") def main(): args = _parse_args() + prompt_record = common.check_prompts(__file__, PROMPTS) + model = common.model_identity(args) + strict = common.is_qwen35(model["identity"]) + if args.runs is None: + args.runs = 10 if strict else 1 + common.strict_inputs(args, model) + import vllm from vllm import LLM, SamplingParams - N, T = len(PROMPTS), args.max_tokens - our = np.fromfile(os.path.join(args.golden_dir, "our_ids.i32"), - dtype="= 0), "teacher forcing needs a complete nonnegative token stream", "STRUCTURE_MISMATCH") + if strict: + dist = np.load(io.BytesIO(inputs["greedy_dist.npy"]), allow_pickle=False) + common.require(dist.shape == (N, T, capture["repetitions"]) and dist.dtype == np.dtype(" max_gap: - max_gap, worst = gap, (i, j, gap) + data = inputs[f"p{i}_prompt.i32"] + common.require(data and len(data) % 4 == 0, "prompt token file is incomplete", "STRUCTURE_MISMATCH") + tokens = np.frombuffer(data, dtype="= 0 for token in tokens), "prompt contains a negative token", "STRUCTURE_MISMATCH") + prefixes.append(tokens) + sp_args = {"temperature": 0.0, "max_tokens": 1, "prompt_logprobs": args.topk, "seed": args.seed or 0} + context["teacher_forcing_sampling"] = common.sampling_record(SamplingParams(**sp_args)) + reference = [] + gap_mnats = np.zeros((N, T), dtype="/qwen3_greedy_4b +# +# The historical Qwen3 distributional workflow above remains available. +# Qwen3.5 uses verified provenance and at least ten deterministic repeats +# under #2773. See docs/USAGE.md for its capture and launcher inputs. import argparse +import io +from pathlib import Path import os import sys import numpy as np +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import qwen3_oracle_common as common + PROMPTS = [ "The capital of France is", "Once upon a time,", @@ -66,7 +75,7 @@ def _parse_args(argv=None): ap.add_argument("--model", default=os.environ.get("QWEN3_MODEL", "Qwen/Qwen3-0.6B")) ap.add_argument("--out-dir", default=None, help="golden dir (default tests/parity/goldens/qwen3_greedy_0_6b)") - ap.add_argument("--runs", type=int, default=int(os.environ.get("QWEN3_RUNS", "10")), + ap.add_argument("--runs", "--repetitions", type=int, default=int(os.environ.get("QWEN3_RUNS", "10")), help="K greedy runs to build the observed distribution (K>=1)") ap.add_argument("--max-tokens", type=int, default=16) ap.add_argument("--gpu-mem", type=float, @@ -77,25 +86,20 @@ def _parse_args(argv=None): help="decode each prompt in its OWN generate() call (batch size 1) " "to match the paged-engine gate's single-request decode " "regime; otherwise all prompts are batched in one call") - ap.add_argument("--enforce-eager", action="store_true", - help="use eager execution for diagnostics instead of the " - "production execution mode") - return ap.parse_args(argv) + common.add_options(ap) + return common.finish_args(ap, argv) + + +def parse_args(argv=None): + return _parse_args(argv) def _llm_kwargs(args): - return { - "model": args.model, - "dtype": "bfloat16", - "enforce_eager": args.enforce_eager, - "gpu_memory_utilization": args.gpu_mem, - } + return common.llm_kwargs(args, args.gpu_mem) def _mode_narration(args): - eager = args.enforce_eager - mode = "eager diagnostic" if eager else "production" - return f"oracle execution mode: {mode} (enforce_eager={eager})" + return common.mode_narration(args) def generate_all(llm, sp, per_prompt): @@ -113,66 +117,63 @@ def generate_all(llm, sp, per_prompt): def main(): args = _parse_args() + prompt_record = common.check_prompts(__file__, PROMPTS) + model = common.model_identity(args) + common.strict_inputs(args, model) + import vllm from vllm import LLM, SamplingParams - out_dir = args.out_dir or default_out_dir() - os.makedirs(out_dir, exist_ok=True) - N = len(PROMPTS) - T = args.max_tokens - K = max(1, args.runs) - + strict = common.is_qwen35(model["identity"]) + runtime = common.runtime_identity(vllm, args, strict) if strict else None print(_mode_narration(args), flush=True) llm = LLM(**_llm_kwargs(args)) - sp = SamplingParams(temperature=0.0, max_tokens=T) - - # runs[k][i] = list of token ids for prompt i on run k (padded to T with -1). + runtime = runtime or common.runtime_identity(vllm, args, False) + context = common.resolved_context(args, llm, model, runtime, prompt_record, + 1 if args.per_prompt else len(PROMPTS)) + context["tool"] = "qwen3-oracle-capture" + strict = context["regime"] == "qwen3_5_strict" + out_dir = Path(args.out_dir or default_out_dir()) + common.require(not strict or not out_dir.exists() or not any(out_dir.iterdir()), + f"capture directory is not empty: {out_dir}") + N, T, K = len(PROMPTS), args.max_tokens, args.runs + sp = SamplingParams(temperature=0.0, max_tokens=T, seed=args.seed or 0) + common.record_sampling(context, sp) dist = np.full((N, T, K), -1, dtype=" {ids} ({o.outputs[0].text!r})") - - np.save(os.path.join(out_dir, "greedy_ids.npy"), run0) - np.save(os.path.join(out_dir, "greedy_dist.npy"), dist) - - # ---- determinism report ------------------------------------------------- - print(f"\n=== determinism report: {args.model} N={N} T={T} K={K} ===") - deterministic = True - total_multi_pos = 0 - for i in range(N): - # distinct full sequences across the K runs - seqs = {tuple(int(x) for x in dist[i, :, k]) for k in range(K)} - # per-position distinct-token count - multi = [j for j in range(T) if len({int(dist[i, j, k]) for k in range(K)}) > 1] - total_multi_pos += len(multi) - if len(seqs) > 1: - deterministic = False - print(f" prompt[{i}] NON-DET: {len(seqs)} distinct sequences; " - f"near-tie positions {multi}") - for j in multi: - observed = sorted({int(dist[i, j, k]) for k in range(K)}) - print(f" pos {j:2d}: observed tokens {observed}") - else: - print(f" prompt[{i}] deterministic (1 sequence over {K} runs)") - print(f"=== {'ALL DETERMINISTIC' if deterministic else 'NON-DETERMINISTIC'} " - f"over K={K}; {total_multi_pos} multi-member (prompt,pos) cells ===") - print(f"wrote {out_dir}/greedy_ids.npy {run0.shape} + " - f"{out_dir}/greedy_dist.npy {dist.shape}") + prompt_ids.append(prefix) + else: + deterministic &= prefix == prompt_ids[i] and np.array_equal(dist[i, :, k], dist[i, :, 0]) + context["deterministic"] = bool(deterministic) + common.require(not strict or deterministic, "oracle tokens or prompt tokenization changed across repeats", "NONDETERMINISTIC") + common.confirm_inputs(args, vllm, context, __file__, PROMPTS) + run0 = dist[:, :, 0].copy() + payloads = {f"p{i}_prompt.i32": np.asarray(ids, dtype="= 2, f"incomplete download metadata for {relative}") + revision, etag = lines[:2] + elif root.parent.name == "snapshots" and path.is_symlink(): + revision, etag = root.name, path.resolve().name + if revision is None: + missing.append(f"artifact revision: {relative}") + continue + require(revision == args.model_revision, f"model revision differs for {relative}") + if re.fullmatch(r"[0-9a-f]{64}", etag or ""): + require(record["sha256"] == etag, f"model hash differs for {relative}") + elif re.fullmatch(r"[0-9a-f]{40}", etag or ""): + require(git_blob_sha1(path) == etag, f"model Git-blob hash differs for {relative}") + else: + require(False, f"unsupported artifact identity for {relative}") + record["revision"] = revision + if not records: + missing.append("model artifacts") + result = {"path": str(root), "requested_revision": args.model_revision, + "identity": observed, "files": records, "missing": missing} + if strict: + require(not missing, "strict model identity is incomplete: " + ", ".join(missing)) + require(observed["model_type"] and observed["architectures"], "strict model identity is incomplete") + require("text_config" not in config or observed["text_model_type"], "strict text model identity is incomplete") + return result + + +def runtime_identity(module, args, strict): + missing = [] + requested = args.vllm_revision + if requested: + require(full_revision(requested), "vLLM revision must be a full commit SHA") + else: + missing.append("requested vLLM revision") + filename = getattr(module, "__file__", None) + package = Path(filename).resolve().parent if filename else None + files = {} + if package and package.is_dir(): + files = {"vllm/" + str(path.relative_to(package)): file_record(path) + for path in sorted(package.rglob("*")) if path.is_file() + and "__pycache__" not in path.parts and path.suffix != ".pyc"} + if not files: + missing.append("imported package bytes") + version = getattr(module, "__version__", None) + revision, verification, source_root = None, None, None + if package: + def git(*argv): + try: + return subprocess.run(["git", "-C", str(package), *argv], capture_output=True, text=True, + env=dict(os.environ, GIT_NO_LAZY_FETCH="1", GIT_OPTIONAL_LOCKS="0")) + except FileNotFoundError: + return subprocess.CompletedProcess(argv, 127, "", "git is unavailable") + probe = git("ls-files", "--error-unmatch", Path(filename).name) + if probe.returncode == 0: + head = git("rev-parse", "HEAD") + require(head.returncode == 0, "cannot read imported source revision") + revision, verification = head.stdout.strip(), "clean_git_source" + source_root = git("rev-parse", "--show-toplevel").stdout.strip() + require(git("diff", "--quiet", "HEAD", "--", source_root).returncode == 0, + "imported vLLM source has tracked changes") + try: + distribution = importlib.metadata.distribution("vllm") + except importlib.metadata.PackageNotFoundError: + distribution = None + if distribution: + require(str(distribution.version) == str(version), "imported and installed vLLM versions differ") + prefix_match = re.search(r"(?:\+|\.)g([0-9a-f]{7,40})(?:[.+-]|$)", str(version)) + prefix = prefix_match.group(1) if prefix_match else None + if revision and prefix: + require(revision.startswith(prefix), "installed version disagrees with imported Git source") + if revision is None and prefix: + revision, verification = prefix, "installed_version_vcs_prefix" + if revision is None: + missing.append("observed source or installed VCS revision") + elif requested: + require(requested.startswith(revision), "observed vLLM revision differs from requested revision") + wheel = {"path": args.vllm_wheel, "sha256": None} + if args.vllm_wheel: + wheel["sha256"] = sha256(args.vllm_wheel) + try: + archive = zipfile.ZipFile(args.vllm_wheel) + except zipfile.BadZipFile as error: + raise CaptureError("ARTIFACT_MISMATCH: vLLM wheel is not a valid archive") from error + with archive: + members = [entry for entry in archive.infolist() + if entry.filename.startswith("vllm/") and not entry.is_dir() + and "__pycache__" not in entry.filename.split("/") + and not entry.filename.endswith(".pyc")] + require(len({entry.filename for entry in members}) == len(members), "duplicate wheel package member") + require({entry.filename for entry in members} == set(files), "wheel and imported package file sets differ") + for entry in members: + digest = hashlib.sha256() + with archive.open(entry) as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + require(digest.hexdigest() == files[entry.filename]["sha256"], + f"wheel and imported package bytes differ: {entry.filename}") + else: + missing.append("vLLM wheel") + image = {"digest": None, "source": "launcher_attestation", + "verification": "The Python process did not independently measure its container image."} + manifest_hash = None + if args.runtime_manifest: + launcher = read_json(args.runtime_manifest) + require(full_revision(launcher.get("vllm_revision")), "launcher vLLM revision is missing or invalid") + require(launcher["vllm_revision"] == requested, "launcher and requested vLLM revisions differ") + require(launcher.get("wheel_sha256") == wheel["sha256"] and wheel["sha256"] is not None, + "launcher and measured wheel hashes differ") + require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(launcher.get("image_digest"))), + "launcher image identity must be an immutable sha256 digest") + image["digest"] = launcher["image_digest"] + manifest_hash = sha256(args.runtime_manifest) + else: + missing.append("launcher image attestation") + if strict: + require(not missing, "strict runtime identity is incomplete: " + ", ".join(missing)) + return {"version": version, "revision": revision, "requested_revision": requested, + "revision_verification": verification, + "revision_limit": "An installed +g suffix verifies only the recorded VCS prefix; the full requested SHA is separate.", + "source_root": source_root, "package_files": files, "wheel": wheel, "image": image, + "launcher_manifest_sha256": manifest_hash, "missing": missing} + + +def resolved_context(args, llm, model, runtime, prompt_record, batch_size): + config = getattr(getattr(llm, "llm_engine", None), "vllm_config", None) + model_config = getattr(config, "model_config", None) + resolved_identity = identity(getattr(model_config, "hf_config", None)) + strict = is_qwen35(model["identity"]) or is_qwen35(resolved_identity) + if strict: + require(is_qwen35(model["identity"]) and not model["missing"], "strict model artifacts are not verified") + require(model["identity"] == resolved_identity, "artifact and runtime model identities differ") + require(not runtime["missing"], "strict runtime identity is incomplete") + require(args.runs >= 10, "strict capture requires at least ten deterministic repeats") + dtype = str(getattr(model_config, "dtype", "unknown")).removeprefix("torch.") + reported_cache = getattr(getattr(config, "cache_config", None), "cache_dtype", None) + resolved_cache = dtype if reported_cache == "auto" else reported_cache + requested_cache = args.kv_cache_dtype or "auto" + expected_cache = "bfloat16" if requested_cache == "auto" else requested_cache + if strict: + require(dtype == "bfloat16" and resolved_cache == expected_cache, "requested and resolved cache dtypes differ") + require(getattr(model_config, "enforce_eager", None) == args.enforce_eager, "runtime execution mode differs") + require(getattr(model_config, "seed", None) == (args.seed or 0), "runtime seed differs") + require(getattr(model_config, "revision", None) == args.model_revision + and getattr(model_config, "tokenizer_revision", None) == args.model_revision, + "runtime model or tokenizer revision differs") + return {"schema_version": 1, "regime": "qwen3_5_strict" if strict else "legacy_distributional", + "provenance_status": "incomplete" if model["missing"] or runtime["missing"] else "complete", + "arguments": vars(args), "command": [sys.executable, *sys.argv], + "model": model, "runtime": runtime, "resolved_model_identity": resolved_identity, + "cache": {"requested": requested_cache, "reported_selector": reported_cache, + "resolved": resolved_cache, "physical_dtype": None, + "verification": "Resolved configuration only; physical cache storage is not measured."}, + "execution_mode": args.execution_mode, + "purpose": "diagnostic" if args.enforce_eager else "production_capture", + "batching": {"batch_size": batch_size, "concurrency": batch_size}, + "sampling": {"temperature": 0.0, "max_tokens": args.max_tokens, "seed": args.seed or 0}, + "repetitions": args.runs, **prompt_record} + + +def strict_inputs(args, model): + if is_qwen35(model["identity"]): + require(args.vllm_revision and args.vllm_wheel and args.runtime_manifest, + "strict capture requires vLLM revision, wheel, and runtime manifest") + require(args.runs >= 10, "strict capture requires at least ten deterministic repeats") + + +def confirm_inputs(args, module, context, script_path, prompts): + require(model_identity(args) == context["model"], "model artifacts changed during capture") + require(runtime_identity(module, args, context["regime"] == "qwen3_5_strict") == context["runtime"], + "runtime identity changed during capture") + require(check_prompts(script_path, prompts) == {key: context[key] for key in ("prompts_sha256", "scripts")}, + "capture scripts or prompts changed during capture") + + +def sampling_record(params): + """Record resolved defaults, including vLLM's seed=-1 normalization.""" + def encode(value): + if value is None or isinstance(value, (str, bool, int, float)): + return value + if isinstance(value, dict): + return {str(key): encode(item) for key, item in value.items()} + if isinstance(value, (set, frozenset)): + return [encode(item) for item in sorted(value, key=repr)] + if isinstance(value, (list, tuple)): + return [encode(item) for item in value] + return str(value) + names = getattr(params, "__struct_fields__", None) + values = vars(params) if names is None else {name: getattr(params, name) for name in names} + return {name: encode(value) for name, value in values.items()} + + +def record_sampling(context, params): + context["sampling_resolved"] = sampling_record(params) + context["sampling"] = {name: getattr(params, name) for name in ("temperature", "max_tokens", "seed")} + + +def publish(directory, payloads, provenance, manifest_name, external=None): + """Publish validated results; legacy callers retain their overwrite contract.""" + directory = Path(directory).resolve() + provenance["outputs"] = {name: {"sha256": hashlib.sha256(data).hexdigest(), "size": len(data)} + for name, data in sorted(payloads.items())} + provenance["output_sha256"] = hashlib.sha256(json_bytes(provenance["outputs"])).hexdigest() + manifest = json_bytes(provenance) + targets = {directory / name: data for name, data in payloads.items()} + targets[directory / manifest_name] = manifest + if external: + path = Path(external).resolve() + require(path not in targets or path == directory / manifest_name, "provenance path overlaps an output") + targets[path] = manifest + legacy = provenance["regime"] == "legacy_distributional" + backups = {} + for target in targets: + require(legacy or not target.exists(), f"refusing to overwrite {target}") + if target.exists(): + backups[target] = target.read_bytes() + created = [] + try: + for target, data in targets.items(): + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("wb" if legacy else "xb") as stream: + created.append(target) + stream.write(data) + except BaseException: + for target in created: + if target in backups: + target.write_bytes(backups[target]) + else: + target.unlink(missing_ok=True) + raise diff --git a/tests/scripts/test_qwen3_capture_outputs.py b/tests/scripts/test_qwen3_capture_outputs.py new file mode 100644 index 0000000000..7b42eefe95 --- /dev/null +++ b/tests/scripts/test_qwen3_capture_outputs.py @@ -0,0 +1,287 @@ +#!/usr/bin/env python3 +"""Real NumPy output checks with a controlled fake vLLM, never a GPU oracle.""" + +import json +from pathlib import Path +import shutil +import unittest +import zipfile + +import numpy as np + +from test_qwen3_capture_tools import CaptureFixture, sha + + +class CaptureOutputTests(CaptureFixture): + def capture(self, *extra, env=None): + result = self.run_script([*self.args(), "--per-prompt", *extra], env=env) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + return json.loads((self.out / "oracle-provenance.json").read_text()) + + def calls(self): + return [json.loads(line) for line in self.log.read_text().splitlines()] + + def test_ten_runs_publish_numpy_and_actual_provenance(self): + external = self.root / "evidence/capture.json" + for dtype in ("auto", "bfloat16", "fp8_e4m3"): + provenance = self.capture("--kv-cache-dtype", dtype, "--seed", "17", + "--provenance-out", str(external)) + ids = np.load(self.out / "greedy_ids.npy", allow_pickle=False) + dist = np.load(self.out / "greedy_dist.npy", allow_pickle=False) + self.assertEqual(ids.dtype, np.dtype(" with __commit_id__=None. + # This fixture proves metadata verification, never wheel gateability. + (self.source / ".git").rename(self.root / "saved-source-git") + package = self.source / "vllm/__init__.py" + original = package.read_text() + metadata = self.source / "vllm-0.28.1.dist-info/METADATA" + metadata.parent.mkdir() + for prefix, accepted in ((self.revision[:9], True), ("0" * 9, False)): + version = "0.28.1rc1.dev132+g" + prefix + package.write_text(original.replace("controlled-test-fixture", version)) + metadata.write_text(f"Name: vllm\nVersion: {version}\n") + with zipfile.ZipFile(self.wheel, "w") as archive: + archive.write(package, "vllm/__init__.py") + archive.write(metadata, "vllm-0.28.1.dist-info/METADATA") + runtime = json.loads(self.runtime_manifest.read_text()) + runtime["wheel_sha256"] = sha(self.wheel) + self.runtime_manifest.write_text(json.dumps(runtime)) + if accepted: + provenance = self.capture() + self.assertEqual(provenance["runtime"]["revision"], prefix) + self.assertEqual(provenance["runtime"]["revision_verification"], + "installed_version_vcs_prefix") + shutil.rmtree(self.out) + else: + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def prepare_neartie(self): + self.capture() + our = np.load(self.out / "greedy_ids.npy", allow_pickle=False) + our[0, 0] = 9999 + our.astype(" 16: + lp -= 0.0001 + d[token] = NS(logprob=lp) + probs.append(d) + return [NS(prompt_logprobs=probs)] + results = [] + for prompt in prompts: + index = int(os.environ["FAKE_PROMPTS"].split("||").index(prompt)) + ids = [700 + index] * sp.max_tokens + prompt_ids = [500 + index, 600] + if os.environ.get("FAKE_TOKEN_DRIFT") and self.calls > 144: + ids[-1] += 1 + if os.environ.get("FAKE_PROMPT_DRIFT") and self.calls > 144: + prompt_ids[0] += 1 + if os.environ.get("FAKE_SHORT"): + ids = ids[:-1] + results.append(NS(prompt=prompt, prompt_token_ids=prompt_ids, + outputs=[NS(token_ids=ids, text="fixture")])) + return results +''' + + +def sha(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def load_script(path): + name = path.stem.replace("-", "_") + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + with mock.patch.dict(sys.modules, {"numpy": ModuleType("numpy")}), mock.patch.object( + sys, "path", [str(path.parent), *sys.path] + ): + spec.loader.exec_module(module) + return module + + +class CaptureFixture(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory(prefix="qwen3-capture-") + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + self.project = self.root / "project" + (self.project / "scripts").mkdir(parents=True) + (self.project / "tests/parity").mkdir(parents=True) + for filename in (*SCRIPTS, "qwen3_oracle_common.py"): + source = ROOT / "scripts" / filename + if source.exists(): + shutil.copy2(source, self.project / "scripts" / filename) + shutil.copy2(ROOT / "tests/parity/test_qwen35_paged_engine.cpp", + self.project / "tests/parity/test_qwen35_paged_engine.cpp") + self.model = self.root / "model" + self.model.mkdir() + (self.model / "config.json").write_text(json.dumps({ + "model_type": "qwen3_5", "architectures": ["Qwen3_5ForConditionalGeneration"], + "text_config": {"model_type": "qwen3_5_text"}, + })) + (self.model / "model.safetensors").write_bytes(b"not real model weights") + self.write_metadata() + self.source = self.root / "source" + (self.source / "vllm").mkdir(parents=True) + (self.source / "vllm/__init__.py").write_text(FAKE_VLLM) + self.git("init", "-q") + self.git("add", "vllm") + self.git("-c", "user.name=Fixture", "-c", "user.email=fixture@invalid", + "commit", "-qm", "controlled fixture") + self.revision = self.git("rev-parse", "HEAD").strip() + self.wheel = self.root / "fixture.whl" + with zipfile.ZipFile(self.wheel, "w") as archive: + archive.write(self.source / "vllm/__init__.py", "vllm/__init__.py") + self.runtime_manifest = self.root / "runtime.json" + self.runtime_manifest.write_text(json.dumps({ + "vllm_revision": self.revision, "wheel_sha256": sha(self.wheel), + "image_digest": "sha256:" + "c" * 64, + })) + self.out = self.root / "golden" + self.log = self.root / "calls.jsonl" + prompts = load_script(self.project / "scripts" / SCRIPTS[0]).PROMPTS + self.env = dict(os.environ, PYTHONPATH=os.pathsep.join( + [str(self.source), os.environ.get("PYTHONPATH", "")]), FAKE_LOG=str(self.log), + FAKE_PROMPTS="||".join(prompts), PYTHONDONTWRITEBYTECODE="1") + + def git(self, *args): + return subprocess.check_output(["git", "-C", str(self.source), *args], text=True) + + def write_metadata(self): + for path in self.model.iterdir(): + if not path.is_file(): + continue + data = path.read_bytes() + etag = (hashlib.sha1(f"blob {len(data)}\0".encode() + data).hexdigest() + if path.suffix == ".json" else hashlib.sha256(data).hexdigest()) + metadata = self.model / ".cache/huggingface/download" / (path.name + ".metadata") + metadata.parent.mkdir(parents=True, exist_ok=True) + metadata.write_text(f"{MODEL_REV}\n{etag}\n0\n") + + def args(self, near=False): + return ["--model", str(self.model), "--model-revision", MODEL_REV, + "--vllm-revision", self.revision, "--vllm-wheel", str(self.wheel), + "--runtime-manifest", str(self.runtime_manifest), "--max-tokens", "2", + "--golden-dir" if near else "--out-dir", str(self.out)] + + def run_script(self, args=None, *, near=False, env=None): + return subprocess.run([sys.executable, str(self.project / "scripts" / SCRIPTS[near]), + *(self.args(near) if args is None else args)], + env=dict(self.env, **(env or {})), text=True, capture_output=True) + + def assert_refused(self, result, reason): + self.assertNotEqual(result.returncode, 0, result.stdout) + self.assertIn(reason, result.stderr, result.stdout + result.stderr) + self.assertFalse(self.out.exists() and any(self.out.iterdir())) + + +class CaptureContractTests(CaptureFixture): + def test_options_and_eager_alias(self): + for script in SCRIPTS: + module = load_script(self.project / "scripts" / script) + required = ["--model", "model", "--golden-dir", "unused"] if "neartie" in script else [] + for mode in ("production", "eager"): + for dtype in ("auto", "bfloat16", "fp8_e4m3"): + args = module._parse_args([*required, "--execution-mode", mode, + "--kv-cache-dtype", dtype, "--seed", "17"]) + kwargs = module._llm_kwargs(args) + self.assertEqual(kwargs["kv_cache_dtype"], dtype) + self.assertEqual(kwargs["seed"], 17) + self.assertIs(kwargs["enforce_eager"], mode == "eager") + args = module._parse_args([*required, "--enforce-eager"]) + self.assertEqual(args.execution_mode, "eager") + self.assertIn("eager diagnostic", module._mode_narration(args)) + with self.assertRaises(SystemExit): + module._parse_args([*required, "--enforce-eager", "--execution-mode", "production"]) + + def test_all_three_prompt_sources_are_checked(self): + for filename in ("scripts/" + SCRIPTS[0], "scripts/" + SCRIPTS[1], + "tests/parity/test_qwen35_paged_engine.cpp"): + path = self.project / filename + original = path.read_text() + path.write_text(original.replace("The capital of France is", "Changed prompt", 1)) + for near in (False, True): + self.assert_refused(self.run_script(near=near), "PROMPTS_MISMATCH") + self.assertFalse(self.log.exists(), "prompt mismatch must refuse before LLM construction") + path.write_text(original) + + def test_strict_identity_inputs_are_required(self): + for option in ("--model-revision", "--vllm-revision", "--vllm-wheel", "--runtime-manifest"): + argv = self.args() + index = argv.index(option) + del argv[index:index + 2] + self.assert_refused(self.run_script(argv), "ARTIFACT_MISMATCH") + + def test_model_revision_and_content_are_verified(self): + path = self.model / "config.json" + original = path.read_bytes() + path.write_bytes(original + b" ") + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + path.write_bytes(original) + meta = self.model / ".cache/huggingface/download/config.json.metadata" + meta.write_text(meta.read_text().replace(MODEL_REV, "b" * 40)) + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_weight_bytes_are_checked_against_download_identity(self): + (self.model / "model.safetensors").write_bytes(b"different weights") + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_dirty_source_is_refused_even_when_the_wheel_matches_it(self): + source = self.source / "vllm/__init__.py" + source.write_text(source.read_text() + "\n# dirty source rebuilt into wheel\n") + with zipfile.ZipFile(self.wheel, "w") as archive: + archive.write(source, "vllm/__init__.py") + runtime = json.loads(self.runtime_manifest.read_text()) + runtime["wheel_sha256"] = sha(self.wheel) + self.runtime_manifest.write_text(json.dumps(runtime)) + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_vllm_revision_and_dirty_source_are_refused(self): + argv = self.args() + argv[argv.index("--vllm-revision") + 1] = "b" * 40 + self.assert_refused(self.run_script(argv), "ARTIFACT_MISMATCH") + with (self.source / "vllm/__init__.py").open("a") as stream: + stream.write("\n# dirty\n") + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_runtime_manifest_and_wheel_are_verified(self): + original = self.runtime_manifest.read_text() + for field, value in (("wheel_sha256", "0" * 64), ("vllm_revision", "b" * 40), + ("image_digest", "mutable:latest")): + data = json.loads(original) + data[field] = value + self.runtime_manifest.write_text(json.dumps(data)) + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + self.runtime_manifest.write_text(original) + with zipfile.ZipFile(self.wheel, "w") as archive: + archive.writestr("vllm/__init__.py", FAKE_VLLM + "\n# changed wheel\n") + data = json.loads(original) + data["wheel_sha256"] = sha(self.wheel) + self.runtime_manifest.write_text(json.dumps(data)) + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_strict_runtime_needs_an_observed_source_or_installed_vcs_revision(self): + (self.source / ".git").rename(self.root / "saved-source-git") + self.assert_refused(self.run_script(), "observed source or installed VCS revision") + + def test_imported_and_installed_runtime_versions_must_agree(self): + metadata = self.source / "vllm-0.28.1.dist-info/METADATA" + metadata.parent.mkdir() + metadata.write_text("Name: vllm\nVersion: different-installed-version\n") + self.assert_refused(self.run_script(), "imported and installed vLLM versions differ") + + def test_untracked_package_bytes_and_missing_artifact_metadata_are_refused(self): + extra = self.source / "vllm/untracked.py" + extra.write_text("untracked = True\n") + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + extra.unlink() + (self.model / ".cache/huggingface/download/model.safetensors.metadata").unlink() + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_runtime_identity_cannot_promote_unverified_legacy_inputs(self): + (self.model / "config.json").write_text(json.dumps({"model_type": "qwen3"})) + self.write_metadata() + self.assert_refused(self.run_script(["--model", str(self.model), "--out-dir", + str(self.out), "--runs", "1"], + env={"FAKE_RUNTIME_QWEN35": "1"}), + "ARTIFACT_MISMATCH") + + def test_strict_short_repetition_count_is_refused(self): + self.assert_refused(self.run_script([*self.args(), "--runs", "9"]), "ten") + self.assertFalse(self.log.exists(), "short strict capture must refuse before LLM construction") + + def test_each_qwen35_identity_indicator_prevents_legacy_downgrade(self): + for config in ({"model_type": "qwen3_5"}, + {"model_type": "qwen3", "architectures": ["Qwen3_5ForCausalLM"]}, + {"model_type": "qwen3", "text_config": {"model_type": "qwen3_5_text"}}): + (self.model / "config.json").write_text(json.dumps(config)) + self.write_metadata() + self.assert_refused(self.run_script(["--model", str(self.model), "--out-dir", + str(self.out), "--runs", "1"]), + "ARTIFACT_MISMATCH") + + def test_options_reject_invalid_counts_and_cache_modes(self): + for script in SCRIPTS: + module = load_script(self.project / "scripts" / script) + required = ["--model", "model", "--golden-dir", "unused"] if "neartie" in script else [] + for option, value in (("--runs", "0"), ("--repetitions", "-1"), + ("--max-tokens", "0"), ("--kv-cache-dtype", "float32")): + with self.assertRaises(SystemExit): + module._parse_args([*required, option, value]) + + +class CaptureRegistrationTests(unittest.TestCase): + def test_preflight_runs_both_suites_or_reports_numpy_pending(self): + source = (ROOT / "scripts/agent-preflight.sh").read_text() + match = re.search(r"# QWEN3-CAPTURE-TOOLS: begin\n(.*?)# QWEN3-CAPTURE-TOOLS: end", source, re.S) + self.assertIsNotNone(match, "capture suites have no preflight registration") + prelude = '''run() { printf 'RUN %s\\n' "$1"; shift; "$@"; } +skip() { printf 'SKIP %s %s\\n' "$1" "$*"; } +python3() { + if test "$1" = -c; then test "$CAPTURE_HAVE_NUMPY" = 1; + else printf 'PYTHON %s\\n' "$*"; fi +} +''' + for available in ("0", "1"): + result = subprocess.run(["bash", "-c", prelude + match.group(1)], text=True, + capture_output=True, env=dict(os.environ, CAPTURE_HAVE_NUMPY=available)) + self.assertEqual(result.returncode, 0, result.stderr) + for name in ("test_qwen3_capture_tools", "test_qwen3_capture_outputs"): + self.assertIn(("RUN " if available == "1" else "SKIP ") + name, result.stdout) + if available == "1": + self.assertIn("PYTHON tests/scripts/" + name + ".py", result.stdout) + if available == "0": + self.assertIn("PENDING", result.stdout) + self.assertNotIn("RUN ", result.stdout) + + def test_ci_runs_both_suites_after_its_existing_numpy_install(self): + source = (ROOT / ".github/workflows/ci.yml").read_text() + for name in ("test_qwen3_capture_tools", "test_qwen3_capture_outputs"): + command = " python3 tests/scripts/" + name + ".py" + self.assertIn(command, source) + prefix = source[:source.index(command)] + block = prefix[prefix.rfind(" run: |") :] + self.assertIn("apt-get install -y --no-install-recommends python3-numpy", block) + + +if __name__ == "__main__": + unittest.main(verbosity=2) From c6b6c1f87a0227f204e945f3663cf99e793f4c0a Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Fri, 4 Sep 2026 19:41:00 -0700 Subject: [PATCH 6/8] fix(BACKEND-ROCM): bind capture evidence to verified inputs Strict near-tie publication accepted missing or contradictory runtime observations. Validate captured completeness and observed model and runtime identities against the independently verified current context. Publication could replace legacy inputs through destination aliases. Protect input file identities before writing any output. Keep ordinary legacy output replacement and valid installed revision prefixes. vLLM resolves sampling on a cloned request. Record constructor normalization separately and mark engine resolution unobserved. Refs #2773. This repairs capture tooling in draft PR #2932 on row/BACKEND-ROCM-NUMERICS-1588. GPU acceptance and the remaining characterization stay PENDING. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [codex] --- docs/USAGE.md | 7 + scripts/qwen3-neartie-gap.py | 26 ++- scripts/qwen3-oracle-capture.py | 3 +- scripts/qwen3_oracle_common.py | 43 +++- tests/scripts/test_qwen3_capture_outputs.py | 226 +++++++++++++++++++- 5 files changed, 291 insertions(+), 14 deletions(-) diff --git a/docs/USAGE.md b/docs/USAGE.md index beba5515ad..73913cb27b 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1256,6 +1256,13 @@ The near-tie tool uses one request at a time, so its strict input capture must use `--per-prompt`. Each tool writes a provenance JSON file beside its NumPy outputs. `--provenance-out PATH` writes an identical additional copy. Strict captures refuse existing outputs and mismatched or incomplete evidence. +Output and provenance paths cannot overwrite capture inputs, including symbolic +links and hardlinks to those inputs. + +`sampling_normalized` records the supplied `SamplingParams` after constructor +normalization. vLLM resolves engine requests on a clone. `sampling_resolved` +remains null, and `sampling_resolution` records that observation limit. +Teacher-forcing sampling carries the same qualification. Legacy Qwen3 distributional calls remain usable, including captures without complete provenance and near-tie inputs without manifests. Their manifests diff --git a/scripts/qwen3-neartie-gap.py b/scripts/qwen3-neartie-gap.py index 549b6f6783..b2bd469837 100644 --- a/scripts/qwen3-neartie-gap.py +++ b/scripts/qwen3-neartie-gap.py @@ -31,6 +31,7 @@ import io import math from pathlib import Path +import re import numpy as np sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) @@ -78,14 +79,26 @@ def validate_capture(context, capture, inputs): and capture.get("deterministic") is True and isinstance(capture.get("repetitions"), int) and capture["repetitions"] >= 10, "near-tie input is not a deterministic strict capture") - for key in ("prompts_sha256", "sampling", "sampling_resolved", "execution_mode", "batching"): + for key in ("provenance_status", "resolved_model_identity", "prompts_sha256", "sampling", + "sampling_normalized", "sampling_resolved", "sampling_resolution", "execution_mode", "batching"): common.require(capture.get(key) == context[key], f"capture {key} differs") for key in ("requested", "resolved"): common.require(capture.get("cache", {}).get(key) == context["cache"][key], f"capture cache {key} differs") - for key in ("requested_revision", "identity", "files"): + for key in ("requested_revision", "identity", "files", "missing"): common.require(capture.get("model", {}).get(key) == context["model"][key], f"capture model {key} differs") - for key in ("requested_revision", "package_files"): + for key in ("requested_revision", "package_files", "missing", "version"): common.require(capture.get("runtime", {}).get(key) == context["runtime"][key], f"capture oracle {key} differs") + runtime = capture["runtime"] + revision = runtime.get("revision") + verification = runtime.get("revision_verification") + valid_revision = verification == "clean_git_source" and common.full_revision(revision) + if verification == "installed_version_vcs_prefix": + # A captured installed revision keeps its observed prefix, independently + # checked against the current verified full revision and package bytes. + match = re.search(r"(?:\+|\.)g([0-9a-f]{7,40})(?:[.+-]|$)", str(runtime.get("version"))) + valid_revision = match is not None and revision == match.group(1) + common.require(valid_revision and context["runtime"]["requested_revision"].startswith(revision), + "capture observed oracle revision is missing, unverified, or differs") for key, field in (("wheel", "sha256"), ("image", "digest")): common.require(capture.get("runtime", {}).get(key, {}).get(field) == context["runtime"][key][field], f"capture oracle {key} differs") @@ -146,7 +159,7 @@ def main(): common.require(all(token >= 0 for token in tokens), "prompt contains a negative token", "STRUCTURE_MISMATCH") prefixes.append(tokens) sp_args = {"temperature": 0.0, "max_tokens": 1, "prompt_logprobs": args.topk, "seed": args.seed or 0} - context["teacher_forcing_sampling"] = common.sampling_record(SamplingParams(**sp_args)) + common.record_sampling(context, SamplingParams(**sp_args), key="teacher_forcing_sampling") reference = [] gap_mnats = np.zeros((N, T), dtype=" Date: Fri, 4 Sep 2026 20:27:56 -0700 Subject: [PATCH 7/8] fix(BACKEND-ROCM): refuse aliased capture destinations Legacy publication could overwrite the distribution metadata used to verify vLLM. Record the selected metadata files and protect their file identities. The final identity check also detects changes to those measured bytes. Distinct output paths could share an inode and replace NumPy payloads with JSON. Refuse output aliases before writing any destination. Preserve ordinary legacy replacement and the explicit default manifest path. Four permanent CLI tests cover metadata selection and fallbacks, publication aliases, metadata changes during capture, and successful replacement hashes. CPU fixtures and mutation checks cover both capture commands. Active-pin GPU acceptance and the remaining characterization stay PENDING under #2773. Refs #2773. This repairs draft PR #2932 on row/BACKEND-ROCM-NUMERICS-1588. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [codex] --- scripts/qwen3_oracle_common.py | 39 +++++- tests/scripts/test_qwen3_capture_outputs.py | 125 ++++++++++++++++++++ 2 files changed, 162 insertions(+), 2 deletions(-) diff --git a/scripts/qwen3_oracle_common.py b/scripts/qwen3_oracle_common.py index 6e609e2063..b7a4aac8c8 100644 --- a/scripts/qwen3_oracle_common.py +++ b/scripts/qwen3_oracle_common.py @@ -209,6 +209,26 @@ def model_identity(args): return result +def distribution_metadata_files(distribution): + """Record the filesystem metadata read by Python's selected distribution.""" + # CPython importlib.metadata: Distribution.version reads metadata, whose + # read_text calls try METADATA, PKG-INFO, then an old egg-info file. + # PathDistribution joins those names to _path, not to the package root or + # a guessed version directory. RECORD need not exist. + root = getattr(distribution, "_path", None) + if not isinstance(distribution, importlib.metadata.PathDistribution) or not isinstance(root, Path): + return {} + records = {} + for name in ("METADATA", "PKG-INFO", ""): + path = root / name + consumed = distribution.read_text(name) + if consumed is not None: + records[str(path.absolute())] = file_record(path) + if consumed: + break + return records + + def runtime_identity(module, args, strict): missing = [] requested = args.vllm_revision @@ -226,6 +246,7 @@ def runtime_identity(module, args, strict): if not files: missing.append("imported package bytes") version = getattr(module, "__version__", None) + metadata_files = {} revision, verification, source_root = None, None, None if package: def git(*argv): @@ -248,6 +269,7 @@ def git(*argv): distribution = None if distribution: require(str(distribution.version) == str(version), "imported and installed vLLM versions differ") + metadata_files = distribution_metadata_files(distribution) prefix_match = re.search(r"(?:\+|\.)g([0-9a-f]{7,40})(?:[.+-]|$)", str(version)) prefix = prefix_match.group(1) if prefix_match else None if revision and prefix: @@ -301,7 +323,8 @@ def git(*argv): return {"version": version, "revision": revision, "requested_revision": requested, "revision_verification": verification, "revision_limit": "An installed +g suffix verifies only the recorded VCS prefix; the full requested SHA is separate.", - "source_root": source_root, "package_files": files, "wheel": wheel, "image": image, + "source_root": source_root, "package_files": files, "distribution_metadata": metadata_files, + "wheel": wheel, "image": image, "launcher_manifest_sha256": manifest_hash, "missing": missing} @@ -396,6 +419,7 @@ def capture_input_paths(args, module, context, script_path): if filename: package_root = Path(filename).resolve().parent.parent paths.update(package_root / name for name in context["runtime"]["package_files"]) + paths.update(Path(path) for path in context["runtime"]["distribution_metadata"]) project_root = Path(script_path).resolve().parents[1] paths.update(project_root / name for name in context["scripts"]) paths.update(Path(path) for path in (args.vllm_wheel, args.runtime_manifest) if path) @@ -412,14 +436,25 @@ def publish(directory, payloads, provenance, manifest_name, external=None, *, pr targets = {directory / name: data for name, data in payloads.items()} targets[directory / manifest_name] = manifest if external: - path = Path(external).resolve() + path = Path(external).absolute() + # Preserve the final name so an alias to another manifest stays a + # distinct target. An explicit default name still publishes once. + path = path.parent.resolve() / path.name require(path not in targets or path == directory / manifest_name, "provenance path overlaps an output") targets[path] = manifest legacy = provenance["regime"] == "legacy_distributional" # stat follows symbolic links and identifies hardlinks to the same input. protected = {file_identity(path) for path in protected_inputs if Path(path).exists()} backups = {} + resolved_targets, output_identities = set(), set() for target in targets: + resolved = target.resolve() + require(resolved not in resolved_targets, f"publication paths alias an output: {target}") + resolved_targets.add(resolved) + if target.exists(): + inode = file_identity(target) + require(inode not in output_identities, f"publication paths alias an output: {target}") + output_identities.add(inode) require(not target.exists() or file_identity(target) not in protected, f"publication path overlaps an input: {target}") require(legacy or not target.exists(), f"refusing to overwrite {target}") diff --git a/tests/scripts/test_qwen3_capture_outputs.py b/tests/scripts/test_qwen3_capture_outputs.py index 7b93345237..8b8010fa01 100644 --- a/tests/scripts/test_qwen3_capture_outputs.py +++ b/tests/scripts/test_qwen3_capture_outputs.py @@ -2,6 +2,7 @@ """Real NumPy output checks with a controlled fake vLLM, never a GPU oracle.""" import json +import os from pathlib import Path import shutil import unittest @@ -502,6 +503,130 @@ def test_legacy_output_hardlink_cannot_overwrite_an_input(self): self.assertTrue(output.samefile(self.out / "our_ids.i32")) self.assertFalse((self.out / "neartie_gap_mnats.npy").exists()) + def test_legacy_publication_protects_selected_distribution_metadata(self): + self.prepare_legacy_neartie() + # Discover metadata on another sys.path entry, with no RECORD file. + # CPython Distribution.metadata tries METADATA, PKG-INFO, then egg-info. + site = self.root / "metadata-site" + site.mkdir() + self.env["PYTHONPATH"] = str(site) + os.pathsep + self.env["PYTHONPATH"] + for layout in ("dist-info", "egg-info-directory", "egg-info-file", "empty-metadata"): + metadata_root = site / ("vllm-unrelated-build.dist-info" if layout == "dist-info" + else "vllm.egg-info") + if layout == "egg-info-file": + source = metadata_root + else: + metadata_root.mkdir() + source = metadata_root / ("METADATA" if layout == "dist-info" else "PKG-INFO") + source.write_text("Name: vllm\nVersion: controlled-test-fixture\n") + sources = [source] + if layout == "empty-metadata": + empty = metadata_root / "METADATA" + empty.write_bytes(b"") + sources.append(empty) + for near in (False, True): + for consumed in sources: + for alias in ("direct", "symlink", "hardlink"): + with self.subTest(layout=layout, near=near, input=consumed.name, alias=alias): + destination = consumed if alias == "direct" else self.root / "metadata-alias.json" + if alias == "symlink": + destination.symlink_to(consumed) + elif alias == "hardlink": + destination.hardlink_to(consumed) + try: + self.assert_publication_refused( + [*self.args(near), "--runs", "1", "--provenance-out", str(destination)], + near=near, protected=tuple(sources)) + finally: + if alias != "direct": + destination.unlink() + if metadata_root.is_dir(): + shutil.rmtree(metadata_root) + else: + metadata_root.unlink() + + def test_legacy_publication_refuses_aliases_between_output_destinations(self): + self.prepare_legacy_neartie() + for near in (False, True): + names = (("our_ids.npy", "neartie_gap_mnats.npy", "neartie-provenance.json") if near else + ("greedy_ids.npy", "greedy_dist.npy", "oracle-provenance.json")) + # Every pair of payload, default manifest, and additional manifest. + outputs = [*(self.out / name for name in names), self.root / "additional.json"] + for left in range(len(outputs)): + for right in range(left + 1, len(outputs)): + source, destination = outputs[left], outputs[right] + for alias in ("direct", "symlink", "hardlink", "dangling-symlink"): + if alias == "direct" and right != 3: + continue # CLI payload names are fixed and distinct. + if alias == "direct" and left == 2: + continue # Explicit default manifest is a valid single target. + with self.subTest(near=near, source=source.name, target=destination.name, alias=alias): + originals = {p: p.read_bytes() for p in self.out.iterdir()} + try: + source.write_bytes(b"previous output") + destination.unlink(missing_ok=True) + external = source if alias == "direct" else outputs[-1] + if alias in ("symlink", "dangling-symlink"): + destination.symlink_to(source) + if alias == "dangling-symlink": + source.unlink() + elif alias == "hardlink": + destination.hardlink_to(source) + # A dangling link has no bytes to snapshot; record its target. + before = {p: (p.readlink() if p.is_symlink() else p.read_bytes()) + for p in [*self.out.iterdir(), *([destination] if right == 3 and alias != "direct" else [])]} + result = self.run_script([*self.args(near), "--runs", "1", "--provenance-out", str(external)], near=near) + self.assertNotEqual(result.returncode, 0, result.stdout) + self.assertIn("ARTIFACT_MISMATCH", result.stderr) + self.assertEqual(set(self.out.iterdir()), {p for p in before if p.parent == self.out}) + for path, value in before.items(): + self.assertEqual(path.readlink() if path.is_symlink() else path.read_bytes(), value) + if external not in before: + self.assertFalse(external.exists()) + finally: + # Unlink aliases before restoring any bytes, including red runs. + for path in [*self.out.iterdir(), outputs[-1]]: + path.unlink(missing_ok=True) + for path, data in originals.items(): + path.write_bytes(data) + + def test_legacy_explicit_default_manifest_and_replacement_keep_valid_hashes(self): + self.prepare_legacy_neartie() + for near in (False, True): + manifest = self.out / ("neartie-provenance.json" if near else "oracle-provenance.json") + for repeat in range(2): + result = self.run_script([*self.args(near), "--runs", "1", "--provenance-out", str(manifest)], near=near) + self.assertEqual(result.returncode, 0, result.stderr) + provenance = json.loads(manifest.read_text()) + for name, record in provenance["outputs"].items(): + self.assertEqual(sha(self.out / name), record["sha256"]) + self.assertEqual((self.out / name).stat().st_size, record["size"]) + + + def test_legacy_capture_records_and_rechecks_consumed_distribution_metadata(self): + self.prepare_legacy_neartie() + metadata = self.source / "vllm-0.28.1.dist-info/METADATA" + metadata.parent.mkdir() + metadata.write_text("Name: vllm\nVersion: controlled-test-fixture\n") + for near in (False, True): + manifest = self.out / ("neartie-provenance.json" if near else "oracle-provenance.json") + result = self.run_script([*self.args(near), "--runs", "1"], near=near) + self.assertEqual(result.returncode, 0, result.stderr) + runtime = json.loads(manifest.read_text())["runtime"] + self.assertEqual(runtime.get("distribution_metadata"), { + str(metadata): {"sha256": sha(metadata), "size": metadata.stat().st_size}}) + before = {p: p.read_bytes() for p in self.out.iterdir()} + original = metadata.read_bytes() + try: + result = self.run_script([*self.args(near), "--runs", "1"], near=near, + env={"FAKE_MUTATE_INPUT": str(metadata)}) + self.assertNotEqual(result.returncode, 0, result.stdout) + self.assertIn("runtime identity changed during capture", result.stderr) + self.assertEqual({p: p.read_bytes() for p in self.out.iterdir()}, before) + finally: + metadata.write_bytes(original) + + if __name__ == "__main__": unittest.main(verbosity=2) From 81855a3f7f0ed0a0bb21edd3fcd6c2f94b4a8766 Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Sun, 6 Sep 2026 19:49:23 -0700 Subject: [PATCH 8/8] record(BACKEND-ROCM): reconcile capture prerequisites with main PR #2856 landed on the pinned implementation base, so its merge is no longer an external prerequisite for #2773. The unchanged default gate still requires an operator rerun on that base. The operator ran the active-pin model in production mode using the reviewed capture snapshot. Ten auto/BF16 repeats agree. Record that runtime evidence without claiming C++ token acceptance, physical cache measurement, or completion of the remaining cache modes and gaps. Refs #2773. The capture slice remains in draft PR #2932. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [codex] --- .../rocm-qwen35-08b-cpu-gfx1100-numerics.md | 44 ++++++++++++------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md index b609b615d9..95a90be1c3 100644 --- a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md +++ b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md @@ -23,17 +23,30 @@ Results cannot be accepted until both correctness prerequisites pass: `488`), while `VT_ROCM_SKINNY=0` passed. The operator reran PR #2856 at `f06619e4c213e3de28359ee10995e682e8c06932`: CPU mode 2/2, prerequisites 77/77/77/86, `wvSplitK` 79796/79796, and sacred 137/137 with 15 strict, one - tied, maximum gap zero, and zero divergence. PR #2856 is still open and the - contributor lacks merge authority. This prerequisite remains `PENDING` - until a maintainer lands it and the unchanged default gate passes on the - implementation base. Disabling skinny GEMM is diagnostic only. -2. The configured source and runnable ROCm wrapper are still at historical - vLLM `5559679229bc961848b121ccdeaa8fa5d79bec98`. The active source object is - available for read-only inspection, but no runnable active-pin ROCm runtime - has been supplied or proved. Issue #2794 records repository pin-validation - context; #2773 itself owns the cache-matched active-pin Qwen3.5-0.8B captures. - Active-pin capture and token revalidation remain `PENDING` until that runtime - exists and runs the model under the GPU mutex. + tied, maximum gap zero, and zero divergence. PR #2856 landed on `main` as + `d6c63e15ae6825f94dc18769163cfe7b037e7954`. The landing requirement is + satisfied. The unchanged default gate on implementation base + `f98b638673b4d2edc0250eec56d229357ea38ab1` remains `PENDING`. + The earlier runtime results use the historical oracle pin. + Disabling skinny GEMM is diagnostic only. +2. A separate active-pin ROCm runtime ran the pinned Qwen3.5-0.8B model on + gfx1100 under the GPU mutex on 7 September 2026 UTC. The operator used + reviewed capture head `c75173f921cdd344e33ad06260ca181d63b198b5` in production + mode with `auto` resolving to `bfloat16`. All 16 prompts produced 16 tokens + in each of 10 identical repeats. This proves active-pin runtime execution + for that arm. It does not prove physical cache storage or C++ token + acceptance. The configured historical wrapper remains a separate runtime. + Issue #2794 records repository pin-validation context. Issue #2773 owns + the remaining cache-matched captures, teacher-forced gaps, and end-to-end + acceptance, which remain `PENDING`. + +The operator's runtime command, mutex identity, and exit 0 are recorded in +`.cache/gfx1100-resume-20260906/active-pin-install/capture-auto-run.json`. +The capture's aggregate output SHA256 is +`bd4f8f8d1961b34f374e9d6a8b8adae88f99a92010142b046c73e5419603bd8a`. +Its greedy tokens differ from the current historical golden at prompt index 7 +and token indices 8 to 15. Teacher-forced adjudication remains `PENDING`. No permanent +golden changes in this capture-tool slice. Do not create characterization goldens from a known-regressed local default or from the historical oracle revision. @@ -510,10 +523,11 @@ push, open, or merge that pull request. ## Owed -- A maintainer owes the merge decision for reviewed PR #2856; #2772 remains - pending until it lands. -- #2773 owes the runnable active-pin Qwen3.5-0.8B ROCm captures, even though - #2794 supplies repository sync context. +- PR #2856 landed as `d6c63e15ae6825f94dc18769163cfe7b037e7954`. The operator + owes the unchanged default gate on this implementation base. +- #2773 owes the remaining cache-matched active-pin Qwen3.5-0.8B ROCm captures, + teacher-forced gaps, and C++ token acceptance. The production `auto` capture + proves runtime execution only. Issue #2794 supplies repository sync context. - [#2923](https://github.com/mudler/vllm.cpp/issues/2923), owned by `BACKEND-ROCM`, owes routing Qwen3.5 paged attention through `dense_attn::AttnBlock`. This tracked exception records existing debt. It