diff --git a/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md new file mode 100644 index 000000000..95a90be1c --- /dev/null +++ b/.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md @@ -0,0 +1,543 @@ +# Qwen3.5-0.8B CPU and gfx1100 numerical characterization + +Row: `BACKEND-ROCM`. + +Issue: [#2773](https://github.com/mudler/vllm.cpp/issues/2773), which supersedes +issue #1588 for implementation traceability. + +Branch: `row/BACKEND-ROCM-NUMERICS-1588`. + +Active primary oracle pin: vLLM +`e126687a9a828d513c01a07cd69f025f27d63280`. + +## Now + +`PENDING`. The amended spec passed fresh review before capture-tool implementation. +The Python capture tools now implement the provenance and deterministic publication +contract. Fresh implementation review and operator verification remain required. +The C++ characterization probes and real-runtime acceptance remain `PENDING`. +Results cannot be accepted until both correctness prerequisites pass: + +1. Issue #2772 and PR #2856 own the default-on ROCm `wvSplitK` sacred-anchor + repair. The reviewed old base failed prompt 10/token 10 (`369` instead of + `488`), while `VT_ROCM_SKINNY=0` passed. The operator reran PR #2856 at + `f06619e4c213e3de28359ee10995e682e8c06932`: CPU mode 2/2, prerequisites + 77/77/77/86, `wvSplitK` 79796/79796, and sacred 137/137 with 15 strict, one + tied, maximum gap zero, and zero divergence. PR #2856 landed on `main` as + `d6c63e15ae6825f94dc18769163cfe7b037e7954`. The landing requirement is + satisfied. The unchanged default gate on implementation base + `f98b638673b4d2edc0250eec56d229357ea38ab1` remains `PENDING`. + The earlier runtime results use the historical oracle pin. + Disabling skinny GEMM is diagnostic only. +2. A separate active-pin ROCm runtime ran the pinned Qwen3.5-0.8B model on + gfx1100 under the GPU mutex on 7 September 2026 UTC. The operator used + reviewed capture head `c75173f921cdd344e33ad06260ca181d63b198b5` in production + mode with `auto` resolving to `bfloat16`. All 16 prompts produced 16 tokens + in each of 10 identical repeats. This proves active-pin runtime execution + for that arm. It does not prove physical cache storage or C++ token + acceptance. The configured historical wrapper remains a separate runtime. + Issue #2794 records repository pin-validation context. Issue #2773 owns + the remaining cache-matched captures, teacher-forced gaps, and end-to-end + acceptance, which remain `PENDING`. + +The operator's runtime command, mutex identity, and exit 0 are recorded in +`.cache/gfx1100-resume-20260906/active-pin-install/capture-auto-run.json`. +The capture's aggregate output SHA256 is +`bd4f8f8d1961b34f374e9d6a8b8adae88f99a92010142b046c73e5419603bd8a`. +Its greedy tokens differ from the current historical golden at prompt index 7 +and token indices 8 to 15. Teacher-forced adjudication remains `PENDING`. No permanent +golden changes in this capture-tool slice. + +Do not create characterization goldens from a known-regressed local default or +from the historical oracle revision. + +## Scope + +### In scope + +- Add opt-in production-boundary dumps for the residual stream, attention + output, MLP output, stored K/V cache, and persistent GDN convolution/SSM + state. +- Compare CPU and gfx1100 on identical weights and identical token prefixes. +- Report descriptive `max_abs`, RMS, and relative-L2 deltas. +- Port applicable active-pin upstream operation tests with their parameters, + modes, fixtures, failure cases, and exact tolerances. +- Audit runtime dtype, byte width, requested/resolved cache dtype, selected + attention backend, and provider counts for `auto`, `bfloat16`, and + `fp8_e4m3`. +- Run cache-matched active-pin vLLM captures and the established end-to-end + oracle-backed token/near-tie gate after the prerequisites pass. + +### Out of scope + +- Changing a numerical tolerance after observing results. +- Treating CPU as the conformance oracle for FP8 physical-cache behavior. +- Permanently expanding the sacred 16-prompt gate merely because this + characterization measures three cache modes. A new permanent case or golden + may land only when #2773's active-pin capture proves it is necessary and the + final implementation records the evidence. Broader gate policy needs its own + issue. +- Enabling ROCm static graph mode; issue #332 owns that work. +- Performance claims or tuning. + +## Artifact and oracle pins + +Use `Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17` +from `${CHECKPOINT_ROOT}/Qwen3.5-0.8B`. Before every model run, require: + +```sh +sha256sum \ + "${CHECKPOINT_ROOT}/Qwen3.5-0.8B/model.safetensors-00001-of-00001.safetensors" \ + "${CHECKPOINT_ROOT}/Qwen3.5-0.8B/model.safetensors.index.json" \ + "${CHECKPOINT_ROOT}/Qwen3.5-0.8B/config.json" +``` + +Expected hashes are: + +```text +04b1c301231dd422b8860db31311ab2721511346a32cb1e079c4c4e5f1fe4696 model.safetensors-00001-of-00001.safetensors +d8a08838a613b025eb7952ed9db11696213e57e76a375661ef5c12f9dd5dcf4e model.safetensors.index.json +b90b86f35c8e6925ef74ee04d0e758f0a845c83a42089ad82bbaa948de9b4204 config.json +``` + +Resolve the active oracle source with: + +```sh +git -C "${VLLM_SOURCE}" rev-parse e126687a9a828d513c01a07cd69f025f27d63280^{commit} +``` + +The configured checkout may remain detached at the historical revision. Read +active-pin files with `git -C "${VLLM_SOURCE}" show :`; do not move +that checkout. Source availability does not prove runtime gateability. + +## Upstream executing chain + +All cross-file anchors name symbols and refer to active pin `e126687a9` unless +explicitly labeled historical: + +- `vllm/model_executor/models/qwen3_5.py::Qwen3_5ForCausalLM` defines the text + model and layer composition. +- `vllm/model_executor/layers/mamba/mamba_utils.py::MambaStateShapeCalculator.gated_delta_net_state_shape` + defines the GDN convolution and SSM state shapes. It starts at line 258 at + the active pin. `MambaStateShapeCalculator.mamba2_state_shape` is a different + architecture and is not the anchor. +- `tests/kernels/mamba/cpu/test_cpu_gdn_ops.py::test_fused_sigmoid_gating_delta_rule_update_cpu` + and `test_chunk_gated_delta_rule_cpu` define CPU operation tolerances. +- `tests/kernels/mamba/test_gdn_forward_core_split.py::test_forward_core_split_matches_unified` + defines split/unified state and output comparisons. +- `tests/kernels/attention/test_cache.py::test_reshape_and_cache` defines + cache-store dtype behavior and FP8 comparison tolerances. +- `tests/kernels/attention/test_cpu_attn.py::varlen_with_paged_kv` starts at + line 415. Its `_FP8_ATOL` and `_FP8_RTOL` constants at lines 45 and 46 + set `atol` to `0.2` for E4M3 and `0.3` for E5M2, with `rtol=0.1`. + The comparisons at lines 629 to 653 select these bounds for FP8 and + `atol=1.5e-2`, `rtol=1e-2` otherwise. Preserve both split and unsplit checks. + +The earlier spec cited these surfaces from historical `555967922`; those +citations remain diagnostic history but cannot satisfy the active-pin gate. The +implementer must refresh line evidence from the active object and run the active +runtime before acceptance. + +The local production entry is `ModelRegistry::Forward`, which dispatches the +registered `ForwardQwen3_5Dense` factory. Its paged forward reaches +`DenseForwardBody` -> `DenseForwardLayers` -> `RunDenseLayerPaged` -> +`FullAttnBlockPaged` -> `dense_attn::WriteKvCache` -> `vt::ReshapeAndCache` or +`vt::ReshapeAndCacheFp8`. The full-attention caller is +`src/vllm/model_executor/models/qwen3_5.cpp:7758`. `FullAttnBlockPaged` starts +at line 5726 and writes through `dense_attn::WriteKvCache` at +`src/vllm/model_executor/models/qwen3_5.cpp:5909`, before `vt::PagedAttention` +at line 5931. Place the stored-K/V probes after that write. Place GDN probes +after the persistent writes in `GdnBlockPaged` for prefill and decode. + +This Qwen3.5 path does not call `dense_attn::AttnBlock`. Issue #2923 under +`Owed` records this existing shared-seam debt. Characterize the actual path +and cite the executing local and active-pin upstream symbols in the evidence. + +## Dtype and state contract + +The checkpoint resolves model dtype BF16 and explicitly sets +`mamba_ssm_dtype=float32`. The measured surfaces are: + +| Surface | Shape per active unit | Physical storage | +|---|---|---| +| Hidden/residual | `[T, 1024]` | BF16 | +| Full-attention K/V | `[T, 2, 256]` per layer | BF16 or FP8 E4M3 | +| GDN convolution state | Per layer and slot: oracle SD `[3, 6144]`, local DS `[6144, 3]` | BF16 | +| GDN SSM state | `[1, 16, 128, 128]` per layer | FP32 | + +The active oracle defaults to SD physical convolution storage, with axes +`(state_len, dim)`. DS reverses these axes to `(dim, state_len)`. +`vllm/model_executor/layers/mamba/mamba_utils.py::get_conv_state_layout`, lines +28 to 44, returns `SD` without an override. `_orient_conv_shape`, lines 162 +to 166, and `gated_delta_net_state_shape`, lines 268 to 272, produce +`[3, 6144]` per slot for this workload. The kernel consumes a DS view +`[6144, 3]` after the transpose in +`vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py:1296` to line 1303. +Local `MakeQwen3_5KVCacheSpec` stores DS directly in +`src/vllm/model_executor/models/qwen3_5_common.cpp:85`. Leave +`VLLM_SSM_CONV_STATE_LAYOUT` unset for the production oracle. Record physical +layout, kernel-view layout, and actual strides on both sides. A transposed +view does not change the underlying physical layout or byte count. + +Across six full-attention layers, BF16 K/V use 12,288 bytes per cached token +and FP8 K/V use 6,144 bytes. Across 18 GDN layers, convolution state uses +663,552 bytes and SSM state uses 18,874,368 bytes per active slot. Confirm +allocated bytes from runtime specs; arithmetic alone is not evidence. + +For every compared buffer and GEMM output, log the local symbol, active-pin +upstream symbol, runtime dtype, shape, active elements, bytes on each side, and +the reason beside every FP32 exception. An unjustified wider local dtype is a +`DTYPE_WIDTH_MISMATCH` even when tokens agree. + +## State dump design + +Enable dumps only when `VT_DUMP_ACT` and `VT_DUMP_ACT_SUB` name the same empty, +writable directory. Preserve existing keys and add: + +| Key | Production source | Dump dtype | Shape | +|---|---|---|---| +| `state_fa_k` | stored K after `dense_attn::WriteKvCache` | FP32 | `[active_tokens, 512]` | +| `state_fa_v` | stored V after `dense_attn::WriteKvCache` | FP32 | `[active_tokens, 512]` | +| `state_gdn_conv` | persistent convolution cache after update | FP32 | `[active_requests, 18432]` | +| `state_gdn_ssm` | persistent SSM cache after update | FP32 | `[active_requests, 262144]` | + +Widen stored BF16 to FP32. Dequantize stored FP8 with the recorded per-layer +scale. The FP32 file format never conceals separately logged physical dtype and +allocation. + +Use canonical DS order for `state_gdn_conv`, with `0 <= d < 6144` and +`0 <= s < 3`: + +```text +local_dump[r, d * 3 + s] = widen(local[slot_local(r), d, s]) +oracle_dump[r, d * 3 + s] = widen(oracle[slot_oracle(r), s, d]) +``` + +Here `r` follows scheduler request order, and each arm resolves its own slot. +Use the recorded strides and storage offsets for each lookup. Record this +mapping in the manifest. The canonical DS dump order does not prescribe +physical storage or make the oracle's transposed view contiguous. + +Gather nonnegative slot mappings in input-token order and active GDN indices in +scheduler request order. Refuse unexplained duplicate destinations, missing +rows, partial joins, or capacity-wide dumps. A 24-layer step emits 48 state +blobs, two per layer. Place each probe after its production write and preserve +queue ordering. + +## Workload and backend identity + +Use one binary and checkpoint copy. Run one request at concurrency one with +`The capital of France is`, greedy sampling, seed 0, MTP disabled, and eight +output tokens. Keep block size, block count, and scheduler budget equal. Run +each arm and cache mode in a separate process and empty directory. Take two +enabled repeats and one dump-disabled identity control. + +Log and assert requested cache dtype, resolved storage and FP8 interpretation, +normalized selector input, physical cache dtype, selected attention backend, +and actual provider execution for every arm and mode. +`LoadedEngine::ApplyResolvedCacheDType` in +`src/vllm/entrypoints/model_loader.cpp:1835` applies `ParseCacheDType` from +`include/vllm/v1/kv_cache_dtype.h:58` to the cache spec. The runner constructs +`cfg.kv_cache_dtype` from resolved storage and interpretation at +`src/vllm/v1/worker/gpu/runner.cpp:1590`. `KvCacheDTypeName` maps `kBF16` to +`auto` in `src/vllm/v1/attention/backend.cpp:91` to line 96. Therefore the +production CPU selector receives these values for this BF16 workload: + +| Requested cache dtype | Resolved physical storage | Normalized selector input | Selected CPU backend | +|---|---|---|---| +| `auto` | BF16 (`kBF16`) | `auto` | `CPU_ATTN` | +| `bfloat16` | BF16 (`kBF16`) | `auto` | `CPU_ATTN` | +| `fp8_e4m3` | E4M3 bytes (`kI8` plus `kFp8E4M3`) | `fp8_e4m3` | `CPU_ATTN` | + +CPU priority starts with `CPU_ATTN` in `src/vllm/platforms/cpu.cpp:51`. +Its accepted selector values include `auto` and `fp8_e4m3` in +`include/vllm/v1/attention/backends/cpu_attn.h:108` to line 110. +`SelectAttentionBackendName` returns the first valid candidate in +`src/vllm/v1/attention/registry.cpp:124` to line 128. +Selection alone does not prove execution. Independently record the executing +attention and cache-store operator providers, selection counts, declines, and +reference-tier hits. The runner's selected name validates cache configuration +but does not dispatch the model's attention call. If main changes this chain +before implementation, update the cited contract before code and obtain fresh +review. + +CPU is a diagnostic comparison arm for CPU-versus-ROCm deltas, including FP8 +state after dequantization. Active-pin vLLM and its upstream tests are the +conformance authority. CPU results cannot accept or reject FP8 physical-cache +semantics. + +Join only steps with identical input token prefixes. Include the first divergent +output step and exclude all later steps. Record the first divergence and number +of excluded steps. + +## Descriptive metrics and acceptance + +Decode joined blobs to FP32. Let `A` be ROCm and `B` be CPU. Report: + +```text +max_abs = max_i(abs(A_i - B_i)) +rms = sqrt(sum_i((A_i - B_i)^2) / N) +rel_l2 = sqrt(sum_i((A_i - B_i)^2)) / sqrt(sum_i(B_i^2)) +``` + +Define relative L2 as zero when both norms are zero and infinity when only `B` +has zero norm. Report the worst index, both manifests, joined/rejected rows, and +excluded steps. Refuse missing or duplicate keys, dtype/shape/byte mismatches, +nonfinite values, and partial joins. + +These layer metrics are descriptive. They have no propagated numerical +acceptance envelope, discontinuity multiplier, or post-measurement threshold. +The rejected `E(k)` construction is removed because operation counts do not +bound cancellation, conditioning, nonlinear sensitivity, correlated +reductions, FP8 scaling/saturation, or subnormals. + +Acceptance consists of both: + +1. Ported active-pin operation tests with their exact upstream per-surface + tolerances. Examples include GDN `atol=rtol=1e-2` where + `test_cpu_gdn_ops.py` specifies it, exact convolution-state equality in the + split-core test, its dtype-specific output/SSM tolerances, and cache/CPU + attention FP8 tolerances selected by the upstream fixtures. Do not collapse + them into one project-wide tolerance. +2. The established end-to-end 16-prompt strict-token and ratified 500 + milli-nat near-tie gate against deterministic, cache-matched, active-pin + vLLM captures. A new physical cache mode needs its own capture and + teacher-forced gaps before it can be accepted. + +If no upstream analogue exists for a layer boundary, report its metrics only. +A future stage-level acceptance rule requires an independently justified bound +or explicit developer ratification before values are inspected. + +## Provider and end-to-end contract + +For `auto` and `bfloat16`, require the established native operator set, +including nonzero `kReshapeAndCache`, zero `kReshapeAndCacheFp8`, zero declines, +and zero reference-tier hits. The two modes resolve to physical BF16 and must +produce identical local token streams. + +For `fp8_e4m3`, replace the store requirement: require nonzero +`kReshapeAndCacheFp8`, zero `kReshapeAndCache`, nonzero native selections for +all other applicable operators, zero declines, and zero reference-tier hits. +Provider sets are mode-specific; never require a BF16 store from a correct FP8 +run. + +The existing gate remains the permanent gate until #2773 produces reviewed +active-pin evidence for an additional case. Characterization must run all three +modes, but this spec alone does not authorize new permanent goldens. + +## Required capture-tool implementation + +`scripts/qwen3-oracle-capture.py` and `scripts/qwen3-neartie-gap.py` expose the +capture interface below. Their CPU fixtures exercise provenance, refusal, and +NumPy serialization. Active-pin ROCm capture and acceptance remain pending. + +```text +--kv-cache-dtype {auto,bfloat16,fp8_e4m3} +--execution-mode {production,eager} +--seed INT +--max-tokens INT +--repetitions INT +--model-revision REV +--vllm-revision REV +--provenance-out PATH +--vllm-wheel PATH +--runtime-manifest PATH +``` + +`production` must instantiate vLLM without `enforce_eager=True`; eager is a +diagnostic arm and never the denominator. The output must record source, wheel, +image and artifact hashes, complete arguments, a hash of the scripts' shared +16-entry `PROMPTS` list, batching, concurrency, sampling, seed, token count, +repetitions, cache mode, resolved +cache dtype, execution mode, and output hash. + +Extend `test_qwen35_paged_engine` with future environment inputs +`VT_QWEN35_GATE_DIR` and `VT_QWEN35_KV_CACHE_DTYPE`. The first selects an empty +issue-evidence directory instead of the committed golden directory. The second +sets `EngineParams::kv_cache_dtype` and prints requested, resolved, and physical +dtype. These commands describe the intended interface and data flow after +those changes. They are future commands and will fail today: +The launcher must first supply the inspected `VLLM_WHEEL` and +`ORACLE_RUNTIME_MANIFEST` paths. Missing values keep capture pending. + +```sh +VLLM_PIN=e126687a9a828d513c01a07cd69f025f27d63280 +MODEL_REV=2fc06364715b967f1860aea9cf38778875588b17 +MODEL="${CHECKPOINT_ROOT}/Qwen3.5-0.8B" +for mode in auto bfloat16 fp8_e4m3; do + GOLDEN_DIR="evidence/2773/sacred-${mode}" + test ! -e "${GOLDEN_DIR}" + mkdir -p "${GOLDEN_DIR}" + "${VLLM_ORACLE}" scripts/qwen3-oracle-capture.py \ + --model "${MODEL}" --model-revision "${MODEL_REV}" \ + --vllm-revision "${VLLM_PIN}" \ + --vllm-wheel "${VLLM_WHEEL}" --runtime-manifest "${ORACLE_RUNTIME_MANIFEST}" \ + --kv-cache-dtype "${mode}" --execution-mode production --seed 0 \ + --max-tokens 16 --runs 10 --per-prompt --out-dir "${GOLDEN_DIR}" \ + --provenance-out "evidence/2773/oracle-${mode}.json" + VT_QWEN35_GATE_DIR="${GOLDEN_DIR}" \ + VT_QWEN35_KV_CACHE_DTYPE="${mode}" \ + VT_DUMP_IDS=1 build-rocm/tests/test_qwen35_paged_engine + test -s "${GOLDEN_DIR}/our_ids.i32" + "${VLLM_ORACLE}" scripts/qwen3-neartie-gap.py \ + --model "${MODEL}" --model-revision "${MODEL_REV}" \ + --vllm-revision "${VLLM_PIN}" \ + --vllm-wheel "${VLLM_WHEEL}" --runtime-manifest "${ORACLE_RUNTIME_MANIFEST}" \ + --kv-cache-dtype "${mode}" --execution-mode production --seed 0 \ + --max-tokens 16 --topk 20 --golden-dir "${GOLDEN_DIR}" \ + --provenance-out "evidence/2773/neartie-${mode}.json" + VT_QWEN35_GATE_DIR="${GOLDEN_DIR}" \ + VT_QWEN35_KV_CACHE_DTYPE="${mode}" \ + build-rocm/tests/test_qwen35_paged_engine +done +``` + +The tools must refuse when their `PROMPTS` lists differ from each other or from +`tests/parity/test_qwen35_paged_engine.cpp::Prompts`. Record exact as-run +commands. Do not publish a Qwen3.5 sacred candidate unless at least 10 repeats +are deterministic. Select this regime from the artifact's verified +`config.json`, including its architecture, model type, and nested text model +type. Any Qwen3.5 indicator selects the strict regime. Require the runtime's +resolved model identity to agree. A missing or inconsistent identity refuses +publication. No command-line option can downgrade this regime. + +Legacy distributional captures retain their existing caller contracts outside +Qwen3.5. Their manifests identify the legacy regime, including observed +nondeterminism and missing provenance. They cannot supply a Qwen3.5 near-tie +capture. The near-tie tool validates regime, model and oracle identity, cache +mode, execution mode, prompts, sampling, token count, and input hashes before +using a Qwen3.5 capture. Existing legacy captures without manifests remain +usable only outside the strict regime. + +Strict capture requires `--vllm-wheel` and `--runtime-manifest`. The launcher +manifest supplies `vllm_revision`, `wheel_sha256`, and `image_digest`, whose +value is an immutable `sha256:` digest. Hash the wheel archive and compare its +package members with the imported package bytes. Independently verify the +imported source or installed VCS revision against the requested revision. +Record the actual source/package hashes and every model artifact hash. +Reject any revision or hash disagreement. Record image identity as an external +launcher attestation with its verification limit. Do not claim that the Python +process independently measured its container image. Missing wheel, image, +source, or artifact identity prevents strict publication. These metadata checks +do not prove runtime gateability, model correctness, or GPU execution. + +The internal CPU/ROCm state characterization remains eight output tokens as +specified under `Workload and backend identity`. The permanent sacred-gate +candidate uses the existing 16-prompt, 16-output-token regime above. Never use +the eight-token characterization files as sacred-gate goldens. + +## Tests and review mutations + +The capture-tool slice used failing CLI and publication tests before implementation. +The later state implementation starts with focused tests that fail because the four +state rows are absent. It must test row shape/dtype, +nonmonotonic and negative slot mappings, inactive-capacity exclusion, stored +BF16 widening, post-quantization FP8 dequantization, persistent GDN prefill and +decode state, incomplete-step refusal, comparator structural refusals and +zero-denominator rules, dtype-width refusal, backend/dtype logging, and +mode-appropriate provider sets. The checkpoint-backed case must enter through +`LoadedEngine::FromModelDir`. + +A fresh reviewer mutates each guarantee in a scratch copy: remove each +production probe, substitute pre-write tensors, remove negative-slot filtering, +sort by physical slot, dump capacity, misreport FP8 as BF16, swap the mode's +cache-store operator, inject a reference-tier hit, remove the production entry +point, and force eager oracle mode. Each focused test must fail for the intended +reason and the reviewer must restore the tree byte for byte. + +The spec-only repair changed no runtime behavior. The capture-tool slice exercises +`IMP-TEST-FIRST` and `IMP-MUTATE` with CPU fixtures. State probes and real-runtime +gates remain future work. + +## Gate order + +The later implementation must satisfy, in order: + +1. #2772 lands and the unchanged default local gate passes on the chosen base. +2. A runnable active-pin ROCm vLLM runtime is identified and proves this model. +3. Artifact, source, wrapper, wheel, image, prompt, and output hashes are saved. +4. The capture tools fail first for missing options, then pass focused tests. +5. Ported active-pin operation tests pass with unchanged upstream tolerances. +6. Dump/comparator CPU tests and checkpoint-backed CPU trace pass. +7. The controlled full preflight runs; every skip remains `PENDING`. +8. Fresh immutable review detects every required mutation. +9. Two CPU and two gfx1100 trace repeats plus disabled controls pass per mode. +10. Cache-matched active-pin production captures are deterministic and the + established end-to-end gate passes per measured mode. +11. Mode-appropriate native selections are nonzero, with zero declines and + zero reference-tier hits. +12. The operator independently reruns focused, full, oracle, and hardware gates. + +The capture-tool slice claims CPU fixture tests and mutations only. It claims no +GPU result, oracle result, model execution, or numerical runtime mutation. + +## Evidence required + +Store evidence under an issue-specific durable path and record commit/tree +hashes, clean status, exact commands/statuses, every skip, active oracle and +artifact hashes, compiler/build/ROCm/driver/board identity, GPU mutex evidence, +requested/resolved/physical cache dtype, selected CPU and ROCm attention +backends, provider counts, enabled repeats, disabled controls, manifests, raw +dumps, metric tables, first divergence, excluded steps, upstream tolerance +anchors, red-first output, mutations, fresh review, and operator reruns. + +Classify structural and execution failures before interpreting metrics: +`ARTIFACT_MISMATCH`, `INSTRUMENTATION_FAIL`, `NONDETERMINISTIC`, +`STRUCTURE_MISMATCH`, `DTYPE_WIDTH_MISMATCH`, `CORRECTNESS_FAIL`, or +`NONFINITE`. When none applies, report descriptive metrics and the separate +operation/end-to-end acceptance results. Do not invent `WITHIN_DTYPE_ENVELOPE` +or `ORDERING_DRIFT` labels. + +## Risks + +- Probes can synchronize queues; disabled controls detect output perturbation. +- FP8 boundaries can change codes; compare dequantized values and record bytes. +- Different valid reduction orders can produce descriptive CPU/ROCm deltas. +- Backend fallback can change the diagnostic denominator; explicit backend + logging exposes it. +- Greedy divergence makes later states incomparable; exclude them. +- The active source object can be read while the active runtime remains absent; + never convert source availability into a gate pass. + +## Stop conditions + +- Stop before implementation until this spec receives a fresh `PASS`. +- Keep correctness `PENDING` until #2772 lands and the unchanged default gate + passes on the implementation base. +- Stop active-pin capture until a runnable active-pin ROCm runtime is proved. +- Stop on a revision/hash/configuration mismatch, dump perturbation, + nondeterminism, incomplete manifest, unexplained backend, provider decline, + reference-tier hit, wrong board, missing GPU mutex, or divergent input prefix. +- Stop before changing tolerances or permanent sacred-gate scope after seeing + results. +- File and assign a new issue before fixing any unexpected defect outside + #2773. + +## Git integration + +Use one pull request for the committed spec and later implementation, following +the recorded repository default. Preserve rejected spec commit `7bc2546e9` in +history. The eventual pull request body must name row `BACKEND-ROCM`, link and +close #2773, and carry the required trailers. The spec implementer does not +push, open, or merge that pull request. + +## Owed + +- PR #2856 landed as `d6c63e15ae6825f94dc18769163cfe7b037e7954`. The operator + owes the unchanged default gate on this implementation base. +- #2773 owes the remaining cache-matched active-pin Qwen3.5-0.8B ROCm captures, + teacher-forced gaps, and C++ token acceptance. The production `auto` capture + proves runtime execution only. Issue #2794 supplies repository sync context. +- [#2923](https://github.com/mudler/vllm.cpp/issues/2923), owned by + `BACKEND-ROCM`, owes routing Qwen3.5 paged attention through + `dense_attn::AttnBlock`. This tracked exception records existing debt. It + does not waive or satisfy the shared-seam requirement. Wiring needs its own + reviewed spec and implementation, outside #2773's instrumentation scope. +- The amended spec passed fresh review before capture-tool implementation. +- A fresh implementer owes the state probes, tests, comparator, + and provider checks after the prerequisites pass. +- A fresh implementation reviewer owes static review and every mutation. +- The operator owes the independent controlled and gfx1100 gates. +- The final implementation adds `## Outcome` with measured results, rejected + alternatives, and reasons for defaults. +- Issue #332 owns the graph-enabled repetition. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0f8a0b075..d8d2dd8f7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -541,6 +541,8 @@ jobs: run: | sudo apt-get update -qq sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-numpy + python3 tests/scripts/test_qwen3_capture_tools.py + python3 tests/scripts/test_qwen3_capture_outputs.py python3 tests/scripts/test_ltx25_render_compare.py && python3 tests/scripts/test_ltx25_absolute_reference.py # The prompt-adherence half of the same tool (#2295, owning #1854's # first sub-question). 42 of its 47 cases need numpy only; the five diff --git a/docs/USAGE.md b/docs/USAGE.md index ca92c355f..73913cb27 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1220,6 +1220,57 @@ checkout at exactly the recorded byte counts, so those two sizes are locally confirmed. The distilled bf16 DiT is absent from the shared checkout, so its 42,018,190,584 bytes rest on the tree listing and a range request alone. +## Capture Qwen oracle evidence + +Use `scripts/qwen3-oracle-capture.py` for greedy continuations and +`scripts/qwen3-neartie-gap.py` for gaps under the local engine's exact token +prefix. Both accept `--kv-cache-dtype auto|bfloat16|fp8_e4m3`, `--seed`, +`--max-tokens`, and `--runs`. `--repetitions` is an alias for `--runs`. +Production execution is the default. `--execution-mode eager` and its +`--enforce-eager` alias select diagnostics, which cannot be a denominator. + +Qwen3.5 captures require verified local model artifacts and at least 10 +deterministic repeats. Supply full model and vLLM revisions, the installed +vLLM wheel archive, and a launcher JSON manifest. That manifest contains +`vllm_revision`, `wheel_sha256`, and an immutable `sha256:` `image_digest`. +Model files need matching HuggingFace download metadata or immutable cache +snapshot paths. The tools hash the files and compare wheel members with the +imported package. An installed `+g` version suffix verifies only its recorded +VCS prefix. Image identity remains a launcher attestation. + +After supplying these paths and revisions, use an empty capture directory: + +```sh +python3 scripts/qwen3-oracle-capture.py \ + --model "$MODEL_DIR" --model-revision "$MODEL_REV" \ + --vllm-revision "$VLLM_REV" --vllm-wheel "$VLLM_WHEEL" \ + --runtime-manifest "$RUNTIME_MANIFEST" --kv-cache-dtype auto \ + --execution-mode production --seed 0 --max-tokens 16 --runs 10 \ + --per-prompt --out-dir "$CAPTURE_DIR" +``` + +Place the matching local gate's `our_ids.i32` dump in that directory before +running `qwen3-neartie-gap.py`. Pass the same model, revisions, runtime inputs, +cache mode, seed, and token count, replacing `--out-dir` with `--golden-dir`. +The near-tie tool uses one request at a time, so its strict input capture must +use `--per-prompt`. Each tool writes a provenance JSON file beside its NumPy +outputs. `--provenance-out PATH` writes an identical additional copy. Strict +captures refuse existing outputs and mismatched or incomplete evidence. +Output and provenance paths cannot overwrite capture inputs, including symbolic +links and hardlinks to those inputs. + +`sampling_normalized` records the supplied `SamplingParams` after constructor +normalization. vLLM resolves engine requests on a clone. `sampling_resolved` +remains null, and `sampling_resolution` records that observation limit. +Teacher-forcing sampling carries the same qualification. + +Legacy Qwen3 distributional calls remain usable, including captures without +complete provenance and near-tie inputs without manifests. Their manifests +record missing provenance and observed nondeterminism. A legacy capture cannot +supply a Qwen3.5 near-tie run. These tool checks do not establish GPU execution +or accept new permanent goldens. Active-pin acceptance remains pending in +[#2773](https://github.com/mudler/vllm.cpp/issues/2773). + ## Look up interface details [Reference pages](reference/README.md) collect dense lookup material such as diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index c91e42611..f8e6c0d5d 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -438,6 +438,17 @@ echo "Mutation suites:" for suite in "${SUITES[@]}"; do run "$suite" python3 "tests/scripts/$suite.py" done +# QWEN3-CAPTURE-TOOLS: begin +if python3 -c 'import numpy' >/dev/null 2>&1; then + run "test_qwen3_capture_tools" python3 tests/scripts/test_qwen3_capture_tools.py + run "test_qwen3_capture_outputs" python3 tests/scripts/test_qwen3_capture_outputs.py +else + for suite in test_qwen3_capture_tools test_qwen3_capture_outputs; do + skip "$suite" "PENDING: numpy is not importable. CI installs python3-numpy for these capture suites." + done +fi +# QWEN3-CAPTURE-TOOLS: end + # THE ONE SUITE HERE WITH A THIRD-PARTY DEPENDENCY (#1612). It exercises # `scripts/ltx25-render-compare.py`, whose only import beyond the standard # library is numpy -- the tool reads PPM and WAV by hand precisely so that a diff --git a/scripts/qwen3-neartie-gap.py b/scripts/qwen3-neartie-gap.py index 692a70e27..b2bd46983 100644 --- a/scripts/qwen3-neartie-gap.py +++ b/scripts/qwen3-neartie-gap.py @@ -23,9 +23,20 @@ # PATH="${VLLM_ORACLE}/bin:$PATH" "${VLLM_ORACLE}/bin/python" \ # scripts/qwen3-neartie-gap.py --model Qwen/Qwen3-4B \ # --golden-dir tests/parity/goldens/qwen3_greedy_4b +# +# Qwen3.5 additionally requires a matching strict capture manifest and ten +# identical raw-logprob repeats. See docs/USAGE.md for its provenance inputs. import argparse, os, sys +import hashlib +import io +import math +from pathlib import Path +import re import numpy as np +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import qwen3_oracle_common as common + PROMPTS = [ "The capital of France is", "Once upon a time,", "In the beginning God created", "The quick brown fox jumps over", "def fibonacci(n):", @@ -49,80 +60,157 @@ def _parse_args(argv=None): # over-commits and REBOOTS the box. 0.40 is the safe ceiling (does not affect the # teacher-forced logprobs — only KV-cache capacity, unused here at max_tokens=1). ap.add_argument("--gpu-mem-util", type=float, default=0.40) - ap.add_argument("--enforce-eager", action="store_true", - help="use eager execution for diagnostics instead of the " - "production execution mode") - return ap.parse_args(argv) + ap.add_argument("--runs", "--repetitions", type=int, default=None) + common.add_options(ap) + return common.finish_args(ap, argv) def _llm_kwargs(args): - return { - "model": args.model, - "dtype": "bfloat16", - "enforce_eager": args.enforce_eager, - "gpu_memory_utilization": args.gpu_mem_util, - } + return common.llm_kwargs(args, args.gpu_mem_util) def _mode_narration(args): - eager = args.enforce_eager - mode = "eager diagnostic" if eager else "production" - return f"oracle execution mode: {mode} (enforce_eager={eager})" + return common.mode_narration(args) + + +def validate_capture(context, capture, inputs): + common.require(capture.get("tool") == "qwen3-oracle-capture" + and capture.get("regime") == "qwen3_5_strict" + and capture.get("deterministic") is True + and isinstance(capture.get("repetitions"), int) and capture["repetitions"] >= 10, + "near-tie input is not a deterministic strict capture") + for key in ("provenance_status", "resolved_model_identity", "prompts_sha256", "sampling", + "sampling_normalized", "sampling_resolved", "sampling_resolution", "execution_mode", "batching"): + common.require(capture.get(key) == context[key], f"capture {key} differs") + for key in ("requested", "resolved"): + common.require(capture.get("cache", {}).get(key) == context["cache"][key], f"capture cache {key} differs") + for key in ("requested_revision", "identity", "files", "missing"): + common.require(capture.get("model", {}).get(key) == context["model"][key], f"capture model {key} differs") + for key in ("requested_revision", "package_files", "missing", "version"): + common.require(capture.get("runtime", {}).get(key) == context["runtime"][key], f"capture oracle {key} differs") + runtime = capture["runtime"] + revision = runtime.get("revision") + verification = runtime.get("revision_verification") + valid_revision = verification == "clean_git_source" and common.full_revision(revision) + if verification == "installed_version_vcs_prefix": + # A captured installed revision keeps its observed prefix, independently + # checked against the current verified full revision and package bytes. + match = re.search(r"(?:\+|\.)g([0-9a-f]{7,40})(?:[.+-]|$)", str(runtime.get("version"))) + valid_revision = match is not None and revision == match.group(1) + common.require(valid_revision and context["runtime"]["requested_revision"].startswith(revision), + "capture observed oracle revision is missing, unverified, or differs") + for key, field in (("wheel", "sha256"), ("image", "digest")): + common.require(capture.get("runtime", {}).get(key, {}).get(field) == context["runtime"][key][field], + f"capture oracle {key} differs") + expected = {"greedy_ids.npy", "greedy_dist.npy", *(f"p{i}_prompt.i32" for i in range(len(PROMPTS)))} + outputs = capture.get("outputs", {}) + common.require(set(outputs) == expected, "capture output manifest is incomplete") + common.require(capture.get("output_sha256") == hashlib.sha256(common.json_bytes(outputs)).hexdigest(), + "capture output manifest hash differs") + for name in expected: + common.require(name in inputs and outputs[name] == { + "sha256": hashlib.sha256(inputs[name]).hexdigest(), "size": len(inputs[name])}, + f"capture input hash differs: {name}") def main(): args = _parse_args() + prompt_record = common.check_prompts(__file__, PROMPTS) + model = common.model_identity(args) + strict = common.is_qwen35(model["identity"]) + if args.runs is None: + args.runs = 10 if strict else 1 + common.strict_inputs(args, model) + import vllm from vllm import LLM, SamplingParams - N, T = len(PROMPTS), args.max_tokens - our = np.fromfile(os.path.join(args.golden_dir, "our_ids.i32"), - dtype="= 0), "teacher forcing needs a complete nonnegative token stream", "STRUCTURE_MISMATCH") + if strict: + dist = np.load(io.BytesIO(inputs["greedy_dist.npy"]), allow_pickle=False) + common.require(dist.shape == (N, T, capture["repetitions"]) and dist.dtype == np.dtype(" max_gap: - max_gap, worst = gap, (i, j, gap) + data = inputs[f"p{i}_prompt.i32"] + common.require(data and len(data) % 4 == 0, "prompt token file is incomplete", "STRUCTURE_MISMATCH") + tokens = np.frombuffer(data, dtype="= 0 for token in tokens), "prompt contains a negative token", "STRUCTURE_MISMATCH") + prefixes.append(tokens) + sp_args = {"temperature": 0.0, "max_tokens": 1, "prompt_logprobs": args.topk, "seed": args.seed or 0} + common.record_sampling(context, SamplingParams(**sp_args), key="teacher_forcing_sampling") + reference = [] + gap_mnats = np.zeros((N, T), dtype="/qwen3_greedy_4b +# +# The historical Qwen3 distributional workflow above remains available. +# Qwen3.5 uses verified provenance and at least ten deterministic repeats +# under #2773. See docs/USAGE.md for its capture and launcher inputs. import argparse +import io +from pathlib import Path import os import sys import numpy as np +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import qwen3_oracle_common as common + PROMPTS = [ "The capital of France is", "Once upon a time,", @@ -66,7 +75,7 @@ def _parse_args(argv=None): ap.add_argument("--model", default=os.environ.get("QWEN3_MODEL", "Qwen/Qwen3-0.6B")) ap.add_argument("--out-dir", default=None, help="golden dir (default tests/parity/goldens/qwen3_greedy_0_6b)") - ap.add_argument("--runs", type=int, default=int(os.environ.get("QWEN3_RUNS", "10")), + ap.add_argument("--runs", "--repetitions", type=int, default=int(os.environ.get("QWEN3_RUNS", "10")), help="K greedy runs to build the observed distribution (K>=1)") ap.add_argument("--max-tokens", type=int, default=16) ap.add_argument("--gpu-mem", type=float, @@ -77,25 +86,20 @@ def _parse_args(argv=None): help="decode each prompt in its OWN generate() call (batch size 1) " "to match the paged-engine gate's single-request decode " "regime; otherwise all prompts are batched in one call") - ap.add_argument("--enforce-eager", action="store_true", - help="use eager execution for diagnostics instead of the " - "production execution mode") - return ap.parse_args(argv) + common.add_options(ap) + return common.finish_args(ap, argv) + + +def parse_args(argv=None): + return _parse_args(argv) def _llm_kwargs(args): - return { - "model": args.model, - "dtype": "bfloat16", - "enforce_eager": args.enforce_eager, - "gpu_memory_utilization": args.gpu_mem, - } + return common.llm_kwargs(args, args.gpu_mem) def _mode_narration(args): - eager = args.enforce_eager - mode = "eager diagnostic" if eager else "production" - return f"oracle execution mode: {mode} (enforce_eager={eager})" + return common.mode_narration(args) def generate_all(llm, sp, per_prompt): @@ -113,66 +117,64 @@ def generate_all(llm, sp, per_prompt): def main(): args = _parse_args() + prompt_record = common.check_prompts(__file__, PROMPTS) + model = common.model_identity(args) + common.strict_inputs(args, model) + import vllm from vllm import LLM, SamplingParams - out_dir = args.out_dir or default_out_dir() - os.makedirs(out_dir, exist_ok=True) - N = len(PROMPTS) - T = args.max_tokens - K = max(1, args.runs) - + strict = common.is_qwen35(model["identity"]) + runtime = common.runtime_identity(vllm, args, strict) if strict else None print(_mode_narration(args), flush=True) llm = LLM(**_llm_kwargs(args)) - sp = SamplingParams(temperature=0.0, max_tokens=T) - - # runs[k][i] = list of token ids for prompt i on run k (padded to T with -1). + runtime = runtime or common.runtime_identity(vllm, args, False) + context = common.resolved_context(args, llm, model, runtime, prompt_record, + 1 if args.per_prompt else len(PROMPTS)) + context["tool"] = "qwen3-oracle-capture" + strict = context["regime"] == "qwen3_5_strict" + out_dir = Path(args.out_dir or default_out_dir()) + common.require(not strict or not out_dir.exists() or not any(out_dir.iterdir()), + f"capture directory is not empty: {out_dir}") + N, T, K = len(PROMPTS), args.max_tokens, args.runs + sp = SamplingParams(temperature=0.0, max_tokens=T, seed=args.seed or 0) + common.record_sampling(context, sp) dist = np.full((N, T, K), -1, dtype=" {ids} ({o.outputs[0].text!r})") - - np.save(os.path.join(out_dir, "greedy_ids.npy"), run0) - np.save(os.path.join(out_dir, "greedy_dist.npy"), dist) - - # ---- determinism report ------------------------------------------------- - print(f"\n=== determinism report: {args.model} N={N} T={T} K={K} ===") - deterministic = True - total_multi_pos = 0 - for i in range(N): - # distinct full sequences across the K runs - seqs = {tuple(int(x) for x in dist[i, :, k]) for k in range(K)} - # per-position distinct-token count - multi = [j for j in range(T) if len({int(dist[i, j, k]) for k in range(K)}) > 1] - total_multi_pos += len(multi) - if len(seqs) > 1: - deterministic = False - print(f" prompt[{i}] NON-DET: {len(seqs)} distinct sequences; " - f"near-tie positions {multi}") - for j in multi: - observed = sorted({int(dist[i, j, k]) for k in range(K)}) - print(f" pos {j:2d}: observed tokens {observed}") - else: - print(f" prompt[{i}] deterministic (1 sequence over {K} runs)") - print(f"=== {'ALL DETERMINISTIC' if deterministic else 'NON-DETERMINISTIC'} " - f"over K={K}; {total_multi_pos} multi-member (prompt,pos) cells ===") - print(f"wrote {out_dir}/greedy_ids.npy {run0.shape} + " - f"{out_dir}/greedy_dist.npy {dist.shape}") + prompt_ids.append(prefix) + else: + deterministic &= prefix == prompt_ids[i] and np.array_equal(dist[i, :, k], dist[i, :, 0]) + context["deterministic"] = bool(deterministic) + common.require(not strict or deterministic, "oracle tokens or prompt tokenization changed across repeats", "NONDETERMINISTIC") + common.confirm_inputs(args, vllm, context, __file__, PROMPTS) + run0 = dist[:, :, 0].copy() + payloads = {f"p{i}_prompt.i32": np.asarray(ids, dtype="= 2, f"incomplete download metadata for {relative}") + revision, etag = lines[:2] + elif root.parent.name == "snapshots" and path.is_symlink(): + revision, etag = root.name, path.resolve().name + if revision is None: + missing.append(f"artifact revision: {relative}") + continue + require(revision == args.model_revision, f"model revision differs for {relative}") + if re.fullmatch(r"[0-9a-f]{64}", etag or ""): + require(record["sha256"] == etag, f"model hash differs for {relative}") + elif re.fullmatch(r"[0-9a-f]{40}", etag or ""): + require(git_blob_sha1(path) == etag, f"model Git-blob hash differs for {relative}") + else: + require(False, f"unsupported artifact identity for {relative}") + record["revision"] = revision + if not records: + missing.append("model artifacts") + result = {"path": str(root), "requested_revision": args.model_revision, + "identity": observed, "files": records, "missing": missing} + if strict: + require(not missing, "strict model identity is incomplete: " + ", ".join(missing)) + require(observed["model_type"] and observed["architectures"], "strict model identity is incomplete") + require("text_config" not in config or observed["text_model_type"], "strict text model identity is incomplete") + return result + + +def distribution_metadata_files(distribution): + """Record the filesystem metadata read by Python's selected distribution.""" + # CPython importlib.metadata: Distribution.version reads metadata, whose + # read_text calls try METADATA, PKG-INFO, then an old egg-info file. + # PathDistribution joins those names to _path, not to the package root or + # a guessed version directory. RECORD need not exist. + root = getattr(distribution, "_path", None) + if not isinstance(distribution, importlib.metadata.PathDistribution) or not isinstance(root, Path): + return {} + records = {} + for name in ("METADATA", "PKG-INFO", ""): + path = root / name + consumed = distribution.read_text(name) + if consumed is not None: + records[str(path.absolute())] = file_record(path) + if consumed: + break + return records + + +def runtime_identity(module, args, strict): + missing = [] + requested = args.vllm_revision + if requested: + require(full_revision(requested), "vLLM revision must be a full commit SHA") + else: + missing.append("requested vLLM revision") + filename = getattr(module, "__file__", None) + package = Path(filename).resolve().parent if filename else None + files = {} + if package and package.is_dir(): + files = {"vllm/" + str(path.relative_to(package)): file_record(path) + for path in sorted(package.rglob("*")) if path.is_file() + and "__pycache__" not in path.parts and path.suffix != ".pyc"} + if not files: + missing.append("imported package bytes") + version = getattr(module, "__version__", None) + metadata_files = {} + revision, verification, source_root = None, None, None + if package: + def git(*argv): + try: + return subprocess.run(["git", "-C", str(package), *argv], capture_output=True, text=True, + env=dict(os.environ, GIT_NO_LAZY_FETCH="1", GIT_OPTIONAL_LOCKS="0")) + except FileNotFoundError: + return subprocess.CompletedProcess(argv, 127, "", "git is unavailable") + probe = git("ls-files", "--error-unmatch", Path(filename).name) + if probe.returncode == 0: + head = git("rev-parse", "HEAD") + require(head.returncode == 0, "cannot read imported source revision") + revision, verification = head.stdout.strip(), "clean_git_source" + source_root = git("rev-parse", "--show-toplevel").stdout.strip() + require(git("diff", "--quiet", "HEAD", "--", source_root).returncode == 0, + "imported vLLM source has tracked changes") + try: + distribution = importlib.metadata.distribution("vllm") + except importlib.metadata.PackageNotFoundError: + distribution = None + if distribution: + require(str(distribution.version) == str(version), "imported and installed vLLM versions differ") + metadata_files = distribution_metadata_files(distribution) + prefix_match = re.search(r"(?:\+|\.)g([0-9a-f]{7,40})(?:[.+-]|$)", str(version)) + prefix = prefix_match.group(1) if prefix_match else None + if revision and prefix: + require(revision.startswith(prefix), "installed version disagrees with imported Git source") + if revision is None and prefix: + revision, verification = prefix, "installed_version_vcs_prefix" + if revision is None: + missing.append("observed source or installed VCS revision") + elif requested: + require(requested.startswith(revision), "observed vLLM revision differs from requested revision") + wheel = {"path": args.vllm_wheel, "sha256": None} + if args.vllm_wheel: + wheel["sha256"] = sha256(args.vllm_wheel) + try: + archive = zipfile.ZipFile(args.vllm_wheel) + except zipfile.BadZipFile as error: + raise CaptureError("ARTIFACT_MISMATCH: vLLM wheel is not a valid archive") from error + with archive: + members = [entry for entry in archive.infolist() + if entry.filename.startswith("vllm/") and not entry.is_dir() + and "__pycache__" not in entry.filename.split("/") + and not entry.filename.endswith(".pyc")] + require(len({entry.filename for entry in members}) == len(members), "duplicate wheel package member") + require({entry.filename for entry in members} == set(files), "wheel and imported package file sets differ") + for entry in members: + digest = hashlib.sha256() + with archive.open(entry) as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + require(digest.hexdigest() == files[entry.filename]["sha256"], + f"wheel and imported package bytes differ: {entry.filename}") + else: + missing.append("vLLM wheel") + image = {"digest": None, "source": "launcher_attestation", + "verification": "The Python process did not independently measure its container image."} + manifest_hash = None + if args.runtime_manifest: + launcher = read_json(args.runtime_manifest) + require(full_revision(launcher.get("vllm_revision")), "launcher vLLM revision is missing or invalid") + require(launcher["vllm_revision"] == requested, "launcher and requested vLLM revisions differ") + require(launcher.get("wheel_sha256") == wheel["sha256"] and wheel["sha256"] is not None, + "launcher and measured wheel hashes differ") + require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(launcher.get("image_digest"))), + "launcher image identity must be an immutable sha256 digest") + image["digest"] = launcher["image_digest"] + manifest_hash = sha256(args.runtime_manifest) + else: + missing.append("launcher image attestation") + if strict: + require(not missing, "strict runtime identity is incomplete: " + ", ".join(missing)) + return {"version": version, "revision": revision, "requested_revision": requested, + "revision_verification": verification, + "revision_limit": "An installed +g suffix verifies only the recorded VCS prefix; the full requested SHA is separate.", + "source_root": source_root, "package_files": files, "distribution_metadata": metadata_files, + "wheel": wheel, "image": image, + "launcher_manifest_sha256": manifest_hash, "missing": missing} + + +def resolved_context(args, llm, model, runtime, prompt_record, batch_size): + config = getattr(getattr(llm, "llm_engine", None), "vllm_config", None) + model_config = getattr(config, "model_config", None) + resolved_identity = identity(getattr(model_config, "hf_config", None)) + strict = is_qwen35(model["identity"]) or is_qwen35(resolved_identity) + if strict: + require(is_qwen35(model["identity"]) and not model["missing"], "strict model artifacts are not verified") + require(model["identity"] == resolved_identity, "artifact and runtime model identities differ") + require(not runtime["missing"], "strict runtime identity is incomplete") + require(args.runs >= 10, "strict capture requires at least ten deterministic repeats") + dtype = str(getattr(model_config, "dtype", "unknown")).removeprefix("torch.") + reported_cache = getattr(getattr(config, "cache_config", None), "cache_dtype", None) + resolved_cache = dtype if reported_cache == "auto" else reported_cache + requested_cache = args.kv_cache_dtype or "auto" + expected_cache = "bfloat16" if requested_cache == "auto" else requested_cache + if strict: + require(dtype == "bfloat16" and resolved_cache == expected_cache, "requested and resolved cache dtypes differ") + require(getattr(model_config, "enforce_eager", None) == args.enforce_eager, "runtime execution mode differs") + require(getattr(model_config, "seed", None) == (args.seed or 0), "runtime seed differs") + require(getattr(model_config, "revision", None) == args.model_revision + and getattr(model_config, "tokenizer_revision", None) == args.model_revision, + "runtime model or tokenizer revision differs") + return {"schema_version": 1, "regime": "qwen3_5_strict" if strict else "legacy_distributional", + "provenance_status": "incomplete" if model["missing"] or runtime["missing"] else "complete", + "arguments": vars(args), "command": [sys.executable, *sys.argv], + "model": model, "runtime": runtime, "resolved_model_identity": resolved_identity, + "cache": {"requested": requested_cache, "reported_selector": reported_cache, + "resolved": resolved_cache, "physical_dtype": None, + "verification": "Resolved configuration only; physical cache storage is not measured."}, + "execution_mode": args.execution_mode, + "purpose": "diagnostic" if args.enforce_eager else "production_capture", + "batching": {"batch_size": batch_size, "concurrency": batch_size}, + "sampling": {"temperature": 0.0, "max_tokens": args.max_tokens, "seed": args.seed or 0}, + "repetitions": args.runs, **prompt_record} + + +def strict_inputs(args, model): + if is_qwen35(model["identity"]): + require(args.vllm_revision and args.vllm_wheel and args.runtime_manifest, + "strict capture requires vLLM revision, wheel, and runtime manifest") + require(args.runs >= 10, "strict capture requires at least ten deterministic repeats") + + +def confirm_inputs(args, module, context, script_path, prompts): + require(model_identity(args) == context["model"], "model artifacts changed during capture") + require(runtime_identity(module, args, context["regime"] == "qwen3_5_strict") == context["runtime"], + "runtime identity changed during capture") + require(check_prompts(script_path, prompts) == {key: context[key] for key in ("prompts_sha256", "scripts")}, + "capture scripts or prompts changed during capture") + + +def sampling_record(params): + """Serialize the supplied sampling object without claiming engine resolution.""" + def encode(value): + if value is None or isinstance(value, (str, bool, int, float)): + return value + if isinstance(value, dict): + return {str(key): encode(item) for key, item in value.items()} + if isinstance(value, (set, frozenset)): + return [encode(item) for item in sorted(value, key=repr)] + if isinstance(value, (list, tuple)): + return [encode(item) for item in value] + return str(value) + names = getattr(params, "__struct_fields__", None) + values = vars(params) if names is None else {name: getattr(params, name) for name in names} + return {name: encode(value) for name, value in values.items()} + + +def record_sampling(context, params, key="sampling"): + context[key + "_normalized"] = sampling_record(params) + context[key + "_resolved"] = None + # Pinned vLLM v1/engine/input_processor.py:356,364-369 resolves a clone. + # Reading the supplied object after generate() cannot observe that request. + context[key + "_resolution"] = { + "status": "unobserved", + "limit": "vLLM resolves a cloned request using generation config and tokenizer; " + "the engine-resolved values are not observed.", + } + context[key] = {name: getattr(params, name) for name in ("temperature", "max_tokens", "seed")} + + +def capture_input_paths(args, module, context, script_path): + """Name the files used to verify this capture before publication.""" + model_root = Path(context["model"]["path"]) + paths = {model_root / name for name in context["model"]["files"]} + paths.update(model_root / ".cache/huggingface/download" / (name + ".metadata") + for name in context["model"]["files"]) + filename = getattr(module, "__file__", None) + if filename: + package_root = Path(filename).resolve().parent.parent + paths.update(package_root / name for name in context["runtime"]["package_files"]) + paths.update(Path(path) for path in context["runtime"]["distribution_metadata"]) + project_root = Path(script_path).resolve().parents[1] + paths.update(project_root / name for name in context["scripts"]) + paths.update(Path(path) for path in (args.vllm_wheel, args.runtime_manifest) if path) + return paths + + +def publish(directory, payloads, provenance, manifest_name, external=None, *, protected_inputs=()): + """Publish validated results; legacy callers retain their overwrite contract.""" + directory = Path(directory).resolve() + provenance["outputs"] = {name: {"sha256": hashlib.sha256(data).hexdigest(), "size": len(data)} + for name, data in sorted(payloads.items())} + provenance["output_sha256"] = hashlib.sha256(json_bytes(provenance["outputs"])).hexdigest() + manifest = json_bytes(provenance) + targets = {directory / name: data for name, data in payloads.items()} + targets[directory / manifest_name] = manifest + if external: + path = Path(external).absolute() + # Preserve the final name so an alias to another manifest stays a + # distinct target. An explicit default name still publishes once. + path = path.parent.resolve() / path.name + require(path not in targets or path == directory / manifest_name, "provenance path overlaps an output") + targets[path] = manifest + legacy = provenance["regime"] == "legacy_distributional" + # stat follows symbolic links and identifies hardlinks to the same input. + protected = {file_identity(path) for path in protected_inputs if Path(path).exists()} + backups = {} + resolved_targets, output_identities = set(), set() + for target in targets: + resolved = target.resolve() + require(resolved not in resolved_targets, f"publication paths alias an output: {target}") + resolved_targets.add(resolved) + if target.exists(): + inode = file_identity(target) + require(inode not in output_identities, f"publication paths alias an output: {target}") + output_identities.add(inode) + require(not target.exists() or file_identity(target) not in protected, + f"publication path overlaps an input: {target}") + require(legacy or not target.exists(), f"refusing to overwrite {target}") + if target.exists(): + backups[target] = target.read_bytes() + created = [] + try: + for target, data in targets.items(): + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("wb" if legacy else "xb") as stream: + created.append(target) + stream.write(data) + except BaseException: + for target in created: + if target in backups: + target.write_bytes(backups[target]) + else: + target.unlink(missing_ok=True) + raise diff --git a/tests/scripts/test_qwen3_capture_outputs.py b/tests/scripts/test_qwen3_capture_outputs.py new file mode 100644 index 000000000..8b8010fa0 --- /dev/null +++ b/tests/scripts/test_qwen3_capture_outputs.py @@ -0,0 +1,632 @@ +#!/usr/bin/env python3 +"""Real NumPy output checks with a controlled fake vLLM, never a GPU oracle.""" + +import json +import os +from pathlib import Path +import shutil +import unittest +import zipfile + +import numpy as np + +from test_qwen3_capture_tools import CaptureFixture, sha + + +class CaptureOutputTests(CaptureFixture): + def capture(self, *extra, env=None): + result = self.run_script([*self.args(), "--per-prompt", *extra], env=env) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + return json.loads((self.out / "oracle-provenance.json").read_text()) + + def calls(self): + return [json.loads(line) for line in self.log.read_text().splitlines()] + + def test_ten_runs_publish_numpy_and_actual_provenance(self): + external = self.root / "evidence/capture.json" + for dtype in ("auto", "bfloat16", "fp8_e4m3"): + provenance = self.capture("--kv-cache-dtype", dtype, "--seed", "17", + "--provenance-out", str(external)) + ids = np.load(self.out / "greedy_ids.npy", allow_pickle=False) + dist = np.load(self.out / "greedy_dist.npy", allow_pickle=False) + self.assertEqual(ids.dtype, np.dtype(" with __commit_id__=None. + # This fixture proves metadata verification, never wheel gateability. + (self.source / ".git").rename(self.root / "saved-source-git") + package = self.source / "vllm/__init__.py" + original = package.read_text() + metadata = self.source / "vllm-0.28.1.dist-info/METADATA" + metadata.parent.mkdir() + for prefix, accepted in ((self.revision[:7], True), (self.revision[:9], True), + (self.revision, True), ("0" * 9, False)): + version = "0.28.1rc1.dev132+g" + prefix + package.write_text(original.replace("controlled-test-fixture", version)) + metadata.write_text(f"Name: vllm\nVersion: {version}\n") + with zipfile.ZipFile(self.wheel, "w") as archive: + archive.write(package, "vllm/__init__.py") + archive.write(metadata, "vllm-0.28.1.dist-info/METADATA") + runtime = json.loads(self.runtime_manifest.read_text()) + runtime["wheel_sha256"] = sha(self.wheel) + self.runtime_manifest.write_text(json.dumps(runtime)) + if accepted: + provenance = self.capture() + self.assertEqual(provenance["runtime"]["revision"], prefix) + self.assertEqual(provenance["runtime"]["revision_verification"], + "installed_version_vcs_prefix") + np.load(self.out / "greedy_ids.npy", allow_pickle=False).tofile(self.out / "our_ids.i32") + result = self.run_script(near=True) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + near = json.loads((self.out / "neartie-provenance.json").read_text()) + self.assertEqual(near["runtime"]["revision"], prefix) + for name in ("our_ids.npy", "neartie_gap_mnats.npy", "neartie-provenance.json"): + (self.out / name).unlink() + # The same verified bytes can be captured from clean source, + # then consumed from an installed wheel with a shorter prefix. + provenance["runtime"].update(revision=self.revision, + revision_verification="clean_git_source") + (self.out / "oracle-provenance.json").write_text(json.dumps(provenance)) + result = self.run_script(near=True) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + for name in ("our_ids.npy", "neartie_gap_mnats.npy", "neartie-provenance.json"): + (self.out / name).unlink() + provenance["runtime"].update(revision=self.revision[:8], + revision_verification="installed_version_vcs_prefix") + (self.out / "oracle-provenance.json").write_text(json.dumps(provenance)) + self.assert_publication_refused(self.args(True), near=True) + shutil.rmtree(self.out) + else: + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def prepare_neartie(self): + self.capture() + our = np.load(self.out / "greedy_ids.npy", allow_pickle=False) + our[0, 0] = 9999 + our.astype(" 16: + lp -= 0.0001 + d[token] = NS(logprob=lp) + probs.append(d) + return [NS(prompt_logprobs=probs)] + results = [] + for prompt in prompts: + index = int(os.environ["FAKE_PROMPTS"].split("||").index(prompt)) + ids = [700 + index] * sp.max_tokens + prompt_ids = [500 + index, 600] + if os.environ.get("FAKE_TOKEN_DRIFT") and self.calls > 144: + ids[-1] += 1 + if os.environ.get("FAKE_PROMPT_DRIFT") and self.calls > 144: + prompt_ids[0] += 1 + if os.environ.get("FAKE_SHORT"): + ids = ids[:-1] + results.append(NS(prompt=prompt, prompt_token_ids=prompt_ids, + outputs=[NS(token_ids=ids, text="fixture")])) + return results +''' + + +def sha(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def load_script(path): + name = path.stem.replace("-", "_") + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + with mock.patch.dict(sys.modules, {"numpy": ModuleType("numpy")}), mock.patch.object( + sys, "path", [str(path.parent), *sys.path] + ): + spec.loader.exec_module(module) + return module + + +class CaptureFixture(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory(prefix="qwen3-capture-") + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + self.project = self.root / "project" + (self.project / "scripts").mkdir(parents=True) + (self.project / "tests/parity").mkdir(parents=True) + for filename in (*SCRIPTS, "qwen3_oracle_common.py"): + source = ROOT / "scripts" / filename + if source.exists(): + shutil.copy2(source, self.project / "scripts" / filename) + shutil.copy2(ROOT / "tests/parity/test_qwen35_paged_engine.cpp", + self.project / "tests/parity/test_qwen35_paged_engine.cpp") + self.model = self.root / "model" + self.model.mkdir() + (self.model / "config.json").write_text(json.dumps({ + "model_type": "qwen3_5", "architectures": ["Qwen3_5ForConditionalGeneration"], + "text_config": {"model_type": "qwen3_5_text"}, + })) + (self.model / "model.safetensors").write_bytes(b"not real model weights") + self.write_metadata() + self.source = self.root / "source" + (self.source / "vllm").mkdir(parents=True) + (self.source / "vllm/__init__.py").write_text(FAKE_VLLM) + self.git("init", "-q") + self.git("add", "vllm") + self.git("-c", "user.name=Fixture", "-c", "user.email=fixture@invalid", + "commit", "-qm", "controlled fixture") + self.revision = self.git("rev-parse", "HEAD").strip() + self.wheel = self.root / "fixture.whl" + with zipfile.ZipFile(self.wheel, "w") as archive: + archive.write(self.source / "vllm/__init__.py", "vllm/__init__.py") + self.runtime_manifest = self.root / "runtime.json" + self.runtime_manifest.write_text(json.dumps({ + "vllm_revision": self.revision, "wheel_sha256": sha(self.wheel), + "image_digest": "sha256:" + "c" * 64, + })) + self.out = self.root / "golden" + self.log = self.root / "calls.jsonl" + prompts = load_script(self.project / "scripts" / SCRIPTS[0]).PROMPTS + self.env = dict(os.environ, PYTHONPATH=os.pathsep.join( + [str(self.source), os.environ.get("PYTHONPATH", "")]), FAKE_LOG=str(self.log), + FAKE_PROMPTS="||".join(prompts), PYTHONDONTWRITEBYTECODE="1") + + def git(self, *args): + return subprocess.check_output(["git", "-C", str(self.source), *args], text=True) + + def write_metadata(self): + for path in self.model.iterdir(): + if not path.is_file(): + continue + data = path.read_bytes() + etag = (hashlib.sha1(f"blob {len(data)}\0".encode() + data).hexdigest() + if path.suffix == ".json" else hashlib.sha256(data).hexdigest()) + metadata = self.model / ".cache/huggingface/download" / (path.name + ".metadata") + metadata.parent.mkdir(parents=True, exist_ok=True) + metadata.write_text(f"{MODEL_REV}\n{etag}\n0\n") + + def args(self, near=False): + return ["--model", str(self.model), "--model-revision", MODEL_REV, + "--vllm-revision", self.revision, "--vllm-wheel", str(self.wheel), + "--runtime-manifest", str(self.runtime_manifest), "--max-tokens", "2", + "--golden-dir" if near else "--out-dir", str(self.out)] + + def run_script(self, args=None, *, near=False, env=None): + return subprocess.run([sys.executable, str(self.project / "scripts" / SCRIPTS[near]), + *(self.args(near) if args is None else args)], + env=dict(self.env, **(env or {})), text=True, capture_output=True) + + def assert_refused(self, result, reason): + self.assertNotEqual(result.returncode, 0, result.stdout) + self.assertIn(reason, result.stderr, result.stdout + result.stderr) + self.assertFalse(self.out.exists() and any(self.out.iterdir())) + + +class CaptureContractTests(CaptureFixture): + def test_options_and_eager_alias(self): + for script in SCRIPTS: + module = load_script(self.project / "scripts" / script) + required = ["--model", "model", "--golden-dir", "unused"] if "neartie" in script else [] + for mode in ("production", "eager"): + for dtype in ("auto", "bfloat16", "fp8_e4m3"): + args = module._parse_args([*required, "--execution-mode", mode, + "--kv-cache-dtype", dtype, "--seed", "17"]) + kwargs = module._llm_kwargs(args) + self.assertEqual(kwargs["kv_cache_dtype"], dtype) + self.assertEqual(kwargs["seed"], 17) + self.assertIs(kwargs["enforce_eager"], mode == "eager") + args = module._parse_args([*required, "--enforce-eager"]) + self.assertEqual(args.execution_mode, "eager") + self.assertIn("eager diagnostic", module._mode_narration(args)) + with self.assertRaises(SystemExit): + module._parse_args([*required, "--enforce-eager", "--execution-mode", "production"]) + + def test_all_three_prompt_sources_are_checked(self): + for filename in ("scripts/" + SCRIPTS[0], "scripts/" + SCRIPTS[1], + "tests/parity/test_qwen35_paged_engine.cpp"): + path = self.project / filename + original = path.read_text() + path.write_text(original.replace("The capital of France is", "Changed prompt", 1)) + for near in (False, True): + self.assert_refused(self.run_script(near=near), "PROMPTS_MISMATCH") + self.assertFalse(self.log.exists(), "prompt mismatch must refuse before LLM construction") + path.write_text(original) + + def test_strict_identity_inputs_are_required(self): + for option in ("--model-revision", "--vllm-revision", "--vllm-wheel", "--runtime-manifest"): + argv = self.args() + index = argv.index(option) + del argv[index:index + 2] + self.assert_refused(self.run_script(argv), "ARTIFACT_MISMATCH") + + def test_model_revision_and_content_are_verified(self): + path = self.model / "config.json" + original = path.read_bytes() + path.write_bytes(original + b" ") + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + path.write_bytes(original) + meta = self.model / ".cache/huggingface/download/config.json.metadata" + meta.write_text(meta.read_text().replace(MODEL_REV, "b" * 40)) + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_weight_bytes_are_checked_against_download_identity(self): + (self.model / "model.safetensors").write_bytes(b"different weights") + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_dirty_source_is_refused_even_when_the_wheel_matches_it(self): + source = self.source / "vllm/__init__.py" + source.write_text(source.read_text() + "\n# dirty source rebuilt into wheel\n") + with zipfile.ZipFile(self.wheel, "w") as archive: + archive.write(source, "vllm/__init__.py") + runtime = json.loads(self.runtime_manifest.read_text()) + runtime["wheel_sha256"] = sha(self.wheel) + self.runtime_manifest.write_text(json.dumps(runtime)) + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_vllm_revision_and_dirty_source_are_refused(self): + argv = self.args() + argv[argv.index("--vllm-revision") + 1] = "b" * 40 + self.assert_refused(self.run_script(argv), "ARTIFACT_MISMATCH") + with (self.source / "vllm/__init__.py").open("a") as stream: + stream.write("\n# dirty\n") + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_runtime_manifest_and_wheel_are_verified(self): + original = self.runtime_manifest.read_text() + for field, value in (("wheel_sha256", "0" * 64), ("vllm_revision", "b" * 40), + ("image_digest", "mutable:latest")): + data = json.loads(original) + data[field] = value + self.runtime_manifest.write_text(json.dumps(data)) + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + self.runtime_manifest.write_text(original) + with zipfile.ZipFile(self.wheel, "w") as archive: + archive.writestr("vllm/__init__.py", FAKE_VLLM + "\n# changed wheel\n") + data = json.loads(original) + data["wheel_sha256"] = sha(self.wheel) + self.runtime_manifest.write_text(json.dumps(data)) + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_strict_runtime_needs_an_observed_source_or_installed_vcs_revision(self): + (self.source / ".git").rename(self.root / "saved-source-git") + self.assert_refused(self.run_script(), "observed source or installed VCS revision") + + def test_imported_and_installed_runtime_versions_must_agree(self): + metadata = self.source / "vllm-0.28.1.dist-info/METADATA" + metadata.parent.mkdir() + metadata.write_text("Name: vllm\nVersion: different-installed-version\n") + self.assert_refused(self.run_script(), "imported and installed vLLM versions differ") + + def test_untracked_package_bytes_and_missing_artifact_metadata_are_refused(self): + extra = self.source / "vllm/untracked.py" + extra.write_text("untracked = True\n") + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + extra.unlink() + (self.model / ".cache/huggingface/download/model.safetensors.metadata").unlink() + self.assert_refused(self.run_script(), "ARTIFACT_MISMATCH") + + def test_runtime_identity_cannot_promote_unverified_legacy_inputs(self): + (self.model / "config.json").write_text(json.dumps({"model_type": "qwen3"})) + self.write_metadata() + self.assert_refused(self.run_script(["--model", str(self.model), "--out-dir", + str(self.out), "--runs", "1"], + env={"FAKE_RUNTIME_QWEN35": "1"}), + "ARTIFACT_MISMATCH") + + def test_strict_short_repetition_count_is_refused(self): + self.assert_refused(self.run_script([*self.args(), "--runs", "9"]), "ten") + self.assertFalse(self.log.exists(), "short strict capture must refuse before LLM construction") + + def test_each_qwen35_identity_indicator_prevents_legacy_downgrade(self): + for config in ({"model_type": "qwen3_5"}, + {"model_type": "qwen3", "architectures": ["Qwen3_5ForCausalLM"]}, + {"model_type": "qwen3", "text_config": {"model_type": "qwen3_5_text"}}): + (self.model / "config.json").write_text(json.dumps(config)) + self.write_metadata() + self.assert_refused(self.run_script(["--model", str(self.model), "--out-dir", + str(self.out), "--runs", "1"]), + "ARTIFACT_MISMATCH") + + def test_options_reject_invalid_counts_and_cache_modes(self): + for script in SCRIPTS: + module = load_script(self.project / "scripts" / script) + required = ["--model", "model", "--golden-dir", "unused"] if "neartie" in script else [] + for option, value in (("--runs", "0"), ("--repetitions", "-1"), + ("--max-tokens", "0"), ("--kv-cache-dtype", "float32")): + with self.assertRaises(SystemExit): + module._parse_args([*required, option, value]) + + +class CaptureRegistrationTests(unittest.TestCase): + def test_preflight_runs_both_suites_or_reports_numpy_pending(self): + source = (ROOT / "scripts/agent-preflight.sh").read_text() + match = re.search(r"# QWEN3-CAPTURE-TOOLS: begin\n(.*?)# QWEN3-CAPTURE-TOOLS: end", source, re.S) + self.assertIsNotNone(match, "capture suites have no preflight registration") + prelude = '''run() { printf 'RUN %s\\n' "$1"; shift; "$@"; } +skip() { printf 'SKIP %s %s\\n' "$1" "$*"; } +python3() { + if test "$1" = -c; then test "$CAPTURE_HAVE_NUMPY" = 1; + else printf 'PYTHON %s\\n' "$*"; fi +} +''' + for available in ("0", "1"): + result = subprocess.run(["bash", "-c", prelude + match.group(1)], text=True, + capture_output=True, env=dict(os.environ, CAPTURE_HAVE_NUMPY=available)) + self.assertEqual(result.returncode, 0, result.stderr) + for name in ("test_qwen3_capture_tools", "test_qwen3_capture_outputs"): + self.assertIn(("RUN " if available == "1" else "SKIP ") + name, result.stdout) + if available == "1": + self.assertIn("PYTHON tests/scripts/" + name + ".py", result.stdout) + if available == "0": + self.assertIn("PENDING", result.stdout) + self.assertNotIn("RUN ", result.stdout) + + def test_ci_runs_both_suites_after_its_existing_numpy_install(self): + source = (ROOT / ".github/workflows/ci.yml").read_text() + for name in ("test_qwen3_capture_tools", "test_qwen3_capture_outputs"): + command = " python3 tests/scripts/" + name + ".py" + self.assertIn(command, source) + prefix = source[:source.index(command)] + block = prefix[prefix.rfind(" run: |") :] + self.assertIn("apt-get install -y --no-install-recommends python3-numpy", block) + + +if __name__ == "__main__": + unittest.main(verbosity=2)