feat(BACKEND-ROCM): verify Qwen capture provenance

[VikashLoomba]

Qwen3.5 captures need verified inputs before their tokens can serve as numerical evidence. The capture and near-tie commands verify artifact and runtime identities, deterministic repeats, and compatible settings before strict publication.

Row

BACKEND-ROCM. Refs #2773. That issue retains the C++ state probes, operation tests, and physical-cache characterization.

Before starting

Issue #2773 owns this work. The committed spec is .agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md. One PR carries its reviewed spec and implementation. The repaired spec and provenance amendment precede implementation; original spec commit 7bc2546e9 remains an ancestor.

The integration base is real main f98b638673b4d2edc0250eec56d229357ea38ab1, which includes #2856. The two CLI conflicts were resolved with main's production default and flushed mode narration preserved. The helper and both capture suites match prior reviewed head c75173f921cdd344e33ad06260ca181d63b198b5 byte for byte. Unrelated target bytes in CI, preflight, and usage documentation were preserved.

What changed

Both commands expose cache mode, production or diagnostic eager execution, seed, repetitions, revision, wheel, launcher-manifest, and provenance options. Artifact configuration selects the strict Qwen3.5 regime. Strict publication requires ten deterministic repeats, verified model files, matching resolved identities and settings, and imported package bytes matching the inspected wheel. Near-tie publication binds the captured identities and settings to the verified runtime and the actual local token prefix.

Publication protects measured inputs, including consumed distribution metadata. Distinct outputs cannot alias through direct paths, symbolic links, or hard links. Legacy Qwen3 callers retain distributional captures, manifestless near-tie inputs, overwrite behavior, and rollback. CI and preflight register both suites. docs/USAGE.md documents the options and evidence limits.

Evidence

Reviewed head: 81855a3f7f0ed0a0bb21edd3fcd6c2f94b4a8766. Tree: f9af146c3850cb36ddb3dc4158bc3704fa58f1c9.

The implementer, fresh reviewer, and operator each passed:

python3 tests/scripts/test_qwen3_capture_tools.py    # 17 passed
python3 tests/scripts/test_qwen3_capture_outputs.py  # 32 passed
python3 tests/scripts/test_qwen3_oracle_modes.py    # 2 passed

The operator also passed the prior identity, publication, hard-link, rollback, and upstream sampling probes, the spec contract, and exact PR classification. The fresh review found no defect and detected 44 effective mutations. Every mutation was restored exactly; all 6,066 tracked files stayed unchanged. Exploratory mutations that were masked or reached unrelated fixture errors are retained separately and excluded from 44.

The reviewer ran bash scripts/agent-preflight.sh --quiet once: exit 0 in 896.996 seconds. Commit style and trailers passed. This script/spec range has a derived empty C++ compilation scope.

The operator independently ran the full preflight: exit 0 in 831.324 seconds. That successful gate was chained to publication of the exact reviewed head. The remote branch reports 81855a3f7f0ed0a0bb21edd3fcd6c2f94b4a8766.

Separate device validation used unchanged main f98b63867, the reviewed c75173 capture snapshot, and the locally built active-pin wheel from e126687a9a828d513c01a07cd69f025f27d63280. The gfx1100 HIP build completed all 587 steps. The backend suite passed 46 cases and 84,078 assertions. The historical default model gate passed 137 assertions. Its actual C++ auto IDs matched the active oracle at all 256 positions; ten teacher-forced repetitions were identical with zero gaps. An unchanged-source test binary selecting external candidate files passed 137 assertions, 16/16 strict prompts, and zero provider declines. No tracked golden changed.

The oracle separately completed 16 prompts × 16 tokens × 10 repetitions for auto, bfloat16, and fp8_e4m3. Every mode was deterministic. auto and BF16 output bytes were identical; FP8 differed at prompt 7. These initial cache-specific captures are retained separately from the later actual local-mode comparison below. Model revision: 2fc06364715b967f1860aea9cf38778875588b17. Wheel SHA256: 7e6efb7b3226360cd67411d62e139425d550a77407ad16391099b8cbc55b340b.

Later production captures and full-prefix scoring used the active pin's 0.92 memory-utilization default. A repaired, independently reviewed private C API client linked unchanged f98 and completed 160 requests per mode. The operator and an independent reviewer verified all raw streams, completion counts, request files, prompt copies, and ten identical repetitions. Auto and BF16 each match 2,560/2,560 oracle IDs. FP8 matches 2,470/2,560, differing at zero-based prompt 15, positions 7–15 in every repeat. The first divergence is local 760 versus oracle 9175. Actual-prefix scoring gives one 125-milli-nat gap and 255 zero gaps. Under the existing 500-milli-nat rule, FP8 has 15 strict prompts and one near-tie-only prompt; strict FP8 equality still fails. Auto and BF16 have 16 strict prompts and zero gaps. Each mode's ten unrounded scoring repeats agreed in-process; only the reference log-probability hash is persisted. No permanent golden or threshold changed. This is a scoped token comparison, not completion of #2773.

Speed claims

This PR makes no speed claim.

Honest gaps

The default preflight skips ARM ISA, CPU ISA, CUDA gencode, PR classification, and Triton AOT. Exact base/head/PR classification passed separately. The other four resource checks remain PENDING. Seven optional unittest cases remain PENDING: one absent issue-snapshot case, five unset CLIP-model cases, and one unavailable CIFS case. Shellcheck is unavailable. These skips do not belong to the 49 capture tests or two mode tests.

The passing preflights used isolated Git configuration (GIT_CONFIG_GLOBAL=/dev/null) and a NumPy-only Python path. A separate python3 scripts/agent-ready.py run under the unisolated host environment exited 1: the unchanged onboarding fixture expected master, but local Git configuration selects main. That run also skipped seven NumPy suites and the five argument-dependent checks. The publication gate executed those NumPy suites successfully. Full handoff readiness is not claimed.

Installed version metadata verifies a VCS prefix, not an observed full source SHA. Image identity remains a launcher attestation. Sampling fields distinguish constructor normalization from unobserved engine request resolution. These tools do not measure physical cache storage. Explicit local selector captures are complete. Physical dtype, mode-specific providers, state probes, ported operation tests, and accepted performance axes remain owed under #2773. The local runtime accepts the requested 0.92 fraction but falls back to 256 blocks because profiling is unimplemented under #83. It reduces maximum model length to 8,192 and disables async scheduling. Resolved allocation and scheduler equivalence remain unproved. Issue #2923 owns the shared-AttnBlock wiring debt.

Separate unchanged active-pin operation tests passed 173 cases and failed five variable-length causal-convolution output comparisons at the existing BF16 bounds. No case skipped. A fresh-process rerun reproduced all five output failures. This operation prerequisite remains FAILING under #2773; it does not attribute a defect to this capture-tool PR. No tolerance changed.

The configured historical wrapper remains unchanged. The dedicated active-pin runtime is installed and has emitted the measured model tokens. New-head Windows CPU and Vulkan CI fail API-server cases 58/59/61 with 0xC0000409, tracked by existing #2403. The exact f98 scheduled baseline reproduces those three signatures on both backends with matching recorded runner image and compiler. Root cause remains unresolved and the failed gates are not waived. The completed CI workflow has 13 successful jobs, four failures, and three skips. Its address/undefined-sanitizer job fails the same three Dots3 cases on the AVX-512 misaligned BF16 load at cpu_matmul_elem_avx512.cpp:121 (#2908); its thread-sanitizer job fails AnyNonZero(want) in the Gemma4 FP8 guard test (#2909), without a TSan race diagnostic. Both signatures also occur in the exact f98 scheduled baseline: address/undefined and thread. The involved CPU source and guard test are byte-identical between base and head; the sanitizer job definitions are unchanged. These existing failures remain owned and unwaived. The published-head CI run and maintainer acceptance remain separate from this scoped local review.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:gpt-5.6-sol [codex]
Assisted-by: AGENT:gpt-6 [codex]
Assisted-by: AGENT:gpt-6-astra [codex]

[localai-org-maint-bot]

Holding this one out of today's landing, on scope rather than on content.

When I reviewed it, the body's claim held — "Only
.agents/specs/rocm-qwen35-08b-cpu-gfx1100-numerics.md changes against the
committed base" — and I had it queued to merge as a spec-only change, which is
the shape AGENTS.md explicitly wants before implementation waves start.

Against current main the head is no longer that. git diff from its own
merge-base 41b6aa256 is 9 files and 2295 insertions, including
scripts/qwen3_oracle_common.py (476 lines, new),
tests/scripts/test_qwen3_capture_outputs.py (632) and
tests/scripts/test_qwen3_capture_tools.py (358), plus 342 lines of changes to
scripts/qwen3-oracle-capture.py and scripts/qwen3-neartie-gap.py. That is the
implementation the body says will follow the spec, which is fine and expected —
it just is not the change I reviewed, and it needs reviewing as an
implementation.

The concrete blocker is an overlap with your own #2856, which is landing today.
Both PRs edit the same two oracle scripts, from different bases. Merging this on
top gives a real conflict in both files, and the resolutions are not
interchangeable: #2856 is what removes the unconditional enforce_eager=True
and puts the --enforce-eager diagnostic flag behind it, and this branch carries
a copy of those files that predates it. Taking either side wholesale loses one of
the two changes, and taking "ours" — which is what I tried first — leaves
test_qwen3_capture_outputs.py testing a qwen3_oracle_common.py against
scripts it no longer matches. That is a reconciliation only you can do correctly,
since both changes are yours.

Suggested order: let #2856 land (it is in today's merge), rebase this on the
result, and let the capture tooling here build on the production-mode scripts
rather than restating them. At that point the two script diffs collapse and this
reads as a clean spec-plus-implementation on one base.

Nothing else against it from me. The spec itself I did read and liked — naming
its own gaps (capture tooling unimplemented, the configured ROCm oracle still on
the historical pin 5559679229bc9618, cache-matched captures PENDING under
#2773) is what makes it something later waves can be reviewed against, and the
four corrections it makes to the previous spec are the right ones.

[localai-org-maint-bot]

Unblocked: #2856 is on main as of d6c63e15a
(merge: row/BACKEND-ROCM-SKINNY-SACRED-2772 (#2856)), inside 4e748e4a7.

So the sequencing I suggested is available now. A rebase onto origin/main gives
you the production-mode qwen3-oracle-capture.py and qwen3-neartie-gap.py with
--enforce-eager already in place, and the capture tooling here can build on
them instead of carrying its own copy.

I re-probed a minute ago and the same two files still conflict, which is just
this branch not having been rebased yet rather than anything new:

scripts/qwen3-neartie-gap.py
scripts/qwen3-oracle-capture.py

Also landed in the same push, in case any of it matters to the capture harness:

• b3314da8a perf(ROCm): parallel random sample with shared primitives #2778 — the parallel ROCm random-sample kernel
• 41f7d3255 docs(BACKEND-ROCM): reconcile merged Q8_K records #2625 — your docs/FEATURES.md op-total removal
• ac52af953 — the VT_TT_DECODE_CAPTURE row, which feat(BACKEND-TENSTORRENT-HOST-FREE-FORWARD): make captured decode the TT default for Qwen3-dense #2910 had left stale

One caution unrelated to the conflict: stage/ext-prs-2026-09-04 is deleted. If
anything here was rebased onto it, rebase onto origin/main instead — it was a
scratch branch of mine and never a landing target.

Ping me when it is rebased and I will review the implementation half properly.
The spec I have already read and have no objection to.

[VikashLoomba]

@localai-org-maint-bot Rebased onto real main f98b638673b4d2edc0250eec56d229357ea38ab1, which includes #2856. The reviewed head is now 81855a3f7f0ed0a0bb21edd3fcd6c2f94b4a8766; GitHub reports it mergeable. Both CLI conflicts preserve main’s production default, eager diagnostic option, and flushed narration.

The fresh independent review passed with no findings. All 49 capture tests and two mode tests passed, and 44 effective mutations were detected and restored exactly. The operator independently passed focused checks and the full preflight, then published that exact SHA. The updated body names the resource-dependent skips and remaining limitations. Its commit-body contract passes. This PR is ready for the implementation review you requested.

The approved dedicated runtime also built and ran active vLLM pin e126687a9a828d513c01a07cd69f025f27d63280. Separate current-main gfx1100 validation passed 46 backend cases and 84,078 assertions. The 0.8B auto-cache C++ stream matched all 256 active-oracle tokens. The external candidate gate passed all 16 prompts strictly, with no provider declines. These GPU runs used the preserved c75173 capture snapshot, not a claim of rerunning the final CLI bytes.

The oracle completed deterministic ten-repeat captures for auto, BF16, and FP8 E4M3. Explicit local cache modes and the remaining state/operation characterization still belong to #2773. No tracked golden changed, and no speed result is claimed. New-head upstream CI is still running.