Skip to content

fix(mq4v2): one MQ-V2 prefill admit rule for llama and qwen35; discriminating GEMM parity; spec §9 - #690

Open
Kaden-Schutt wants to merge 5 commits into
masterfrom
fix/mq4v2-followups
Open

fix(mq4v2): one MQ-V2 prefill admit rule for llama and qwen35; discriminating GEMM parity; spec §9#690
Kaden-Schutt wants to merge 5 commits into
masterfrom
fix/mq4v2-followups

Conversation

@Kaden-Schutt

@Kaden-Schutt Kaden-Schutt commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Follow-ups from the MQ4G256V2 kernel-family audit (docs/plans/audit-2026-09-02/audit-Mq4v2Kernels.md, PR #685):

  1. Lockstep (audit Broken 1). hipfire_runtime::llama::is_batchable_la admitted MQ-V2 (qt 44/47–50) WMMA prefill only on gfx12; qwen35::is_batchable_la admitted gfx11 + gfx12 with the HIPFIRE_MQV2_GFX11_WMMA kill-switch. Both doc-comments claimed they "match exactly". The rule now lives once in llama::mqv2_wmma_batchable / llama::mqv2_gfx11_wmma_enabled_from_env; both gates delegate; a test asserts agreement over 6 dtypes × {gfx1100, gfx1151, gfx1201, gfx1030, gfx1010}. MQ4CG256 (qt=45) stays gfx12-only in both, per the spec. Behaviour change (revised in 4e360fb2a): plain Llama/Qwen3 dense models on MQ-V2 stay per-token on every archllama.rs::forward_prefill_chunk has no V2 arms (matchers at ~2570/3025/3117/3248), so admitting them would run V1 launchers on V2 blobs; the hw-gate Fable seat caught this by source trace (run 33895641944). The shared rule still admits qwen35 on gfx11+gfx12 (its chunk path has the V2 arms). master's old gfx12-only arm had the same hole for llama; this PR closes it instead of widening it.
  2. Discriminating GEMM parity (audit Missing 1). mq4v2_gemm_parity used Gaussian weights whose two halves have near-identical headers, so a wrong half-select hides inside quantization noise — contradicting its own header comment. It gains a disjoint-halves arm (half 0 in [-1,1], half 1 in [96,160], same batch sweep) with a negative control (swapped-header reference must disagree) and a CPU test that the fixture discriminates.
  3. Docs/comments: spec §9 (MoE is production-wired for qt=44; XBATCH is ported; gfx11 bodies step kt += 2), gemm_mq4g256v2_residual_mmq.hip "Experimental" → production fast path (gfx1100/1151, batch ≥128), gemm_qkv_mq4g256v2_wmma.gfx12.hip "HYPOTHESIS" → the validated C-map statement its sibling carries.

Which crate(s) does this touch?

  • crates/hipfire-runtime, crates/hipfire-arch-qwen35, kernels/ (comments only), docs

Evidence

  • mq4v2_residual_parityPASS on gfx1100 (hipx) and gfx1201 (hiptrx): all shapes, swapped-headers baseline at 0.9996 rel error.
  • mq4v2_gemm_parityPASS on gfx1201 (the example calls the *_gfx12 launchers directly; gfx12-only by construction): disjoint arm v2 rel-rms 2.5e-4 vs 1.263 with swapped headers; gate_up/qkvza v1/v2 arms unchanged.
  • The gfx11 WMMA prefill TUs the lockstep change admits for Llama/Qwen3 are the same ones Qwen3.5/3.8 already run in production on gfx1100/1151 (hw-gate lanes). Not run: a model-level prefill on an arch-0/1 MQ-V2 artifact on gfx11 — none exists on hipx/hiptrx (qwen3-8b.hf4 is qt=6). If a reviewer wants that before merge, it needs a --format mq4 quantization of qwen3-8b first.

Test plan

  • cargo test -p hipfire-runtime -p hipfire-arch-qwen35 — 597 + 189 pass, incl. mqv2_admit_llama_qwen35_lockstep and disjoint_fixture_discriminates_half_select
  • Parity examples above

Kaden-Schutt and others added 4 commits September 4, 2026 16:02
llama::is_batchable_la admitted MQ4G256V2/MQ6/5/3/2G256V2 for WMMA
prefill only on gfx1200/gfx1201 while qwen35::is_batchable_la admitted
them on gfx11+gfx12 behind HIPFIRE_MQV2_GFX11_WMMA, so plain Llama/Qwen3
dense qt=44 models prefetched per-token on gfx1100/1151 while Qwen3.5/3.8
took WMMA — despite both doc-comments claiming an exact match (audit
2026-09-02 Broken 1).

Move the dtype set + arch set + kill-switch helper into
llama::mqv2_wmma_batchable / llama::mqv2_gfx11_wmma_enabled_from_env in
hipfire-runtime and delegate from both callers, so the lockstep is
structural. MQ4CG256 (qt=45) stays gfx12-only in both by intent.

Tests: rename the two gfx12-only llama admit tests to gfx11+gfx12
expectations, repoint qwen35 env-escape test at the shared helper, and
add mqv2_admit_llama_qwen35_lockstep asserting both gates agree over
the MQ-V2 dtypes x {gfx1100, gfx1151, gfx1201, gfx1030, gfx1010}.
…ections

mq4v2_gemm_parity's Gaussian weights give both halves near-identical
headers, so a wrong half-select hides in quantization noise despite the
header comment promising a systematic blow-up (audit 2026-09-02
Missing 1). Add arm 2 using the disjoint-halves construction from
mq4v2_residual_parity (half0 [-1,1], half1 [96,160]) over the same
batch-size sweep: v2 output must match the f32 reference within 5%
rel-RMS, and the swapped-headers negative control (as in
mq4v2_moe_parity) must DISAGREE. Keep the Gaussian v1-vs-v2 arm and fix
the header comment to state what each arm can and cannot detect. Add a
host-side test proving the fixture discriminates with no GPU.

Docs (audit Would-change 1-2): spec section 9 now records MoE as
production-wired for qt=44, the XBATCH single-row path as ported, and
the gfx11 kt+=2 / residual kt++ stepping; residual_mmq.hip loses its
stale Experimental tag; the gfx12 QKV kernel loses its
HYPOTHESIS/scaffold wording for the validated C-map statement.
@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate sol prelim

summary: Centralizes the MQ-V2 dtype/architecture/kill-switch predicate used by Qwen3.5-family batched prefill, while deliberately changing plain Llama/Qwen3 MQ-V2 on gfx12 from batched prefill to per-token fallback because its chunk path lacks V2 dispatch arms. It also adds a disjoint-half MQ4V2 GEMM parity fixture with a swapped-header negative control and updates kernel/spec comments; the changed HIP files contain comments only.

run_hardware: true
run_hardware_reasons: The executable changes are ordinary in-process Rust admission logic and a GPU parity example; they do not add dependencies, process spawning, network access, credential access, build-system changes, or unexplained unsafe code.; The HIP source changes are comments only, but the Rust admission change determines whether existing MQ-V2 kernels receive real model blobs, so hardware execution is necessary.; The qwen3.8:27b-mq4-xt fixture is available and is the closest artifact-specific route for the changed MQ-V2 Qwen3.5-family prefill/load behavior.

routes:

mode tag source why
battery qwen3.6:27b bucket bucket kernel,load
battery ornith-1.5:35b-a3b-mq4r bucket bucket kernel,load
battery lfm2.5:1.2b bucket bucket kernel,load
battery qwen3.8:27b-mq4-xt bucket bucket kernel,load
chain qwen3.8:27b-mq4-xt bucket Load bucket: loads the affected MQ4-XT artifact and exercises repeated related turns through the admitted prefill path, exposing load/type-routing or subsequent-turn corruption.

unavailable_routes:

(none)

claim_assessment: The author claims Qwen3.5-family MQ-V2 admission is correct on gfx11/gfx12, plain-Llama V2 must fall back everywhere, and the GEMM fixture now detects half-header selection errors. Coherent qwen3.8 MQ4-XT battery/chain output can evidence the available real-artifact load and gfx12 Qwen3.5 path. The plain-Llama fallback claim is not fully provable without a plain-Llama qt44/47-50 fixture, and the discriminating kernel claim requires the actual mq4v2 GEMM parity output rather than unit-test counts or PR prose.

questions_for_author:

  • Please provide or identify a plain Llama/Qwen3 qt44 or qt47-50 fixture for base-versus-PR gfx1201 prefill correctness and performance; none is listed on this runner.
  • Ensure the hardware artifact includes the full mq4v2_gemm_parity output for every disjoint-half batch and its swapped-header negative control, since ordinary generation routes do not prove that fixture or direct kernel parity.

@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate evidence — 2 lane(s) — verdict pass

lane hiptrx (gfx1201)

hw-gate evidence

field value
base 931b571c978b976fc6135ab8cd0e92b2c150a5bc
head 4e360fb2a8308e7522c3ea4e28b251b5b35da253
buckets kernel,load
host gfx gfx1201
host rocm 7.15.26333-0000000
device 3
runner hiptrx
daemon_md5 c8ee12f8886af75d6441de0404520763
hipfire_md5 bbd6de3fed8391a6c6836e592f4b070e
build_seconds 40.55622220039368
verdict pass
logs_dir hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 33.4 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 132 62 3.1 15.0 False False False True
battery 1 stop 30 0 87 76 503.7 36.1 False False False True
battery 2 stop 31 0 16 13 519.8 36.2 False False False True
battery 3 stop 47 0 31 8 560.6 36.1 False False False True
battery 4 stop 47 0 82 46 561.5 36.0 False False False True
qwen3.6:27b battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.6:27b battery turn 1
During the day, sunlight passes through a shorter thickness of Earth's atmosphere, causing blue light to scatter more widely than other colors due to its shorter wavelength. At sunset, sunlight travels through a much greater depth of the atmosphere, which scatters most of the blue and violet light out of the line of sight. This leaves the longer wavelengths of red and orange light to dominate the sky, creating the characteristic warm hues seen at dusk.
qwen3.6:27b battery turn 2
The capital of France is Paris, and the River Seine runs through it.
qwen3.6:27b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.6:27b battery turn 4
17 + 26 can be broken down as (10 + 7) + (20 + 6). Adding the tens gives 10 + 20 = 30. Adding the ones gives 7 + 6 = 13. Finally, adding these sums together gives 30 + 13 = 43.

Answer: 43

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 32.7 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 146 68 23.1 16.4 False False False True
battery 1 stop 30 0 94 84 448.0 52.3 False False False True
battery 2 stop 31 0 16 13 471.6 53.0 False False False True
battery 3 stop 47 0 31 8 661.1 59.7 False False False True
battery 4 stop 47 0 17 7 657.1 50.6 False False False True
ornith-1.5:35b-a3b-mq4r battery turn 0
```python
def longest_substring_without_repeating_chars(s):
    if not s:
        return ""
    
    start = 0
    max_start = 0
    max_len = 0
    char_index = {}
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_len:
            max_len = end - start + 1
            max_start = end - max_len + 1
    
    return s[max_start:max_start + max_len]
```
ornith-1.5:35b-a3b-mq4r battery turn 1
The sky appears blue during the day because sunlight interacts with the atmosphere through a process called Rayleigh scattering, where shorter blue wavelengths scatter more easily off air molecules than longer wavelengths. At sunset, sunlight travels through a greater thickness of atmosphere to reach your eyes, causing most of the blue light to be scattered away before it arrives. This leaves primarily the longer red and orange wavelengths to reach you directly, which is why the sky and sun appear reddish during this time.
ornith-1.5:35b-a3b-mq4r battery turn 2
The capital of France is Paris, and the Seine River runs through it.
ornith-1.5:35b-a3b-mq4r battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
ornith-1.5:35b-a3b-mq4r battery turn 4
17 + 26 = 43

Answer: 43

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 8.1 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 0 0 129 58 None 207.4 False False False True
battery 1 stop 0 0 73 65 None 208.57 False False False True
battery 2 stop 0 0 15 13 None 208.33 False False False True
battery 3 stop 0 0 26 8 None 208.0 False False False True
battery 4 stop 0 0 20 11 None 208.33 False False False True
lfm2.5:1.2b battery turn 0
```python
def longest_unique_substring(s):
    char_map = {}
    start = 0
    max_length = 0
    max_substring = ""

    for i, char in enumerate(s):
        if char in char_map and char_map[char] >= start:
            start = char_map[char] + 1
        else:
            if i - start + 1 > max_length:
                max_length = i - start + 1
                max_substring = s[start:i+1]

        char_map[char] = i

    return max_substring
```
lfm2.5:1.2b battery turn 1
The sky appears blue during the day due to the scattering of sunlight by atmospheric particles in a phenomenon called Rayleigh scattering, which spreads shorter blue wavelengths more across the sky. At sunset, the sunlight travels through more atmosphere, scattering the longer red wavelengths and making them dominate the color we see. Thus, both effects are responsible for the distinct color changes in the sky.
lfm2.5:1.2b battery turn 2
The capital of France is Paris, and the River Seine runs through it.
lfm2.5:1.2b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
lfm2.5:1.2b battery turn 4
Let's calculate the sum: 17 + 26 equals 43.  
Answer: 43

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 11.7 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 147 67 63.1 46.9 False False False True
battery 1 stop 30 0 85 75 346.1 52.2 False False False True
battery 2 stop 31 0 16 13 363.8 62.7 False False False True
battery 3 stop 47 0 22 6 441.3 49.7 False False False True
battery 4 stop 47 0 17 7 442.6 67.5 False False False True
qwen3.8:27b-mq4-xt battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    if not s:
        return ""
    
    char_index = {}
    start = 0
    longest_start = 0
    longest_length = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > longest_length:
            longest_length = end - start + 1
            longest_start = start
    
    return s[longest_start:longest_start + longest_length]
```
qwen3.8:27b-mq4-xt battery turn 1
During the day, sunlight passes through a thicker column of atmosphere, causing shorter blue wavelengths to scatter across the entire sky in a phenomenon known as Rayleigh scattering. At sunset, sunlight must travel through a much longer path in the atmosphere, which filters out most of the blue and green light before it reaches your eyes. This leaves the longer red and orange wavelengths to dominate, creating the characteristic warm hues of the evening sky.
qwen3.8:27b-mq4-xt battery turn 2
The capital of France is Paris, and the River Seine runs through it.
qwen3.8:27b-mq4-xt battery turn 3
{"name": "Alice", "age": 34, "city": "Lisbon"}
qwen3.8:27b-mq4-xt battery turn 4
17 + 26 = 43

Answer: 43

chain — exit 0 seconds 21.1 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 44 0 120 51 367.4 63.8 False False False True
chain 1 stop 224 0 230 121 101.7 63.4 False False False True
chain 2 stop 484 0 73 59 755.9 36.4 False False False True
chain 3 stop 595 0 107 85 723.3 44.3 False False False True
chain 4 stop 738 0 74 55 792.9 54.6 False False False True
qwen3.8:27b-mq4-xt chain turn 0
```python
def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list."""
    i, j = 0, 0
    result = []
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1
    # Append remaining elements
    result.extend(a[i:])
    result.extend(b[j:])
    return result
```
qwen3.8:27b-mq4-xt chain turn 1
To find the total distance traveled, we calculate the distance for each segment of the trip separately and then add them together. The formula for distance is:

$$\text{Distance} = \text{Speed} \times \text{Time}$$

**Step 1: Calculate the distance of the first segment**
*   Speed = 60 mph
*   Time = 2.5 hours
*   Distance$_1$ = $60 \times 2.5 = 150$ miles

**Step 2: Calculate the distance of the second segment**
*   Speed = 40 mph
*   Time = 1.5 hours
*   Distance$_2$ = $40 \times 1.5 = 60$ miles

**Step 3: Add the two distances**
*   Total Distance = Distance$_1$ + Distance$_2$
*   Total Distance = $150 + 60 = 210$ miles

**Final Answer:**
The train traveled a total of **210 miles**.
qwen3.8:27b-mq4-xt chain turn 2
The primary cause of seasons is the Earth's axial tilt of approximately 23.5 degrees relative to its orbital plane. As the Earth orbits the Sun, this tilt causes different hemispheres to receive varying intensities and durations of sunlight throughout the year. This variation in solar radiation leads to the cyclical patterns of temperature and weather that define the seasons.
qwen3.8:27b-mq4-xt chain turn 3
Elias pulled a tangled mass of kelp from the surf, only to find a brass pocket watch clutched in the seaweed. The face was intact, the hands still ticking despite decades underwater, and engraved on the back was his own name, dated 1924. His hands trembled as he realized he had died fifty years ago, yet here was proof he had not. He pressed the cold metal to his chest, feeling the familiar, terrifying rhythm of a life that had been granted a second chance.
qwen3.8:27b-mq4-xt chain turn 4
1. Write descriptive names for variables, functions, and classes.
2. Keep functions short and focused on a single responsibility.
3. Add clear comments to explain why code exists, not just what it does.
4. Use consistent coding standards and format across the entire project.
5. Write comprehensive unit tests to verify behavior and catch regressions.

kernel

status: pass

report pass: True

lane hipx (gfx1100)

hw-gate evidence

field value
base 931b571c978b976fc6135ab8cd0e92b2c150a5bc
head 4e360fb2a8308e7522c3ea4e28b251b5b35da253
buckets kernel,load
host gfx gfx1100
host rocm 7.15.26333-0000000
device 0
runner hipx
daemon_md5 c8ee12f8886af75d6441de0404520763
hipfire_md5 bbd6de3fed8391a6c6836e592f4b070e
build_seconds 45.295546770095825
verdict pass
logs_dir hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 32.7 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 144 67 3.2 18.5 False False False True
battery 1 stop 30 0 81 70 428.6 50.3 False False False True
battery 2 stop 31 0 19 16 447.7 50.5 False False False True
battery 3 stop 47 0 31 8 458.5 50.4 False False False True
battery 4 stop 47 0 6 2 465.6 50.4 False False False True
qwen3.6:27b battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    if not s:
        return ""
    
    start = 0
    max_start = 0
    max_length = 0
    char_index = {}
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.6:27b battery turn 1
During the day, sunlight passes through a shorter path in the atmosphere, causing shorter blue wavelengths to scatter more effectively and dominate our view. At sunset, light travels through a much thicker layer of air, which scatters away most of the blue and green light before it reaches your eyes. Consequently, only the longer red and orange wavelengths penetrate through the dense atmosphere, creating the characteristic warm hues of twilight.
qwen3.6:27b battery turn 2
The capital of France is Paris, and the river that runs through it is the Seine.
qwen3.6:27b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.6:27b battery turn 4
Answer: 43

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 33.7 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 134 60 442.6 20.9 False False False True
battery 1 stop 30 0 93 81 450.5 88.1 False False False True
battery 2 stop 31 0 16 13 446.6 106.1 False False False True
battery 3 stop 47 0 31 8 630.6 92.5 False False False True
battery 4 stop 47 0 17 7 621.3 75.3 False False False True
ornith-1.5:35b-a3b-mq4r battery turn 0
```python
def longest_substring_without_repeating_chars(s):
    char_index = {}
    max_length = 0
    start = 0
    result = ""
    
    for end in range(len(s)):
        char = s[end]
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        current_length = end - start + 1
        if current_length > max_length:
            max_length = current_length
            result = s[start:end + 1]
    
    return result
```
ornith-1.5:35b-a3b-mq4r battery turn 1
The sky appears blue during the day because sunlight interacts with Earth's atmosphere through a process called Rayleigh scattering, where shorter blue wavelengths scatter off air molecules much more readily than longer wavelengths. This scattered blue light spreads across the sky, reaching our eyes from all directions. At sunset, sunlight travels through a thicker slice of atmosphere, so most of the blue light is scattered away before reaching us, leaving the longer red and orange wavelengths to dominate the sky's appearance.
ornith-1.5:35b-a3b-mq4r battery turn 2
The capital of France is Paris, and the Seine River runs through it.
ornith-1.5:35b-a3b-mq4r battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
ornith-1.5:35b-a3b-mq4r battery turn 4
17 + 26 = 43

Answer: 43

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 8.1 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 0 0 133 59 None 223.91 False False False True
battery 1 stop 0 0 65 56 None 228.87 False False False True
battery 2 stop 0 0 15 13 None 230.77 False False False True
battery 3 stop 0 0 26 8 None 228.07 False False False True
battery 4 stop 0 0 19 11 None 228.92 False False False True
lfm2.5:1.2b battery turn 0
```python
def longest_unique_substring(s):
    seen = {}
    left = 0
    max_len = 0
    max_sub = ""
    for right in range(len(s)):
        if s[right] in seen:
            while s[left] != s[right] and left < right:
                seen.remove(s[left])
                left += 1
        seen[s[right]] = right
        if right - left + 1 > max_len:
            max_len = right - left + 1
            max_sub = s[left:right+1]
    return max_sub
```
lfm2.5:1.2b battery turn 1
The sky appears blue during the day because sunlight scatters in all directions, and blue light scatters more easily due to its shorter wavelength. At sunset, the sky appears red because the sunlight travels through more atmosphere, scattering away the blue light and allowing longer red wavelengths to dominate. This phenomenon is known as Rayleigh scattering.
lfm2.5:1.2b battery turn 2
The capital of France is Paris, and the River Seine runs through it.
lfm2.5:1.2b battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
lfm2.5:1.2b battery turn 4
Let's calculate the sum:  
17 + 26 = 43  
Answer: 43

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 22.1 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
battery 0 stop 34 0 143 67 310.7 28.4 False False False True
battery 1 stop 30 0 103 86 416.7 48.8 False False False True
battery 2 stop 31 0 16 13 429.5 49.0 False False False True
battery 3 stop 47 0 31 8 478.4 48.8 False False False True
battery 4 stop 47 0 17 7 478.1 49.0 False False False True
qwen3.8:27b-mq4-xt battery turn 0
```python
def longest_substring_without_repeating(s: str) -> str:
    if not s:
        return ""
    
    char_index = {}
    start = 0
    max_start = 0
    max_length = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```
qwen3.8:27b-mq4-xt battery turn 1
During the day, sunlight passes through the atmosphere where shorter blue wavelengths scatter more effectively than other colors, a phenomenon known as Rayleigh scattering, causing the sky to appear blue. As the sun approaches the horizon at sunset, its light must travel through a much thicker layer of atmosphere, scattering nearly all the blue and green light away from the line of sight. This leaves the longer-wavelength red and orange light to pass through relatively unobstructed, reaching our eyes and illuminating the sky in warm hues.
qwen3.8:27b-mq4-xt battery turn 2
The capital of France is Paris, and the River Seine runs through it.
qwen3.8:27b-mq4-xt battery turn 3
{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}
qwen3.8:27b-mq4-xt battery turn 4
17 + 26 = 43

Answer: 43

chain — exit 0 seconds 24.4 status pass

mode idx genre finish ctx cached gen ans_words prefill_tok_s decode_tok_s attractor empty runaway recall_ok
chain 0 stop 44 0 114 47 389.9 48.8 False False False True
chain 1 stop 218 44 127 73 60.8 48.9 False False False True
chain 2 stop 375 44 62 47 722.4 49.2 False False False True
chain 3 stop 475 44 105 78 252.7 49.2 False False False True
chain 4 stop 616 44 74 54 251.0 49.0 False False False True
qwen3.8:27b-mq4-xt chain turn 0
```python
def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list."""
    result = []
    i, j = 0, 0
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1
    result.extend(a[i:])
    result.extend(b[j:])
    return result
```
qwen3.8:27b-mq4-xt chain turn 1
**Step 1: Calculate distance for the first segment.**
Distance = Speed × Time
Distance₁ = 60 mph × 2.5 hours = 150 miles

**Step 2: Calculate distance for the second segment.**
Distance = Speed × Time
Distance₂ = 40 mph × 1.5 hours = 60 miles

**Step 3: Add the distances together.**
Total Distance = Distance₁ + Distance₂
Total Distance = 150 miles + 60 miles = 210 miles

**Final Answer:** 210 miles
qwen3.8:27b-mq4-xt chain turn 2
The seasons are caused by the Earth's 23.5-degree axial tilt as it orbits the Sun. This tilt results in varying amounts of sunlight and differing day lengths at different latitudes throughout the year. Consequently, hemispheres tilted toward the Sun experience summer, while those tilted away experience winter.
qwen3.8:27b-mq4-xt chain turn 3
Elara found the smooth, glass-like sphere tumbled between the barnacles, humming with a faint, rhythmic pulse. She lifted the heavy object, feeling warmth seep into her frozen fingers, and noticed the sand-colored swirls inside began to glow brighter. As she carried it up the spiral stairs, the lighthouse beam outside suddenly dimmed, seemingly responding to the light in her hands. By morning, the sphere had vanished, leaving only a faint, salt-dry smell of jasmine where it had been.
qwen3.8:27b-mq4-xt chain turn 4
1. Write clear, descriptive variable and function names that explain intent.
2. Keep functions short and focused on a single responsibility.
3. Add concise comments only to explain complex logic, not obvious code.
4. Write unit tests to verify behavior and prevent regressions.
5. Refactor regularly to remove duplication and keep the codebase clean.

kernel

status: pass

report pass: True

@hipfire-sol

hipfire-sol Bot commented Sep 4, 2026

Copy link
Copy Markdown

hw-gate sol verdict

{
  "claim_verdict": "not-exercised",
  "confidence": 0.93,
  "coverage": {
    "gaps": [
      "No plain-Llama/Qwen3 qt44/47-50 artifact exercised the new all-architecture fallback in crates/hipfire-runtime/src/llama.rs.",
      "The added mq4v2_gemm_parity disjoint-halves arm was not run in hw-gate.json; the Redline kernel report used qwen3.6-27b.mq4 and does not provide the required direct MQ4V2 half-header parity output.",
      "No base-versus-head measurement quantifies the expected gfx12 plain-Llama prefill performance regression caused by replacing batched prefill with per-token fallback."
    ],
    "surfaces_evidenced": [
      "load",
      "qwen35 MQ-V2 prefill on gfx1100 and gfx1201",
      "decoded generation coherence"
    ],
    "surfaces_touched": [
      "kernel",
      "load",
      "prefill",
      "docs"
    ]
  },
  "decision": "needs-human",
  "eyeball": [
    "qwen3.8:27b-mq4-xt battery outputs on gfx1100 and gfx1201 are coherent across code, science, factual recall, JSON, and arithmetic prompts; all expected substrings are present and no attractors, empty responses, special-token leakage, or runaways appear.",
    "qwen3.8:27b-mq4-xt chain outputs on both architectures remain coherent through increasing context and related turns, including valid merge code and the correct 210-mile calculation.",
    "The unusually slow first battery prefill on several fixtures is consistent with cold initialization; subsequent turns recover and decoded text remains coherent."
  ],
  "phase": "verdict",
  "rationale": "The available MQ4-XT artifact validates the Qwen3.5-family admission path on both gfx1100 and gfx1201: battery and chain pass with coherent decoded text. Load behavior is also exercised. However, the behavioral change at crates/hipfire-runtime/src/llama.rs:1917 is untested because no plain-Llama/Qwen3 V2 fixture ran, and the central discriminating-kernel claim is unsupported because hw-gate.json contains no mq4v2_gemm_parity output or swapped-header negative-control results. The generic Redline pass uses qwen3.6-27b.mq4 and cannot substitute for direct MQ4V2 parity. Human review is therefore required before accepting the uncovered fallback and parity claims.",
  "regressions": []
}

Floor: hard=[] soft=["coverage_gaps: ['No plain-Llama/Qwen3 qt44/47-50 artifact exercised the new all-architecture fallback in crates/hipfire-runtime/src/llama.rs.', 'The added mq4v2_gemm_parity disjoint-halves arm was not run in hw-gate.json; the Redline kernel report used qwen3.6-27b.mq4 and does not provide the required direct MQ4V2 half-header parity output.', 'No base-versus-head measurement quantifies the expected gfx12 plain-Llama prefill performance regression caused by replacing batched prefill with per-token fallback.']", 'model needs-human'] model_decision=needs-human final=needs-human

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict needs-human: verdict parse failed, fail-closed to needs-human

@hipfire-fable

hipfire-fable Bot commented Sep 4, 2026

Copy link
Copy Markdown

announcement: Both lanes pass and I closed the three gaps Sol flagged. A base-vs-PR Redline A/B on qwen3.8:27b-mq4-xt (gfx1201) is bit-identical — same prefill/decode launch sequences and the same logits/KV/recurrent hashes — so the qwen35 delegation is a true no-op, and HIPFIRE_MQV2_GFX11_WMMA=0 correctly leaves gfx12 alone. I ran the PR's mq4v2_gemm_parity on gfx1201: the disjoint-halves arm passes (v2 rel-rms ~2-3e-4) with the swapped-header control disagreeing by 0.5-1.6, so the negative control is doing real work. The plain-Llama refusal can't be exercised on hardware because no arch 0/1 MQ-V2 artifact exists in the registry, but I traced forward_prefill_chunk myself (llama.rs:2577-2605, :2731-2745) and agree master's gfx12 admission skipped the rotate and ran V1 launchers on V2 blobs — the refusal fixes a silent-corruption hole rather than removing working behavior. One small follow-up when convenient: the new comment in gemm_mq4g256v2_residual_mmq.hip cites gemm.rs:17860-17909 for the batch>=128 gate, but that is the launcher helper; the actual gate is at the WMMA wrapper call sites (e.g. gemm.rs:28029-28036) and also excludes replay-recording and graph-capture. Merging to staging.

override: Override Sol needs-human: I closed Sol's three gaps on this host. (1) The added mq4v2_gemm_parity disjoint-halves arm was run on gfx1201 from the PR build and PASSES with a load-bearing negative control (v2 rel-rms 1.8e-4..3.1e-4 vs swapped-header reference 0.53..1.62 across batch 1/8/12/16/32). (2) The qwen35 admission change is proven a no-op by a base-vs-PR Redline A/B on the real MQ4G256V2 artifact qwen3.8:27b-mq4-xt on gfx1201: prefill-128/512 and decode launch sequences, launch counts, and logits/KV/recurrent output hashes are bit-identical, and HIPFIRE_MQV2_GFX11_WMMA=0 leaves gfx12 untouched as specified. (3) The plain-Llama V2 refusal cannot be exercised on any host because no arch 0/1 MQ-V2 artifact exists in the registry (the only arch-1 entries are qwen3:0.6b/8b .hf4); but the source trace is unambiguous that master's gfx12 admission ran the V1 hfq4g256 launchers on V2 blobs with the FWHT rotate skipped (llama.rs:2577-2605, :2731-2745), so the PR converts a silently-incorrect path into per-token decode and no working behavior is lost. A 'perf regression' measurement against a path that produced wrong activations is not meaningful.

investigation:

| question | route | result | evidence |

|---|---|---|---|

| Does the qwen35 delegation to llama::mqv2_wmma_batchable change MQ-V2 prefill dispatch or output on gfx12? (A/B master vs PR on the real qt=44 fixture) | HIP_VISIBLE_DEVICES=1 python3 scripts/redline_daemon_harness.py --model $HIPFIRE_MODELS_DIR/qwen3.8-27b.mq4-xt --daemon $HW_GATE_BASE_BIN/daemon --out $HW_GATE_EVIDENCE/redline-qwen38-27b-mq4xt-gfx1201-BASE.json --capture-repeats 2 --measure-repeats 2 --decode-iterations 16 --timeout 300 ; then identical with --daemon $HW_GATE_BIN/daemon → ...-PR.json | Bit-identical across base and PR: prefill128 hash 2446cb9fe53c0779 (1140 launches), prefill512 hash 04eea88a8cd26904 (2280 launches = 2 chunks of 256, i.e. batched WMMA prefill, not per-token), decode hash 9afd3ae37c853481 (899 launches), shadow bit_exact=true with identical logits e3db4bf8eea6d23a / kv 81cdc593a6c30367 / recurrent c45b226d89a1dcf5. Decode capture names fused_qkv_mq4g256v2, fused_qkvza_mq4g256v2, gemv_mq4g256v2_residual — the fixture is genuinely MQ4G256V2. Only kernarg pointer values differ (different processes). prefill512 median 862.0 vs 857.4 tok/s, decode 36.1 vs 36.2. | .stderr.log) |

| Does the shared rule's kill-switch plumbing (HIPFIRE_MQV2_GFX11_WMMA=0) leave gfx12 admission untouched end-to-end through hipfire_config::developer_var? | HIP_VISIBLE_DEVICES=0 HIPFIRE_MQV2_GFX11_WMMA=0 python3 scripts/redline_daemon_harness.py --model $HIPFIRE_MODELS_DIR/qwen3.8-27b.mq4-xt --daemon $HW_GATE_BIN/daemon --out $HW_GATE_EVIDENCE/redline-qwen38-27b-mq4xt-gfx1201-PR-killswitch0.json --capture-repeats 2 --measure-repeats 2 --decode-iterations 16 --timeout 300 | Identical to the PR default arm: same prefill/decode sequence hashes, same launch counts, shadow bit_exact=true, same output hashes. gfx12 is unaffected by the gfx11 kill-switch, matching llama.rs:1849-1861. | redline-qwen38-27b-mq4xt-gfx1201-PR-killswitch0.json |

| Does the new disjoint-halves GEMM parity arm actually discriminate a half-select bug, and does the live v2 residual WMMA kernel pass it? (Sol gap 2) | HIP_VISIBLE_DEVICES=2 $HW_GATE_BIN/examples/mq4v2_gemm_parity (PR build, md5 e3b815787fd53ef18a579200f7ca1c6c, contains the arm-2 strings; head 4e360fb) | PASS, exit 0. Disjoint arm: batch 1/8/12/16/32 v2 rel-rms 1.85e-4 / 3.07e-4 / 3.03e-4 / 2.70e-4 / 2.50e-4 against the exact-dequant reference, while the swapped-header negative control disagrees by 5.35e-1 / 1.62 / 1.32 / 1.22 / 1.26 — three to four orders of magnitude apart, so the control is load-bearing. Gaussian arm unchanged (v2 2.7e-4 vs v1 8-9.6e-4); gate_up v2 2.62e-4; qkvza v2 2.62e-4 on all four outputs. | mq4v2_gemm_parity-gfx1201-PR.log |

| Is the author's justification for refusing MQ-V2 in llama::is_batchable_la true — does llama.rs::forward_prefill_chunk really lack V2 arms so that master's gfx12 admission ran V1 launchers on V2 blobs? And who is actually affected? | Source trace of PR head and git show 931b571c:crates/hipfire-runtime/src/llama.rs; registry/v1.json arch_id inventory; git diff 931b571..HEAD -- kernels/ crates/hipfire-daemon | Confirmed. llama.rs:2577-2580 qkv_is_mq = MQ4G256\|MQ6G256\|MQ3G256\|MFP4G32 — MQ4G256V2 is false, so the FWHT rotate is skipped (:2587-2605) and the projection falls to the final else → gemm_qkv_hfq4g256 (:2731-2745); wo/ffn/w_down matchers (:3032/:3124/:3255) behave the same. Master has the identical V1-only matchers and admitted V2 on gfx1200/1201 (master :1877-1891). Only arch_id 0/1 (LlamaCarrier, carriers.rs:733-737) reach this path; arch 5/6 use qwen35's own chunk path with 76 MQ4G256V2 hits. The registry's only arch-1 entries are qwen3:0.6b/qwen3:8b .hf4 (HFQ4) — no registry artifact is refused by the change. kernels/ diff is comment-only; hipfire-daemon has 0 diff lines. | source-trace-llama-v2-arms.md |

unproven:

  • End-to-end generation on a plain Llama/Qwen3 (arch 0/1) MQ-V2 artifact: no such artifact exists in the registry or on this host, so the per-token fallback that replaces master's gfx12 batched path is proven only by source trace (llama.rs:2577-2605, :2731-2745) and unit tests. Proving it on hardware needs a hipfire-quantize --format mq4 (qt=44) of qwen3:8b on a gfx1201 box; the expected result is coherent-but-slower prefill on the PR and incoherent prefill on master.

  • Base-vs-PR A/B of the qwen35 MQ-V2 path on gfx11: this host is gfx1201-only. The gfx11 admission rule is a verbatim move of the helper qwen35 already used, the mandatory hipx lane's battery+chain on qwen3.8:27b-mq4-xt decoded coherently with prefill rates consistent with the batched path (251-722 tok/s), and unit tests pin gfx11 admission, but no gfx1100 sequence-hash A/B was run.

rationale: The PR has exactly one behavior change and one refactor. The refactor — qwen35::is_batchable_la (prefill.rs:1543-1560) delegating to llama::mqv2_wmma_batchable (llama.rs:1871-1880) — is proven a no-op on gfx1201 by a base-vs-PR Redline A/B on the real MQ4G256V2 fixture qwen3.8:27b-mq4-xt (sha256 9f91556f…, size 14980361216): prefill-128/512 sequence hashes, launch counts (1140/2280, i.e. two 256-token chunks of batched WMMA prefill), decode sequence hash, and logits/KV/recurrent output hashes are identical between $HW_GATE_BASE_BIN/daemon and $HW_GATE_BIN/daemon ($HW_GATE_EVIDENCE/ab-summary-qwen38-mq4xt-gfx1201.json), and HIPFIRE_MQV2_GFX11_WMMA=0 leaves gfx12 unchanged. The mandatory lanes add coherent battery+chain output for the same fixture on gfx1100 and gfx1201. The behavior change — llama::is_batchable_la (llama.rs:1917-1947) refusing MQ4G256V2/MQ4CG256/MQ6-2G256V2 on every arch instead of admitting them on gfx12 — only affects arch_id 0/1 (carriers.rs:733-737), for which the registry carries no MQ-V2 artifact (qwen3:0.6b/8b are .hf4). I verified the author's justification directly: forward_prefill_chunk's matchers (llama.rs:2577-2580, :3032, :3124, :3255) list only V1 dtypes, so an admitted V2 layer skips the FWHT rotate and lands on gemm_qkv_hfq4g256 / gemm_hfq4g256_residual / gemm_gate_up_hfq4g256 — master's gfx12 path for that class was silently wrong, and the PR moves it to per-token decode. That is a fail-closed rule that refuses no real artifact and replaces incorrect behavior, so it is not a regression under the master-is-the-oracle standard. The new mq4v2_gemm_parity disjoint-halves arm was run from the PR build on gfx1201 and passes with a negative control that separates correct and swapped-header decodes by three to four orders of magnitude ($HW_GATE_EVIDENCE/mq4v2_gemm_parity-gfx1201-PR.log). The two .hip changes are comment-only and the daemon is untouched. One doc nit for the author: gemm_mq4g256v2_residual_mmq.hip:12-14 cites gemm.rs:17860-17909 for the batch>=128 && batch%128==0 selection, but that range is the launcher helper; the gate lives at the WMMA wrapper call sites (e.g. gemm.rs:28029-28036) and also requires not-recording and not-capturing. To reverse this decision a maintainer would need either an arch 0/1 MQ-V2 artifact that decodes coherently through master's batched path on gfx12 (the source says it cannot), or a gfx11 A/B showing the qwen35 sequence hash changed.

merged: beta d14b3a0

soft floor: ["coverage_gaps: ['No plain-Llama/Qwen3 qt44/47-50 artifact exercised the new all-architecture fallback in crates/hipfire-runtime/src/llama.rs.', 'The added mq4v2_gemm_parity disjoint-halves arm was not run in hw-gate.json; the Redline kernel report used qwen3.6-27b.mq4 and does not provide the required direct MQ4V2 half-header parity output.', 'No base-versus-head measurement quantifies the expected gfx12 plain-Llama prefill performance regression caused by replacing batched prefill with per-token fallback.']", 'model needs-human']

@hipfire-fable hipfire-fable Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fable unavailable; holding for human review. omp decide: no JSON object in assistant text

@hipfire-fable hipfire-fable Bot added the needs-human hw-gate reviewer requests a human decision label Sep 4, 2026

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict needs-human: The available MQ-V2-adjacent fixture qwen3.8:27b-mq4-xt loaded and generated coherent battery and chain output on both gfx1100 and gfx1201, and the Redline harness passed stable capture plus bit-exact HIP/AQL parity. However, the principal behavior change at crates/hipfire-runtime/src/llama.rs:1916 newly admits plain Llama/Qwen3 MQ-V2 WMMA prefill on gfx11, and no artifact exercising that architecture path was run. The dedicated disjoint-halves GPU parity example was also absent from the gate evidence. These are material coverage gaps for a kernel-admission change, requiring human disposition rather than a greenlight.

@hipfire-fable hipfire-fable Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two of the three gaps Sol flagged closed clean on gfx1201: your parity example passes here with the disjoint arm discriminating exactly as you reported (2.5e-4 vs 1.263 at batch 32), and a base-vs-PR redline A/B on qwen3.8-27b.mq4-xt is bit-identical down to the logits hash, with the prefill capture naming gemm_qkv/qkvza/gate_up/residual_mq4g256v2_wmma_gfx12 as the executed path. The blocker is the lockstep change itself: llama.rs forward_prefill_chunk has no MQ-V2 arms (matchers at 2570-2573 / 3025-3028 / 3117-3120 / 3248-3251 list MQ4G256|MQ6G256|MQ3G256|MFP4G32 only), so a plain Llama/Qwen3-dense qt44/47-50 model admitted on gfx11 now skips the FWHT rotate and lands on gemm_qkv_hfq4g256 / gemm_hfq4g256_residual / gemm_gate_up_hfq4g256 — the V1-on-V2 mis-decode your own comment at llama.rs:1501-1507 describes — where master ran it per-token and correct. Either give the llama chunk path real V2 arms and show a --format mq4 qwen3-8b on gfx1100 through serve_harness + redline, or keep the shared rule but have llama exclude MQ-V2 until then (that also fixes master's existing gfx12 admission) and make the lockstep test assert the intended asymmetry. The parity arm and doc corrections are good and should come back with the fix; evidence is in fable-evidence/, see README.md and STATIC-llama-batched-prefill-no-mqv2-arms.md. Hard floor: clear. Mandatory routes: all 4 fixtures x 2 lanes pass with coherent decoded turns, and the hw-gate redline route passed on qwen3.6-27b.mq4 (a V1 fixture, so it did not touch the dtype this PR is about). What I added: (a) redline on the actual MQ-V2 fixture qwen3.8-27b.mq4-xt on gfx1201, PR vs master — sequence hashes and logits/kv/recurrent hashes bit-identical (redline-mq4v2xt-gfx1201-AB-summary.txt), so on gfx12 this PR changes nothing for the qwen35 path, as the diff predicts; (b) the prefill kernel names for that fixture, which are the gemm__mq4g256v2_wmma_gfx12 family (Sol gap 3 closed); (c) the PR's mq4v2_gemm_parity example on gfx1201, PASS with the disjoint arm at 1.8e-4..3.1e-4 vs 0.53..1.62 swapped (Sol gap 2 closed). The block is item 1 of the PR body: 'plain Llama/Qwen3 dense models on MQ-V2 now take WMMA prefill on gfx11 (they were per-token)'. That admission leads into crates/hipfire-runtime/src/llama.rs forward_prefill_chunk, which has no MQ-V2 branch in any of its four GEMM chains (matchers 2570-2573, 3025-3028, 3117-3120, 3248-3251; fallthroughs 2725, 3105, 3236, 3335 to hfq4g256 V1 launchers; no rotate at 2590). llama.rs:1501-1507 spells out the result of routing a V2 blob to a V1 launcher. The lockstep premise — same admit rule for llama and qwen35 — is only sound if both batched bodies dispatch the same dtype set; qwen35/prefill.rs does (hence the clean gfx1100/gfx1201 lanes on the arch-5 fixture), llama.rs does not. The audit's 'Broken 1' was two crates disagreeing; the fix chosen makes them agree on the side that is wrong for llama. Correct options: (1) add MQ-V2 arms to forward_prefill_chunk (rotate + gemm_qkv_mqg256v2_wmma / gemm_gate_up_mqg256v2_wmma / gemm_mqg256v2_residual_wmma via the same rdna-compute entry points qwen35 uses, per the scout table) and evidence it on gfx1100 with a --format mq4 qwen3-8b through serve_harness battery + redline; or (2) keep mqv2_wmma_batchable shared but have llama::is_batchable_la exclude MQ-V2 on every arch until (1) lands — which also closes master's pre-existing gfx12 hole — and rewrite mqv2_admit_llama_qwen35_lockstep to assert the intended asymmetry instead of equality. Items 2 and 3 of the PR (parity arm, doc/comment corrections) are verified and fine to carry into the resubmission. Registry impact today is nil (no arch-0/1 MQ-V2 artifact), which is why nothing in the gate could see this; that is an argument for why it slipped, not for merging it. Override Sol needs-human: Sol's three coverage gaps split two ways once I ran them. Gaps 2 and 3 closed clean on this host: the PR's parity example passes on gfx1201 with the disjoint-halves arm discriminating (v2 rel-rms 1.8e-4..3.1e-4 vs swapped-header 0.53..1.62), and a base-vs-PR redline A/B on qwen3.8-27b.mq4-xt is bit-identical (same prefill/decode sequence hashes, same logits/kv/recurrent hashes) with the prefill capture naming gemm_qkv/qkvza/gate_up/residual_mq4g256v2_wmma_gfx12 as the executed path. Gap 1 — the PR's headline behaviour change — is not a coverage gap; it is a defect. llama.rs's batched prefill body (forward_prefill_chunk) has no MQ-V2 arms at all: the per-layer matchers at llama.rs:2570-2573/3025-3028/3117-3120/3248-3251 list MQ4G256|MQ6G256|MQ3G256|MFP4G32 only, so a *G256V2 layer gets rmsnorm_batched (no FWHT) and the else arms gemm_qkv_hfq4g256 (2725), gemm_hfq4g256_residual (3105, 3335), gemm_gate_up_hfq4g256 (3236) — V1 launchers on V2 blobs, the failure llama.rs:1501-1507 itself describes as 'bit_casts an fp16 pair to f32 and decodes every weight to ~1e-14 — no error, full speed, pure noise'. On master, arch-0/1 MQ-V2 on gfx11 is not eligible (llama.rs:2181-2208) and takes the per-token path whose GEMV arms (1339-1349, 1418-1428) decode V2 correctly. The PR turns correct-but-slow into silently wrong for that class. More hardware evidence cannot fix this; the code must change, so block rather than needs-human.

@hipfire-fable hipfire-fable Bot added hw-gate-blocked hw-gate reviewer blocked on evidence and removed needs-human hw-gate reviewer requests a human decision labels Sep 4, 2026
…efill_chunk has no V2 arms

hw-gate Fable seat on #690 (run 33895641944), source trace verified: the shared MQ-V2 admit rule made llama::is_batchable_la admit plain Llama/Qwen3-dense qt44/47-50 artifacts to WMMA prefill on gfx11 and gfx12, but llama.rs::forward_prefill_chunk's per-layer matchers (qkv_is_mq ~:2570, wo_is_mq ~:3025, ffn_is_mq ~:3117, w_down_is_mq ~:3248) list only MQ4G256|MQ6G256|MQ3G256|MFP4G32 — an admitted V2 model skips the FWHT rotate and runs the V1 hfq4g256 launchers on V2 blobs: silently incoherent prefill. master's pre-existing mq4_v2_gfx12 arm had the same hole on gfx12; no gfx12 Llama-V2 artifact has ever tripped it.

llama::is_batchable_la now refuses every *G256V2 dtype and MQ4CG256 on every arch, with the reason at the site. qwen35::is_batchable_la keeps the shared mqv2_wmma_batchable rule (its chunk path has the V2 arms; gfx11 kill-switch intact). llama_spec::batched_verify_eligible routes all seven weights through is_batchable_la, so it is covered without an edit. Lockstep test now asserts the true contract: agreement on every non-V2 dtype across 5 arches; for V2, qwen35 admits on gfx11/gfx12 and llama refuses everywhere. Spec §9 row and crate maps corrected.

hipfire-runtime is_batchable_la: 7 passed; qwen35 is_batchable + lockstep: 9 passed.
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator Author

Fable's finding on run 33895641944 is fixed in 4e360fb2a — and it was a real bug, not a coverage gap: llama.rs::forward_prefill_chunk has no V2 arms (matchers at ~2570/3025/3117/3248 list only MQ4G256|MQ6G256|MQ3G256|MFP4G32), so the "lockstep" would have sent a Llama/Qwen3-dense V2 model through V1 hfq4g256 launchers on V2 blobs. llama::is_batchable_la now refuses every *G256V2 + MQ4CG256 on every arch (master's old gfx12-only arm had the same hole; closed rather than widened). qwen35 keeps the shared rule — its prefill has the arms, and Fable's base-vs-PR Redline A/B on qwen3.8-27b.mq4-xt was bit-identical to the logits hash with the *_mq4g256v2_wmma_gfx12 kernels named in the capture. batched_verify_eligible routes through is_batchable_la, so spec-decode verify is covered too. Lockstep test rewritten to the true contract; spec §9 and the PR body corrected.

@github-actions github-actions Bot removed the hw-gate-blocked hw-gate reviewer blocked on evidence label Sep 4, 2026

@hipfire-sol hipfire-sol Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hw-gate sol verdict needs-human: The available MQ4-XT artifact validates the Qwen3.5-family admission path on both gfx1100 and gfx1201: battery and chain pass with coherent decoded text. Load behavior is also exercised. However, the behavioral change at crates/hipfire-runtime/src/llama.rs:1917 is untested because no plain-Llama/Qwen3 V2 fixture ran, and the central discriminating-kernel claim is unsupported because hw-gate.json contains no mq4v2_gemm_parity output or swapped-header negative-control results. The generic Redline pass uses qwen3.6-27b.mq4 and cannot substitute for direct MQ4V2 parity. Human review is therefore required before accepting the uncovered fallback and parity claims.

hipfire-fable Bot added a commit that referenced this pull request Sep 4, 2026
…llama and qwen35; discriminating GEMM parity; spec §9) to staging

@hipfire-fable hipfire-fable Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both lanes pass and I closed the three gaps Sol flagged. A base-vs-PR Redline A/B on qwen3.8:27b-mq4-xt (gfx1201) is bit-identical — same prefill/decode launch sequences and the same logits/KV/recurrent hashes — so the qwen35 delegation is a true no-op, and HIPFIRE_MQV2_GFX11_WMMA=0 correctly leaves gfx12 alone. I ran the PR's mq4v2_gemm_parity on gfx1201: the disjoint-halves arm passes (v2 rel-rms ~2-3e-4) with the swapped-header control disagreeing by 0.5-1.6, so the negative control is doing real work. The plain-Llama refusal can't be exercised on hardware because no arch 0/1 MQ-V2 artifact exists in the registry, but I traced forward_prefill_chunk myself (llama.rs:2577-2605, :2731-2745) and agree master's gfx12 admission skipped the rotate and ran V1 launchers on V2 blobs — the refusal fixes a silent-corruption hole rather than removing working behavior. One small follow-up when convenient: the new comment in gemm_mq4g256v2_residual_mmq.hip cites gemm.rs:17860-17909 for the batch>=128 gate, but that is the launcher helper; the actual gate is at the WMMA wrapper call sites (e.g. gemm.rs:28029-28036) and also excludes replay-recording and graph-capture. Merging to staging. The PR has exactly one behavior change and one refactor. The refactor — qwen35::is_batchable_la (prefill.rs:1543-1560) delegating to llama::mqv2_wmma_batchable (llama.rs:1871-1880) — is proven a no-op on gfx1201 by a base-vs-PR Redline A/B on the real MQ4G256V2 fixture qwen3.8:27b-mq4-xt (sha256 9f91556f…, size 14980361216): prefill-128/512 sequence hashes, launch counts (1140/2280, i.e. two 256-token chunks of batched WMMA prefill), decode sequence hash, and logits/KV/recurrent output hashes are identical between $HW_GATE_BASE_BIN/daemon and $HW_GATE_BIN/daemon ($HW_GATE_EVIDENCE/ab-summary-qwen38-mq4xt-gfx1201.json), and HIPFIRE_MQV2_GFX11_WMMA=0 leaves gfx12 unchanged. The mandatory lanes add coherent battery+chain output for the same fixture on gfx1100 and gfx1201. The behavior change — llama::is_batchable_la (llama.rs:1917-1947) refusing MQ4G256V2/MQ4CG256/MQ6-2G256V2 on every arch instead of admitting them on gfx12 — only affects arch_id 0/1 (carriers.rs:733-737), for which the registry carries no MQ-V2 artifact (qwen3:0.6b/8b are .hf4). I verified the author's justification directly: forward_prefill_chunk's matchers (llama.rs:2577-2580, :3032, :3124, :3255) list only V1 dtypes, so an admitted V2 layer skips the FWHT rotate and lands on gemm_qkv_hfq4g256 / gemm_hfq4g256_residual / gemm_gate_up_hfq4g256 — master's gfx12 path for that class was silently wrong, and the PR moves it to per-token decode. That is a fail-closed rule that refuses no real artifact and replaces incorrect behavior, so it is not a regression under the master-is-the-oracle standard. The new mq4v2_gemm_parity disjoint-halves arm was run from the PR build on gfx1201 and passes with a negative control that separates correct and swapped-header decodes by three to four orders of magnitude ($HW_GATE_EVIDENCE/mq4v2_gemm_parity-gfx1201-PR.log). The two .hip changes are comment-only and the daemon is untouched. One doc nit for the author: gemm_mq4g256v2_residual_mmq.hip:12-14 cites gemm.rs:17860-17909 for the batch>=128 && batch%128==0 selection, but that range is the launcher helper; the gate lives at the WMMA wrapper call sites (e.g. gemm.rs:28029-28036) and also requires not-recording and not-capturing. To reverse this decision a maintainer would need either an arch 0/1 MQ-V2 artifact that decodes coherently through master's batched path on gfx12 (the source says it cannot), or a gfx11 A/B showing the qwen35 sequence hash changed. Override Sol needs-human: I closed Sol's three gaps on this host. (1) The added mq4v2_gemm_parity disjoint-halves arm was run on gfx1201 from the PR build and PASSES with a load-bearing negative control (v2 rel-rms 1.8e-4..3.1e-4 vs swapped-header reference 0.53..1.62 across batch 1/8/12/16/32). (2) The qwen35 admission change is proven a no-op by a base-vs-PR Redline A/B on the real MQ4G256V2 artifact qwen3.8:27b-mq4-xt on gfx1201: prefill-128/512 and decode launch sequences, launch counts, and logits/KV/recurrent output hashes are bit-identical, and HIPFIRE_MQV2_GFX11_WMMA=0 leaves gfx12 untouched as specified. (3) The plain-Llama V2 refusal cannot be exercised on any host because no arch 0/1 MQ-V2 artifact exists in the registry (the only arch-1 entries are qwen3:0.6b/8b .hf4); but the source trace is unambiguous that master's gfx12 admission ran the V1 hfq4g256 launchers on V2 blobs with the FWHT rotate skipped (llama.rs:2577-2605, :2731-2745), so the PR converts a silently-incorrect path into per-token decode and no working behavior is lost. A 'perf regression' measurement against a path that produced wrong activations is not meaningful.

@hipfire-fable hipfire-fable Bot added the merged-staging Fable merged this head into the staging branch (beta); promotion to master is the maintainer's. label Sep 4, 2026
Kaden-Schutt added a commit that referenced this pull request Sep 4, 2026
…te map

hw-gate staging merge of 136e3f0 hit a 409 on crates/hipfire-runtime/map.md — the generated line-count table that #690 (on beta) and this PR both refreshed. Regenerated via scripts/check-crate-maps.py --write on the merged tree; no source conflicts.
Kaden-Schutt added a commit that referenced this pull request Sep 4, 2026
…rate map

hw-gate staging merge of 5b130d1 hit a 409 on crates/hipfire-runtime/map.md — the generated line-count block that #690 (on beta) and this PR both refreshed. Regenerated on the merged tree; no source conflicts.
ghazni101 pushed a commit to ghazni101/hipfire that referenced this pull request Sep 5, 2026
First kernel-bucket run (warpfront#690, 33892920406): both lanes passed, then both
seats returned empty text. omp log: "Pre-prompt context maintenance
triggered by pending prompt size" contextTokens=1,357,577 against a
1,000,000 window. The Redline report's decode.captures (516 KB of raw
per-dispatch records per lane) was inlined verbatim into the verdict and
decide prompts with the rest of hw-gate.json.

run.py: elide_captures() replaces capture-dump lists/dicts over 4 KB
(keys captures/dispatches/packets/raw) with their entry count; every
verdict field (pass, sequence_stable, measurement, aql_shadow, failures)
stays, and the full report is still on disk as hw-gate-logs/redline.json
in the artifact. On warpfront#690's real evidence: 1,282,896 -> 41,435 bytes per
lane; seat prompt ~10 K tokens.

review.py: evidence_for_prompt() caps inlined evidence at 600 KB with an
explicit truncation marker (the diff was already capped at 400 KB; the
evidence was not) so no future field can push a seat past the window.

103/103 in scripts/hw-gate/tests.
ghazni101 pushed a commit to ghazni101/hipfire that referenced this pull request Sep 5, 2026
Every rung of the 2026-09-04 ladder hit the same 409 on the staging merge:
`crates/*/map.md` carries a `<!-- crate-map:generated -->` block that both
branches regenerate, so any two PRs touching the same crate conflict there
while their real code merges cleanly. warpfront#689, warpfront#690, warpfront#691, warpfront#686, warpfront#687, warpfront#688 and
warpfront#682 all needed the same three manual steps -- merge staging in, regenerate the
block with scripts/check-crate-maps.py, merge -- six of them tonight. A gate
that decides merge-staging and then holds on a generated file is asking a human
to run a script, which is not review.

On a 409 the decide phase now retries locally: merge staging into the PR head,
and if the conflicted set is generated maps only, re-run check-crate-maps.py
for those crates, commit, and merge the result.

The retry is deliberately narrow, because auto-resolving conflicts is exactly
where a gate can do damage:
- if ANY conflicted path is not a `map.md`, it declines and the hold stands
  with the offending paths named -- a real code conflict must reach a human
- it regenerates rather than picking a side, so the committed block is what the
  tree actually generates, not whichever branch won
- a failed regeneration, a git error, or a timeout all decline rather than
  force

Test: `test_generated_map_retry_refuses_real_code_conflicts` builds a real repo
with a conflicting `.rs` and asserts the retry returns no merge SHA and names
the file. The guard is the part worth pinning; the happy path is exercised by
the ladder itself.

122/122 hw-gate tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merged-staging Fable merged this head into the staging branch (beta); promotion to master is the maintainer's.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant