Skip to content

strix-halo branch on Windows/TheRock gfx1151: one-line build fix; LLAMA_MMB_HC16 corrupts output; 964-1045 t/s prefill otherwise clean #24

Description

@olliehm

Environment: Strix Halo (Ryzen AI MAX+ 395 / gfx1151, 128 GB, 96 GB carve), Windows 11,
TheRock ROCm 10.1.0 SDK, Clang 23, stock runtime (no retained-PM4), model
unsloth Qwen3.8-Flash-Next UD-IQ4_XS (not the PROJFIX quant), -b/-ub 8192,
--load-mode none (lazy-direct unavailable on Windows; PLE on the mmap fallback).

1. Build fix needed (one line)

src/models/qwen4exp.cpp:1776 calls ple_reader->prefetch(...) unconditionally, but the
_WIN32 branch of llama_lazy_reader (llama-lazy-reader.h) only stubs gather().
Fix: add a no-op void prefetch(const int32_t *, int64_t) const {} beside the stubbed
gather (prefetch is a page-cache hint; no-op is behavior-correct on the mmap fallback).

2. Results with that fix (all correctness-gated: math probe + 24k needle, verbatim answers)

  • All 35 launcher env gates ON: corrupted output — reasoning degenerates into a slash
    flood (//////...), empty content. pp meanwhile 1080–1256 t/s (fast-but-wrong).
  • All gates OFF: clean, pp ~418.
  • Each family alone (MMB / HC / NORMCONV / QSA): clean.
  • Full set minus MMB: clean. Minus HC: clean. Minus QSA: still floods. Minus NORMCONV: no
    flood (one ambiguous non-flood wrong-answer run, unreproduced).
  • Full set with only LLAMA_MMB_HC16=0: fully clean — correct at 2k/8k/16k depth,
    needle exact at 24k — at pp 964–1045 t/s (262144 ctx), ~6x stock-mainline-class
    prefill on this hardware and ~85-90% of the Linux reference numbers, with no custom
    runtime and no lazy-direct.

Conclusion: LLAMA_MMB_HC16 (bf16 HC streams end-to-end) miscompiles or misbehaves in
combination with the HC family on Windows/TheRock 10.1/Clang 23. Everything else ports.
Given its ablation worth is only 1.08x, defaulting it off on _WIN32 (or gating it on the
tested runtime) would make the branch Windows-viable as-is.

3. MTP on Windows (informational)

unsloth self-contained mtp-...-Q8_0.gguf loads and drafts via
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75: acceptance 77–79%,
but net decode only ~24 t/s (vs ~34–38 on a branch carrying device-resident speculative
checkpoints on the same box) — consistent with the known host-round-trip cost per draft
round without on-device state save/restore. Not a bug report, just a data point:
the ggml-org#28118-class on-device checkpoint work matters more than kernels for MTP decode here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions