feat(sdk): run llama_cpp HTP on Hexagon SDK 5.5.x functional-safety firmware

[Tadayuki Okada (TadayukiOkada)]

Summary

Makes the llama_cpp plugin's Hexagon (HTP) path usable on DSP firmware that is older than upstream ggml-hexagon assumes, including functional-safety firmware. Four commits:

feat(sdk): build the HTP skels for Hexagon SDK 5.5.x firmware

• sdk/patches/llama-hexagon-sdk-select.patch adds cache variables, all defaulting to upstream behavior:
  • GGML_HEXAGON_HTP_SDK_ROOT / GGML_HEXAGON_HTP_TOOLS_ROOT: build only the skels with another SDK (the CPU-side libggml-hexagon.so still needs SDK 6.x headers).
  • GGML_HEXAGON_TOOLV: prebuilt lib dir prefix (toolv87 / toolv86 for SDK 5.5.x).
  • GGML_HEXAGON_HTP_ARCHS: SDK 5.5.x has no QuRT for v79/v81.
  • GGML_HEXAGON_SWIV_UTILITY: embed the SWIV CRC section that functional-safety firmware verifies before loading a library.
  • It also finds qaic in the SDK 5.5.x layout without writing into the SDK.
• arm64-oe-linux-fusa-{debug,release} presets and sdk/cmake/arm64-oe-linux.cmake: build outside the container with an OpenEmbedded gcc and sysroot, for rootfs images older than the container targets (glibc 2.35 / libstdc++ 6.0.29).
• The Rust model-manager build takes its compiler and archiver from the toolchain file (GENIEX_RUST_CC / GENIEX_RUST_AR, defaults unchanged) and passes the sysroot when one is set.
• notes/build.md documents the variables and the two on-device load failures they address.

fix(sdk): release VTCM and HMX between ops in the HTP skel

• sdk/patches/llama-hexagon-yield-between-ops.patch: the skel checked for a VTCM/HMX release request only between op batches. A batch can be a whole graph, so a long prefill held both resources for its full duration. On functional-safety firmware the periodic DSP self-test then timed out and the host shut the DSP down. The skel now checks after every op; with no pending request the cost is one branch.

fix(sdk): yield VTCM and HMX inside long copies in the HTP skel

• sdk/patches/llama-hexagon-yield-between-ops.patch now also copies non-contiguous tensors in 128 KiB chunks and yields between them. A transposed copy in a vision encoder (SigLIP, 1152 x 1024 F32) runs about 70 ms as one op, longer than the 50 ms self-test period. It is folded into the existing patch because the configure-time git apply --reverse --check needs patches that do not overlap.

perf(sdk): run F16/F32 matmuls with K not a multiple of 32 on HMX

• sdk/patches/llama-hexagon-hmx-pad-k.patch: the flat 2D HMX matmul pads K to a multiple of 32 with zeros for F16/F32 weights. Such matmuls fell back to HVX; the SigLIP so400m FFN down projection (K = 4304) took about 100 ms per op, long enough to trip the self-test. On HMX it takes about 3 ms. Batched and fused paths keep requiring K aligned to 32.

Test plan

Tested on an automotive board with functional-safety cDSP firmware (v73), Qwen3-0.6B Q4_0, all 29 layers on HTP0, built with arm64-oe-linux-fusa-release and -DGGML_HEXAGON_TOOLV=toolv86.

• Skel loads: no unresolved imports against the firmware shell, SWIV section present
• Without the yield patch: 128-token prefill at the default op batch shuts the DSP down (reproduced once)
• With the yield patch, default op batch: prefill of 64 / 128 / 256 / 512 / 1024 / 2048 / 3500 tokens passes
• Default bench (512-token prefill + 128 generated) x 11 runs, 128-token prefill x 21 runs, 1024-token generation x 3 runs: all pass, generated text identical across runs
• Decode speed unchanged by the patch (about 56 tok/s at short context); generated text identical to the unpatched skel where that one runs
• Release requests observed mid-batch with an instrumented skel: resources handed over within 1.2 ms of the request
• Real prompts on NPU produce coherent output; prefill logits vs CPU: top-1 agrees at 58 of 64 positions, mean absolute logit difference 0.066
• Existing container presets still configure, build and install in the geniex-toolchain-{linux,android}:v0.1.0 images: arm64-linux-snapdragon-{release,debug}, arm64-linux-cpu-release, arm64-android-snapdragon-release, arm64-android-cpu-release. Snapdragon packages still carry all four skels (v73/v75/v79/v81), built with Hexagon Tools 19.0.07 and without an SWIV section
• arm64-windows-snapdragon-* (no Windows build host available)
• Preset default toolv87 (Hexagon Tools 8.7.06 bundled in SDK 5.5.x): full configure, build and install pass; the skel has no unresolved imports against the firmware shell. On device: prefill of 32 to 2048 tokens and the default bench x 6 runs pass, generated text identical to the toolv86 skel
• -DGGML_HEXAGON_HTP_ARCHS="v73;v75" with SDK 5.5.7: full build and install pass, both skels carry the SWIV section (build only; no v75 device with such firmware was available). A v79 skel does not build with SDK 5.5.7, as documented

Vision-model checks for the last two commits (all runs had #1505 applied as well):

• test-backend-ops on a v73 dev kit, built from the patched submodule with regression cases added locally (K = 100 / 4304 for F16 and F32 weights, 128 KiB-boundary and transposed CONT): MUL_MAT 735/735, CONT 68/68, CPY 134/134
• LFM2.5-VL-3B Q4_0 with its mmproj, LM and vision encoder both on HTP (GGML_HEXAGON_DEVICES=2), 10-image prompt, 1 + 3 runs on the functional-safety board: same answer as the dev kit, no self-test failures. Longest op is the transposed copy (77 ms, yielding inside); FFN down is 3.4 ms. A VTCM + HMX client polling every 50 ms waited at most 6.8 ms (1200 samples, none over 50 ms)

🤖 Generated with Claude Code