Investigate Galaxy S24 Vulkan crash during first GGUF generation

[leehack]

A Firebase Test Lab pilot of the actual Flutter/llamadart application crashed on the first GGUF Vulkan generation on a physical Galaxy S24. CPU generation completed beforehand. This is a native crash investigation; the stack identifies the failing path but does not yet establish whether the origin is the driver, upstream llama.cpp, the packaged artifact, or consumer integration.

Reproduction inputs

• Observed 2026-09-16; llamadart main commit a5df1c4fcbb1766d26efb5b1d9becda191df89a3, llama.cpp artifact v0.4.0.
• Flutter 3.47.1 / Dart 3.13.1, Android Debug APK, arm64, both runtime families included. Package inference code was unmodified; the pilot only added a Flutter integration-test wrapper.
• Firebase physical model SC-51E, Galaxy S24, API 36 / Android 16, Adreno 750.
• Build fingerprint: samsung/SC-51E/SC-51E:16/BP2A.250605.031.A3/SC51EOMS1CZB4:user/release-keys.
• Model: stories15M.gguf at pinned revision, 98,357,920 bytes, SHA256 61b50d457809a5194818fd22e6724b456cd7bb9a6264c52c8110684c53f3704a. Download hash verified on device.

Using package:llamadart/llamadart.dart and a local path to that model:

final engine = LlamaEngine(LlamaBackend());
await engine.setNativeLogLevel(LlamaLogLevel.info);
await engine.loadModel(modelPath, modelParams: const ModelParams(
  contextSize: 1024,
  numberOfThreads: 4,
  numberOfThreadsBatch: 4,
  gpuLayers: ModelParams.maxGpuLayers,
  preferredBackend: GpuBackend.vulkan,
));
await for (final piece in engine.generate(
  'Once upon a time',
  params: const GenerationParams(maxTokens: 32, temp: 0, seed: 1),
)) {
  print(piece);
}
await engine.dispose();

The observed execution first ran the CPU configuration (gpuLayers: 0, preferredBackend: GpuBackend.cpu) through one warm-up and three generations, disposed that engine, then created the Vulkan engine. A fresh-process GPU-only reproduction is still needed to separate standalone failure from prior CPU initialization effects. No successful Release-build or second-device Vulkan control has been collected yet.

Expected and actual

Expected: nonempty streamed generation, a clean completion, and evidence of actual GPU offload.

Actual: load returned and reported Vulkan; the first generation killed the app with SIGSEGV / SEGV_MAPERR, fault address 0x2d0. Essential crash frames:

#00 /vendor/lib64/libllvm-qgl.so
#09 /vendor/lib64/libllvm-qgl.so CreateQGLCProgram(QGPUCompiler::CompileData*)
#12 /vendor/lib64/hw/vulkan.adreno.so
#14 /vendor/lib64/hw/vulkan.adreno.so qglinternal::vkCreateComputePipelines(...)
#15..19 base.apk native Vulkan module
#20 libggml-base.so ggml_backend_sched_graph_compute_async
#21 libllama.so llama_context::graph_compute
#22 libllama.so llama_context::process_ubatch
#23 libllama.so llama_context::decode
#24 libllama.so llama_decode

Relevant build IDs: vendor compiler e45a6c21c17a933fce15407e32a702bd; Vulkan driver 16fb0afc08003d7e9cb34edb847a0e53; packaged Vulkan module cb3986d59f53f6bdedc8bb85f9bff8173df839f5; libllama 2bf312b67373a8d028386b84581073cc8ccbfc66.

Firebase execution (project access required). CPU success is recorded in app events; the crashed overall instrumentation execution is not a passing matrix. iPhone CPU/Metal passed with the same GGUF, which does not qualify Android Vulkan.

Investigation and acceptance

• Reproduce in a fresh process on this exact device/OS, in Release and the original Debug mode; retain the CPU-before-GPU sequence as a separate case.
• Compare the same model, prompt and settings using a direct native/upstream reference at the artifact's recorded upstream revision. Preserve build flags, driver version and runtime identity.
• Cross-check a different Adreno generation and a Mali device before claiming a general Android or vendor-specific failure.
• Fix in the owning layer, or provide an explicit actionable unsupported-path policy if the combination cannot be supported. A silent CPU fallback must not count as a GPU pass.
• Add model-backed regression qualification and rerun CPU and verified Vulkan generation on the failing device after the fix.

Related qualification tracker: leehack/llamadart#476. That older-CPU/ISA coverage gap is separate from this reproduced GPU crash. No upstream defect attribution or new regression range has been established.

[leehack]

Verified v0.4.1 and fresh standalone control — 2026-09-19

The crash persists on the current published v0.4.1 runtime (upstream b29c606e28a01b1bc8c1351026a0fa6e616bf6c4). Two bounded SC-51E/API36 executions now establish:

1. Original Flutter pilot with v0.4.1, matrix matrix-2jkcwdec9di91: CPU warmup plus three generations passed and disposed; first Vulkan generation crashed. Original core a5df1c4... and settings retained, native tag overridden to v0.4.1. This is not a current-core-main test.
2. Fresh standalone C-API application, matrix matrix-29hxzbl7p2w4a: same crash without Dart, Flutter/Impeller, libllamadart, or preceding CPU-only inference. The Android app loads unchanged official v0.4.1 libraries and calls llama.cpp through a minimal JNI shim. This is not a raw-upstream rebuild or a modified Vulkan backend.

Both returned the original fingerprint samsung/SC-51E/SC-51E:16/BP2A.250605.031.A3/SC51EOMS1CZB4:user/release-keys, compiler BuildId e45a6c21c17a933fce15407e32a702bd, and Vulkan driver BuildId 16fb0afc08003d7e9cb34edb847a0e53.

Standalone package SHA256 146c84029fee7fcc721719a04e65a6b5f68066733e314862e85f5a5342d378ed. All 11 runtime libraries match the official archive byte-for-byte; only the JNI consumer shim is newly compiled. Model hash/size match the issue. Native logs confirm 57 F32 tensors, Adreno750, and 7/7 layers actually offloaded (57.95MiB Vulkan model buffers). CPU backend remains for non-offloaded ops; no prior CPU-only model run occurred.

Context settings match the original Android path: context1024, batch1024, microbatch512, threads4/4, F16 KV, offload_kqv=false, op_offload=false, Flash Attention disabled; max GPU layers. The first llama_decode received the five-token prompt Once upon a time and crashed before prefill returned or any token was generated.

Same SIGSEGV/SEGV_MAPERR at 0x2d0. Current Vulkan BuildId b16b1f2040b472a8f57561d1cdb7f0be2f1816f4; symbolized PCs match in both v0.4.1 runs:

1b36a9c createComputePipeline / ggml_vk_create_pipeline_func
1b282d4 ggml_vk_load_shaders                         :6511
1b81aa0 ggml_pipeline_request_descriptor_sets       :3336
1b8df58 ggml_vk_mul_mat_vec_q_f16                   :10011
1b512e0 ggml_vk_build_graph / graph_compute         :16467

Conclusion: Dart/wrapper lifecycle, Flutter rendering, and prior CPU disposal are not required to trigger this crash. We have not yet distinguished shader selection/configuration, artifact build, raw upstream behavior, and Qualcomm compiler compatibility. No driver-only attribution or safe fix yet. The exact selected shader/specialization remains uncaptured. All-F32 weights also mean this must not be labeled the Q4_K issue reported in upstream #28635; that report's initial NDK theory was retracted.

Next bounded investigation: same-revision pre-compilation diagnostics or upstream matrix-vector control to identify the selected F32 pipeline and then test a single configuration change against CPU correctness. Release and raw-upstream controls remain missing. No runtime pin/publication or speculative production patch made.