Skip to content

Fix shared-runtime linkage for desktop GPU builds - #50

Merged
leehack merged 2 commits into
mainfrom
fix/desktop-gpu-runtime-link
Sep 17, 2026
Merged

leehack merged 2 commits into
mainfrom
fix/desktop-gpu-runtime-link

Conversation

@leehack

@leehack leehack commented Sep 16, 2026

Copy link
Copy Markdown
Owner

Problem and correction

Desktop GPU crashes have two packaging causes: modern Linux/Windows builds use an obsolete shared-runtime flag, and Linux bundles a source-built core with upstream prebuilt accelerators. A matching SONAME does not establish ABI compatibility.

Select litert_runtime_link_mode=dynamic for Linux/Windows on upstream v0.16+, preserving older and Apple/Android configurations. On Linux, stage libLiteRt.so from the same upstream prebuilt directory as the accelerators; reject missing or non-ELF cores instead of falling back to Bazel outputs. Windows already stages the matching prebuilt DLL set.

Validation

  • All 178 tooling tests passed, including 15 focused build-command and staging tests.
  • Independent blocking review accepted exact head 4c66cb9f1f037f91eed9b1cbe9101463efd8e46a against dd86d49, with no blocking findings.
  • Linux x64 NVIDIA L4 / driver 580.173.02: published baseline and dynamic host with source-built core crash. Keeping host, accelerator, model and driver fixed while substituting the matching prebuilt core passes Qwen3-0.6B and Gemma 4 E2B GPU streaming, cancellation, reuse and disposal.
  • Original and tokenizer-repaired Qwen3 both pass GPU, establishing that tokenizer repair is independent. Repaired Qwen3 also passes full CPU chat, thinking and history smoke.
  • Windows Server 2022, NVIDIA L4 / driver 582.53, D3D12: final candidate isolates incorrect output to GPU sampling with temperature=0 and topK=40 for both Qwen3 and Gemma 4. Both pass at temperature=0/topK=1 and temperature=0.8/topK=40. Explicit native CPU-sampling fallback also passes while model execution remains on GPU. Consumer PR Make zero-temperature LiteRT generation greedy llamadart#508 normalizes zero-temperature native CPU/GPU generation to topK=1; actual service before/after tests fail before and pass after for both models, with CPU control passing. Hardware qualification overrides availability only; production desktop GPU gates remain unchanged. This packaging PR does not repair the upstream multi-candidate zero-temperature sampler itself.
  • Final-head Linux x64 host, matching core and constraint provider are byte-identical to the successful hardware-test inputs. Linux arm64 links the shared core and its checksum matches the upstream prebuilt.
  • Final-head release qualification run 35158029962 passed, including all nine platform builds, candidate verification, and packaged Qwen inference. Retain existing consumer GPU gates until publication and downstream qualification. Windows sampling mitigation is tracked separately in Make zero-temperature LiteRT generation greedy llamadart#508.

No release, consumer pin, or GPU gate changes. Publication and downstream support expansion remain separate. The runtime core source is an upstream prebuilt, not a patched binary.

Refs #47 and leehack/llamadart#506.

@leehack
leehack marked this pull request as ready for review September 17, 2026 00:39
@leehack
leehack merged commit 8174e0e into main Sep 17, 2026
17 checks passed
@leehack
leehack deleted the fix/desktop-gpu-runtime-link branch September 17, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant