Enhance LiteRT performance: enable GPU (OpenCL/Metal) and NPU (QNN) backends

[sagar-develop]

Summary

Inference today runs CPU-only (6 threads). Our Phase-0 benchmark concluded GPU/NPU "didn't work," but that was a property of the litert-community .litertlm bundles we pull — they don't ship GPU/NPU artifacts — not a limitation of LiteRT itself. Current LiteRT-LM has real, shipping GPU and NPU acceleration. This issue tracks unlocking it so we get the runtime's actual performance ceiling instead of leaving the GPU/NPU on the table or reaching for a second backend (llama.cpp) that would be slower on Android anyway.

Why this is the right lever (2026 facts)

• LiteRT-LM reaches ~52 tok/s decode on Gemma 4 E2B via the Android GPU (OpenCL) and ~56 tok/s on iOS (Metal), with NPU support via Qualcomm QNN.
• By contrast, llama.cpp on Android is effectively CPU-bound — the OpenCL Adreno backend gives only modest gains (~6–8 tok/s on a 7B) and Vulkan is unreliable/crashes on Adreno & Mali. So switching runtimes would not buy us speed.
• Conclusion: our "we're slow" problem is solvable within LiteRT by enabling the GPU/NPU path. (Model-catalog/bring-your-own-model is a separate concern and a separate decision.)

Scope

• Source or build GPU-capable .litertlm bundles (Google's optimized artifacts) rather than the community bundles that fail to load on GPU/NPU.
• Wire the GPU backend (OpenCL on Android, Metal on iOS) through the LocalAiEngine interface with graceful CPU fallback.
• Investigate the NPU path via Qualcomm QNN for supported devices (Snapdragon).
• Add a backend-selection / capability-detection step (NPU → GPU → CPU) so we always pick the best available accelerator per device.
• Re-run the Phase-0 benchmark across CPU vs GPU vs NPU and record tok/s + TTFT per device tier.
• Confirm multimodal (Gemma 4 E2B vision) still works on the accelerated path.

Acceptance criteria

• Measurable decode-speed improvement over the current CPU-only baseline on at least one mid/high-tier Android device, with GPU enabled and CPU fallback intact.
• Documented benchmark table (CPU vs GPU vs NPU) committed to the repo.

References

• LiteRT-LM overview: https://ai.google.dev/edge/litert-lm/overview
• Blazing-fast on-device GenAI with LiteRT-LM: https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm/
• Gemma 4 E2B + Qualcomm QNN deep-dive: https://medium.com/google-developer-experts/bringing-multimodal-gemma-4-e2b-to-the-edge-a-deep-dive-into-litert-lm-and-qualcomm-qnn-4e1e06f3030c
• Run LLMs using LiteRT-LM (NPU): https://ai.google.dev/edge/litert/next/litert_lm_npu

[sagar-develop]

Folding in the concrete implementation plan from #37 (closed as duplicate of this issue), which narrows the GPU-on-Android slice into actionable steps:

Concrete first step — GPU on Android with CPU fallback

1. Bump LiteRT-LM 0.11.0 → 0.13.1 in gradle/libs.versions.toml:15 (mind the Kotlin metadata-version-check workaround already in lib/build.gradle.kts). The CPU-only decision was made on 0.11.0; GPU bundle loading appears resolved on current runtime + bundles.
2. Backend selector in LiteRtLmLocalAiEngine.initializeEngine(...) (LiteRtLmLocalAiEngine.kt:93): attempt Backend.GPU() for text (and vision) backend, and in the existing try/catch that emits HardwareFault.DelegateFailure, automatically fall back to Backend.CPU(numOfThreads = 6) and re-init — so devices where GPU bundles don't load behave exactly as today. Log which backend actually loaded.
3. Gate GPU to E2B/E4B, never 12B — upstream Gemma 4 12B LiteRT-LM model fails to load on Android with LiteRT-LM 0.13.0 google-ai-edge/LiteRT-LM#2461 shows Gemma 4 12B still fails to load on GPU in 0.13.0.
4. (Optional) surface the chosen backend via EngineDescriptor / a setting so the UI can show "Running on GPU/CPU" and users can force CPU.

Notes

• This is fundamentally an on-device verification task — GPU bundle loading can only be confirmed on real Adreno hardware (benchmarked target: CPH2723 / Snapdragon 8 Elite, the same device Phase-0 used). CI can't prove it.
• Stays within the existing LiteRT-LM dependency — no new runtime, zero-telemetry posture intact. The llama.cpp OpenCL path remains the heavier fallback only if GPU bundles still won't load on 0.13.1.

Acceptance criteria from #37 merged into this issue's list:

• LiteRT-LM bumped to 0.13.1 and the project builds.
• Engine attempts GPU and falls back to CPU on init failure (no regression).
• Verified on-device on CPH2723: GPU loads, generation works, GPU utilization observable during decode.
• 12B stays on CPU.
• Decode tok/s recorded for CPU vs GPU on the same device + bundle.

---

Generated by Claude Code