Investigate Gemma3 LiteRT-LM history failures on CPU and S24 NPU

[leehack]

Triage update — 2026-09-18

PR #507 has now merged the native system-content serialization correction. Earlier statements below that the integration fix is only local/unpublished are historical. Keep this investigation open at P1 because the recorded exact-history failures remain, including direct-native controls. A merged serialization fix does not establish semantic qualification or identify the remaining owning layer.

---

Current status (2026-09-17, after original-model and tokenizer controls): The system-content integration fix is locally committed. The original Gemma model passes all four strict history variants in three repetitions (12/12), while the LiteRT CPU public/native paths retain the failures below. The CPU tokenizer file is byte-identical and rendered prompt/token IDs match after accounting for BOS. Quantization/conversion versus runtime execution remains unresolved; this is not an exclusively Dart or NPU failure. Keep the exact cedar17 oracle and this issue open. No fix is pushed or merged. Earlier observations remain historical evidence.

A physical Galaxy S24 Firebase run through the public llamadart API generated 32 repeated 7 characters instead of recalling cedar17 from conversation history. Keep the history case failed: successful NPU execution and good throughput do not qualify semantic correctness.

This is an investigation, not a confirmed Dart-only or upstream regression. The direct-native control passed its single-turn cases but did not exercise this history input, so it does not yet isolate the failing layer.

Observed result

• 2026-09-17, Galaxy S24 SC-51E, actual SoC SM8650, Android API 36, arm64-v8a, Debug app.
• Clean validation bundle source 2288eed19c482e7774df2c93b1e7d9a96e6e9c4e; Flutter 3.47.1. Harness work is currently local; the essential repro inputs are reproduced here.
• LiteRT-LM artifact 0.17.0-3; QAIRT 2.47.0.260601; dispatch ABI matched to LiteRT revision 9fe5be45564c868408e6514c8aabb83e211a0911.
• Model: Gemma3-1B-IT_q4_ekv1280_sm8650.litertlm, 690,094,080 bytes, pinned model revision, SHA256 85d2ea5199802f913818d53897b3a304bcf983abb993393e6b1749fbdb005552 verified in the APK and on device.
• Public run qa-1789667821365130, matrix matrix-21m98jmyipzo2: 13 PASS, 1 FAIL (C06.history). Results collected and terminal completion verified.
• C06.history observed 33 completed synchronous vendor calls, no failed/in-flight/async-only proof. This establishes NPU participation; CPU partition coverage is unknown.
• Load, Unicode tokenize/detokenize, raw generation, hello, arithmetic, cancellation/recovery, reload, one-token limit, invalid-path recovery and short benchmarks passed. Median native decode throughput was 82.27 tokens/s across three short benchmark samples after warmup; this is not a passing correctness baseline for the whole profile.

Exact public input

Use LlamaEngine with the staged model, NPU backend, the verified Qualcomm dispatch directory, context 1280, four threads, max output 32 tokens and enableThinking: true:

[
  {"role":"system","content":"Remember the secret code exactly."},
  {"role":"user","content":"The secret code is cedar17."},
  {"role":"assistant","content":"I will remember the code."},
  {"role":"user","content":"What is the secret code? Reply with only the code."}
]

Expected trimmed output: cedar17.

Observed output: 77777777777777777777777777777777.

The runtime recorded 66 prompt tokens and 32 decoded tokens; five public stream chunks, empty thinking, finish reason stop. The harness requested temperature 0, seed 1, top-k 40 and top-p 0.9, but the NPU path intentionally does not apply session sampler overrides. Effective NPU sampling is the compiled model/runtime default and remains unknown; do not claim this was greedy/seeded execution. Only one history generation was observed; no failure rate is established.

The independent native C API control (qa-1789667329901788, matrix-pt34kzd9mgxsa) passed eight cases and verified NPU participation. Its hello/arithmetic/reload/benchmark coverage excludes history and system-message seeding.

Investigation and acceptance

• Extend the direct C API control to replay these identical system/history messages on the same model/device. Compare history-free, single combined-prompt and seeded-conversation controls; retain their exact wire JSON and rendered preface where available.
• Inspect both system-message serialization and separate preface/history prefilling. In the tested Dart source, _nativeConversationSeed serializes a role/content object and createConversation JSON-encodes the resulting string again. The pinned native C API parses its system-message argument into the content of a new system message. This is a candidate integration mismatch, not yet established as the cause of repeated-token output.
• Run the compatible Gemma CPU semantic control. The separate Qwen CPU retry cannot isolate this Gemma NPU behavior.
• Attribute any fix to the owning layer only after the controls distinguish Dart serialization, upstream conversation/prefill, NPU model conversion and model behavior. Add positive and negative regression coverage and retain the original failure.
• Keep this NPU profile unqualified until the history obligation passes. Do not weaken its oracle to nonempty output.

The raw local result includes prompts, output, effective configuration, model/library hashes, device identity and per-generation counters. Provider results require project access. Related Qwen arithmetic investigation #509 uses a different model and input and is not a duplicate or an established common cause.

Direct-native replay on latest main / v0.17.0-5 (2026-09-17)

The control now exercises the history inputs. Validation source fb8d2beb4db65f1b8987d69022049b4d0ae4fbda includes main 21135e37dadf60882ea427db6078ccc90f84a28a and its merged runtime update #510. The harness commits remain local. Same Gemma model hash, QAIRT 2.47 kit, S24 SC-51E/SM8650/API 36, Debug build, context 1280, four threads and max output 32. Compiled NPU sampling defaults remain unknown; no greedy/seeded claim.

Firebase native replay: run qa-1789670325547288, matrix matrix-1serhazjregr5. 8 PASS, 4 FAIL, 0 ERROR/NOT_RUN. Completed synchronous vendor calls verify NPU participation in all 11 generations; CPU partition coverage remains unknown. Result collection is COMPLETE and terminal cleanup VERIFIED. Test process duration was 46 seconds, one rounded free minute.

Direct C API control	Actual output	Strict cedar17 oracle
Normal system content + prior user/assistant messages	Cedar17	FAIL: uppercase C
Same history, system content reproducing public literal JSON	77777777777777777777777777777777	FAIL
Prior user/assistant messages without a system message	Cedar17	FAIL: uppercase C
All four text contents joined by newlines into one user prompt, no preface	7777\n	FAIL

The literal-system control passes these exact bytes to litert_lm_conversation_config_set_system_message:

"{\"role\":\"system\",\"content\":[{\"type\":\"text\",\"text\":\"Remember the secret code exactly.\"}]}"

The normal system control instead passes a JSON string containing only Remember the secret code exactly.. Both seed the same two prior user/assistant messages and send the same final user question. Both also set extra context {"enable_thinking":true} and constrained decoding false, matching the public settings. The control records all actual setter/send JSON, the native response and before/after vendor counters. No native vendor calls were observed during conversation creation in these four cases; successful work was counted during send/generation, so this run does not establish a separate initialization-prefill failure.

The v0.17.0-5 core library SHA256 is 502d017d8375c0796adf5f720da29fb1915b2a70103e3cf8f6e8e8880ac0f614; Android NDK 28.2.13676358 llvm-strip --strip-unneeded exactly reproduces packaged SHA256 5424262e8a396b1cc32813a5d3f061c0ccba06cdff2ddc61d5bfdc40a249d2c4. Its published manifest retains upstream commit e9fd8c53ff968071774206163027dd84bedfe925. The required C API symbols and all staged model/kit hashes were checked.

Interpretation and next controls

• Repeated 7 output reproduces through the direct native C API on the latest runtime when given the public system-message representation. It does not require Dart streaming or worker bindings.
• Source inspection establishes a system-content contract mismatch: the current service serializes a role/content object, and the runtime JSON-encodes it as a string, while the upstream setter consumer treats the parsed value as system content. Fix that boundary with positive and negative regression tests, then replay the public path.
• Changing to normal content removed the repeated-token degeneration in this observed comparison, but it did not satisfy exact recall: capitalization still failed, and the combined single-prompt control also failed. Do not weaken the oracle or claim serialization is the only cause.
• Only one generation per history variant was observed. Repeat controlled comparisons and add a compatible Gemma CPU/model-reference control before attributing the remaining behavior to the NPU runtime versus model conversion or model behavior. Latest public API execution has not been rerun on v0.17.0-5.
• Hello, arithmetic, reload and the existing short benchmark obligations passed. Three measured benchmark samples had median native decode 80.58 TPS, estimated wall 71.71 TPS, and native TTFT 37.05 ms; these do not qualify the failed history cases.

The newer main changes repair desktop GPU loading. The Qwen tokenizer repair is a distinct model artifact, not an automatic replacement of the original Qwen fixture or a Gemma NPU history fix. Preserve the earlier v0.17.0-3 failures alongside this result.

Public replay after correcting system-content serialization (2026-09-17)

The service now supplies joined plain system text to LiteRtLmRuntimeClient.createConversation; the runtime performs the single JSON-string encoding expected by the C API. Empty system messages remain omitted, multiple instructions retain their newline join, literal JSON stays literal text, prior history order is unchanged, and unsupported system media remains rejected.

Three regression assertions failed on the old service. All 161 LiteRT VM tests now pass, including literal JSON/quotes/newlines/Unicode, multiple/absent/empty system instructions, existing history-order/media guards, and the compiled C fixture checking actual runtime JSON encoding. Full repository dart analyze, changed-file formatting and diff checks pass. Coverage reaches every executable line in the history-seeding function.

Clean validation source d0bd029b07e31649ab5760756dcc1841ba487282 includes main 21135e37dadf60882ea427db6078ccc90f84a28a. Public Firebase retest: run qa-1789672372212252, matrix matrix-23gzng6la12qr. Same model SHA256, QAIRT kit, S24/API 36, Debug build, context 1280, four threads, max 32 output tokens and thinking enabled as the canonical native control. Runtime 0.17.0-5 is verified against packaged core SHA256 5424262e8a396b1cc32813a5d3f061c0ccba06cdff2ddc61d5bfdc40a249d2c4.

13 PASS, 1 FAIL, 0 ERROR/NOT_RUN. C06.history returned Cedar17, not repeated 7 characters. It remains failed against the unchanged exact lowercase cedar17 oracle. Recorded history metrics: 52 prompt tokens, four decoded tokens, two public stream chunks, empty thinking, finish reason stop, five completed synchronous vendor calls. NPU participation is verified across all fourteen generations; CPU partition coverage remains unknown. Provenance is complete, result collection COMPLETE and terminal cleanup VERIFIED.

The one device execution took 46 seconds / one rounded free minute, within the verified free allowance. No repeat was dispatched. The three short measured benchmark samples after warmup had median native decode 77.84 TPS, estimated wall 66.13 TPS, and time to first public output 152.17 ms; native TTFT is unavailable on the public path. Benchmark assertions cover bounded nonempty output, not number-list semantics, and these samples do not establish a performance regression or qualify failed history behavior.

Remaining obligation

The serialization correction removes the repeated-token degeneration in this observed public request, but exact recall still fails in both public and canonical native paths. Effective NPU sampling remains the unknown compiled model/runtime default; requested temperature/seed are not applied. Retain the native combined-prompt failure too. Compare a compatible Gemma CPU/model reference and repeated controlled history inputs before attributing the residual behavior to model quality, conversion or the NPU runtime. Do not weaken the oracle or claim S24 semantic qualification. Local source commits remain unpublished.

Repeated CPU and direct-native controls (2026-09-17)

Added a separate locked gemma3-litert-cpu profile in clean local source 2b604775ac49f0532e2a0a9f34024c93d3998c10. It extends the public quick core with the same four history diagnostics, retaining the exact cedar17 oracle. Existing core profiles keep their original case inventories/defaults. Local CPU controls consume no Firebase minutes.

• Apple M4 Max, macOS 26.6.2 arm64, Dart 3.13.1 JIT; LiteRT-LM 0.17.0-5.
• Same pinned model repository revision, CPU file gemma3-1b-it-int4.litertlm, 584,417,280 bytes, SHA256 1325ae366d31950f137c9c357b9fa89448b176d76998180c08ceaca78bba98be. This differs from the NPU compiled artifact; weights/conversion equivalence is not established.
• Mac core library SHA256 42a1fa7cc0666ceda1bb00f864065e7b6bcae041234bd501ecc1c56de54b7530, matching the cached release archive and published release manifest's macOS smoke record.
• Context 1280, four threads, max output 32, thinking enabled, constrained/speculative decoding disabled. CPU uses greedy decoding: public temperature zero/top-k 40 resolves to top-k 1 in the service. The direct C API explicitly used TopP sampler type 2, top-k 1, top-p .9, temperature zero and seed 1. NPU effective sampling remains unknown compiled defaults.

Each of three fresh public-API processes completed 13 PASS, 4 FAIL, 0 ERROR/NOT_RUN, with complete provenance. A separate Python/ctypes diagnostic binds the pinned upstream C API directly, loads the CPU model, and creates a fresh native engine per repetition and a fresh conversation per input. It imports no Dart/service/worker code and uses blocking native responses. The setter JSON is equal to the previous NPU control JSON. All twelve outputs match the public CPU results exactly:

Input	Public CPU, each of 3 runs	Direct-native CPU, each of 3 repetitions
Canonical system + prior history	Cedar17	Cedar17
Original literal-JSON system content	77777777777777777777777777777777	77777777777777777777777777777777
History without system	Cedar17	Cedar17
Combined user prompt	77777777777777777777777777777777	77777777777777777777777777777777

All four variants still FAIL exact cedar17. The earlier S24 combined native response was 7777\n, so do not claim every output is identical across CPU and NPU. The canonical and literal-system failures do reproduce across those environments.

Nine short public benchmark samples after three separate warmups had median native decode 74.06 TPS, estimated wall 67.46 TPS, and time to first public output 137.33 ms. These are Mac CPU diagnostic metrics, not a pure NPU speed comparison or semantic qualification. Different platform, model artifact/conversion, native context capacity and effective sampling prevent that inference.

Updated attribution and remaining work

• Remaining capitalization and repeated-token history behavior occurs without Dart and without NPU hardware. It is not an exclusively Qualcomm, NPU, or Dart streaming failure.
• The earlier service serialization correction is still required: canonical system content no longer sends a role/content JSON object as literal instructions. It does not guarantee exact model recall.
• Distinguish original Gemma model behavior from tokenizer/template/conversion differences and the shared LiteRT runtime using an original-model reference and rendered/tokenized prompt comparison. These CPU controls alone do not identify that final cause.
• The separate S24 CPU device row remains NOT_RUN. The gated CPU model returns HTTP 401 without authentication, so it needs verified private model transfer before any Firebase execution. Do not put access credentials or signed URLs in the bundle/profile.
• Keep Investigate Gemma3 LiteRT-LM history failures on CPU and S24 NPU #513 open and both history profiles unqualified; do not relax the strict oracle.

Reproduction using the local harness (commits remain unpublished):

dart run tool/testing/validation.dart local --profile gemma3-litert-cpu \
  --model /path/to/gemma3-1b-it-int4.litertlm \
  --out .dart_tool/validation/runs/gemma-cpu-control-1

Repeat into fresh directories. Exact JSON requests/responses, native diagnostic code, model/runtime hashes, stderr, per-case reports and three public manifests are preserved locally.

Original Gemma and prompt/tokenizer comparison — 2026-09-17

Original model: google/gemma-3-1b-it, revision dcc83ea841ab6100d6b47a070329e1ba4cf78752. All downloaded inputs verified against repository Git blobs/LFS hashes. The 1,999,811,208-byte safetensors file has SHA256 3d4ef8d71c14db7e448a09ebe891cfb6bf32c57a9b44499ae0d1c098e48516b6.

Local reference: Apple M4 Max, CPU float32, PyTorch 2.14.0, Transformers 5.17.0, Tokenizers 0.23.2, four threads, eager attention, greedy decoding, seed 1, maximum 32 new tokens. Used the original chat template and the exact four native-control role/content inputs, including the literal JSON system variant.

Variant	Original model, each of three repetitions	LiteRT CPU public/native controls
Canonical	cedar17\n — PASS	Cedar17 — FAIL
Literal-JSON system	cedar17\n — PASS	32 repeated 7 characters — FAIL
No system	cedar17\n — PASS	Cedar17 — FAIL
Combined user prompt	cedar17\n — PASS	32 repeated 7 characters — FAIL

Original reference: 12/12 PASS. Predicate remains trimmed and case-sensitive. This is a semantic control; different weight precision and inference implementations prevent treating its timing as a comparative benchmark.

Tokenizer and template evidence

• The CPU LiteRT file embeds a byte-identical copy of the original SentencePiece tokenizer: SHA256 1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c.
• For all four variants, litert_lm_conversation_render_message_to_string returns the same full prompt as the original template, excluding BOS. It already includes the supplied preface; do not concatenate render_preface a second time.
• The native tokenizer API output plus metadata BOS ID 2 equals the original Transformers input IDs for every variant. Pinned upstream runtime/core/session_utils.cc adds BOS separately on the first turn even when conversation-level templating disables session templates. This is rendering/tokenizer API evidence plus source inspection, not an executor tensor trace.
• The NPU tokenizer has 6,144 different vocabulary entries at IDs 256000–262143. Its normalizer and core special IDs match; none of the observed input IDs touches this range. This is not a demonstrated cause, and NPU runtime prompt tokenization remains unmeasured by this local control.
• Both LiteRT metadata sections use non-Jinja role affixes, BOS 2, and end-of-turn/EOS stop tokens. NPU metadata additionally specifies the 1280-token capacity.

Attribution and next acceptance step

The original reference passes the strict oracle; a mismatched CPU tokenizer file and different rendered CPU prompts do not explain the failures. Remaining candidates include conversion/quantization effects and LiteRT executor/runtime behavior. A controlled alternative conversion or upstream execution comparison is needed to distinguish them. No NPU-only, tokenizer-corruption or confirmed upstream-regression claim is justified yet.

Keep the public serialization fix, the strict failure evidence and the separate S24 CPU obligation. This entire comparison was local: no Firebase run, VM or paid resource. Original weights/tokenizer files remain private; exact rendered inputs, token IDs, outputs, dependency versions, hashes and native diagnostics are retained locally.