Investigate incorrect LiteRT Qwen3 arithmetic output on Android and iOS

[leehack]

The physical-device Firebase pilot now produces visible LiteRT-LM Qwen3 chat content, but answers 2 to What is 2+2? Answer only with the number. on Android CPU, iOS CPU and iOS GPU. Track the semantic failure separately from the earlier empty-content issue #505 and from native byte-level output decoding in leehack/litert-lm-native#48.

This is an investigation, not a confirmed llamadart regression. A small converted model can answer incorrectly; matching native/model-reference controls are required before assigning a defect to the Dart layer, runtime, tokenizer/template or model conversion.

Exact inputs

• 2026-09-16, llamadart main a5df1c4fcbb1766d26efb5b1d9becda191df89a3 (includes Fix LiteRT Qwen3 thinking control and native system messages #507), LiteRT-LM artifact v0.17.0-3.
• Flutter 3.47.1 / Dart 3.13.1. Runtime and parser source unmodified; Flutter integration-test wrapper only.
• Android: physical Galaxy S24 SC-51E, API 36, build BP2A.250605.031.A3.SC51EOMS1CZB4, Debug app.
• iOS: physical iPhone 16 Pro, actual iOS 18.3.2 / 22D82 (catalog 18.3), Release app. Native GPU logs show Metal initialization.
• Model: Qwen3-0.6B.litertlm at pinned revision, 614,236,160 bytes; on-device SHA256 verified as 555579ff2f4fd13379abe69c1c3ab5200f7338bc92471557f1d6614a6e5ab0b4.

Using a local path to the model and package:llamadart/llamadart.dart:

final engine = LlamaEngine(LlamaBackend());
await engine.setNativeLogLevel(LlamaLogLevel.info);
await engine.loadModel(modelPath, modelParams: const ModelParams(
  contextSize: 1024,
  numberOfThreads: 4,
  numberOfThreadsBatch: 0,
  gpuLayers: 0,
  preferredBackend: GpuBackend.cpu,
  liteRtLmBackend: LiteRtLmBackendPreference.cpu,
));
final chunks = await engine.create(
  [const LlamaChatMessage.fromText(
    role: LlamaChatRole.user,
    text: 'What is 2+2? Answer only with the number.',
  )],
  params: const GenerationParams(maxTokens: 32, temp: 0, seed: 1),
  enableThinking: false,
).toList();
final content = chunks
    .map((c) => c.choices.first.delta.content ?? '').join().trim();
print(content); // observed: 2; diagnostic oracle: ^4[.!]?$
await engine.dispose();

For the iOS GPU case: gpuLayers: ModelParams.maxGpuLayers, preferredBackend: GpuBackend.metal, liteRtLmBackend: LiteRtLmBackendPreference.gpu. Other sampling settings retained the exact source revision's defaults. Only one arithmetic generation per backend was observed; do not infer a failure rate.

Recorded output

[
  {"device":"SC-51E","backend":"cpu","output":"2","expected":"4"},
  {"device":"iphone16pro","backend":"cpu","output":"2","expected":"4"},
  {"device":"iphone16pro","backend":"gpu/Metal","output":"2","expected":"4"}
]

Corrected Android execution and corrected iOS execution require project access; the essential inputs/output are reproduced above. Android GPU produced a separate incoherent response tracked in leehack/litert-lm-native#51.

An earlier pilot incorrectly supplied llama.cpp-only numberOfThreadsBatch: 4 to LiteRT. Those typed configuration rejections are excluded from this report; the values above were corrected before these runs. Timing assertions were after the arithmetic assertion, so LiteRT TPS was not collected.

Acceptance

• Run the identical pinned model and prompt through direct native LiteRT conversation and public Dart on the same device. Capture exact serialized messages, rendered template/template options, token IDs where exposed, text/thinking/tool deltas, finish reason and effective sampler settings.
• Compare CPU/GPU and a model-author/reference inference path; distinguish conversion/tokenizer/model behavior from integration behavior. Keep related fix(chat_app): honor backend switch and context auto mode #48 findings separate unless causally linked.
• Include hello, arithmetic and whitespace/non-ASCII cases; qualify an appropriate semantic oracle against the reference model rather than silently weakening it to nonempty output.
• If the wrapper/runtime is at fault, fix the owning layer and add positive/negative regression coverage. If the same model fails the reference, document that result and select a reference-qualified semantic fixture while retaining this diagnostic case and provenance.
• Continue independent performance collection after a semantic failure when the process is healthy; correctness remains FAIL and failed-output timings are excluded from passing baselines.