Fix Unicode corruption in native GGUF detokenization

[leehack]

Summary

Native LlamaEngine.detokenize corrupts non-ASCII text because its Int8 buffer exposes UTF-8 bytes above 127 as negative values. Read the same native buffer as Uint8 before joining and decoding token pieces. Montréal 👋\n한글 café now survives the public tokenize/detokenize path, including characters split across byte tokens.

Fixes #511.

Production-readiness scope

• User-facing scope: correct native GGUF detokenization for Unicode. No public signature, runtime pin, native ABI, or tokenizer normalization changes.
• Supported paths: shared native llama.cpp service. Real-model verification on macOS arm64 CPU and Metal; the new CPU regression runs in existing Linux, macOS, and Windows VM CI.
• Preserved behavior: empty token lists, special-token visibility, and replacement decoding of genuinely incomplete UTF-8 token sequences.
• Out of scope: LiteRT-LM semantic failures (Investigate incorrect LiteRT Qwen3 arithmetic output on Android and iOS #509, Investigate Gemma3 LiteRT-LM history failures on CPU and S24 NPU #513), additional validation-suite work (Complete cross-platform validation cases, device coverage and reports #514), and device qualification. The validation suite stays in Add portable validation suite and guarded Firebase/GCE runners #515.

Completeness checklist

• Declared runtime scope is implemented.
• No new unsupported combination or silent fallback.
• Both changelogs updated; existing API documentation, examples, and platform support remain accurate.
• Regression coverage includes the original failure and unaffected boundary behavior.
• No credentials, signed URLs, models, or native SDKs are committed.
• Separate follow-ups linked above.

Test Plan

• dart run tool/prepare_workspace.dart with Flutter 3.47.1 / Dart 3.13.1; no tracked dependency changes.
• dart format --output=none --set-exit-if-changed .
• dart analyze
• dart run tool/testing/check_platform_boundaries.dart
• Targeted tokenizer regression, inference smoke, and native service tests: 126 passed.
• New regression against unchanged main: 5 failed / 3 passed; all 8 pass with the fix.
• Full VM regression: 2,094 passed / 77 optional skips. Coverage: 78.71% (13,180/16,744 lines), above the 70% gate.
• ./tool/docs/validate_links.sh: docs build and link validation passed.
• Chrome runtime tests: PR CI passed; changed production code is VM-only.

Matrix Evidence

Matrix row	Platform / model / backend	Result	Evidence
root-vm	Linux, macOS, Windows CI / stories15M / CPU	PASS	Existing full VM lanes include the new eight-case public-engine regression
root-vm targeted	macOS arm64 / stories15M / CPU	PASS	Exact ASCII, accented Latin, emoji, Korean, mixed text, cross-token UTF-8, special and empty-input assertions
macos-arm64-runtime-smoke	macOS 26.6.2 arm64 / stories15M / CPU + Metal	PASS	5 exact round-trips per backend; explicit one-space SentencePiece prefix
macos-arm64-runtime-smoke	macOS 26.6.2 arm64 / Qwen3.5-0.8B-Q4_0 / CPU + Metal	PASS	5 exact round-trips per backend; no added prefix

Native artifact: v0.4.0. Both CPU and Metal requests reported the requested backend. Model SHA256: stories15M 61b50d457809a5194818fd22e6724b456cd7bb9a6264c52c8110684c53f3704a; Qwen3.5 57d1997790d1744fba5b40a7317df71ea5e2acee28c47e78f0cce39c0703f8cf. This is tokenizer correctness evidence, not an inference-throughput comparison. No paid cloud execution.

CI result

All 13 checks passed on 719ec4eba1e94953d29b7085c6ee6014f760a351: Linux VM/coverage, macOS and Windows native suites, Chrome, Web chat contract, analysis, docs, companion packages, prompt reuse, preview, advisory and the final test aggregate. CI run. The PR is mergeable with zero unresolved review threads; the independent runtime review is accepted with zero blocking findings.

High-risk regression review

• Classification: high-risk / backendRuntime, per repository classifier.
• Implementation task: Unicode fix in fix/native-detokenization-utf8.
• Independent blocking QA task: accepted by fresh independent agent audit_pr516, auditor codex-adversarial:pr516:20260917:audit-pr516; 126 targeted tests and 20 real-model checks passed independently. Signed-byte and malformed-policy mutations failed as expected, then were restored.
• Exact head / current base: 719ec4eba1e94953d29b7085c6ee6014f760a351 / 21135e37dadf60882ea427db6078ccc90f84a28a.
• Production-branch deletion, bypass, or miswire proof: reverting the unsigned-byte change reproduces the 5 failing public-path tests. Empty/ASCII/special controls pass before and after; split-byte reconstruction passes only after the fix.
• Affected-family evidence: two actual GGUF vocabularies, both CPU and Metal, 20/20 exact round-trips after the fix.
• Unavailable evidence: Android/iOS physical-device execution is not claimed. No structured-output or template branch changed.
• Known PR-caused P1 regressions: 0, independently reviewed against the exact head/base below.
• Unresolved review threads: 0 at the recorded head; refresh required before mark-ready.

The repository-local evaluator accepted evidence consistency and returned the documented exit 2 (unverifiedPrerequisites): external App/authentication/ruleset enforcement is intentionally unconfigured and is not a pending manual merge requirement.

Independent readiness evidence

{
  "schema": "llamadart.high-risk-readiness-evidence",
  "schema_version": "1.0.0",
  "timestamp": "2026-09-17T21:32:06Z",
  "correlation_id": "pr516-independent-audit-20260917",
  "repository": "leehack/llamadart",
  "pr_number": 516,
  "expected_pr_head_sha": "719ec4eba1e94953d29b7085c6ee6014f760a351",
  "current_base_sha": "21135e37dadf60882ea427db6078ccc90f84a28a",
  "pr_author": "leehack",
  "classification": "high-risk",
  "surfaces": [
    "backendRuntime"
  ],
  "required_matrix_row_ids": [
    "high-risk-exact-head-independent-qa"
  ],
  "matrix_row_evidence": {
    "high-risk-exact-head-independent-qa": {
      "row_id": "high-risk-exact-head-independent-qa",
      "result": "pass",
      "command": "dart test -p vm -j 1 --reporter expanded test/integration/backends/llama_cpp/tokenizer_integration_test.dart test/integration/inference_smoke_test.dart test/unit/backends/llama_cpp/llama_cpp_service_test.dart",
      "evidence_notes": "Accepted independent blocking-only audit by fresh agent /root/audit_pr516. Exact GitHub head/base verified separately. Production engine/router/worker/service path inspected. Exact-head tokenizer 8/8 and targeted tokenizer/inference/service 126/126 pass. Isolated Int8 mutation causes 5 Unicode failures with 3 controls passing; isolated strict UTF-8 mutation causes only the incomplete-byte test to fail. Restored source is clean. Two verified GGUFs on CPU and Metal produce 20/20 exact public-engine round-trips. All 13 exact-head CI checks success, zero unresolved review threads, zero known PR-caused P1 regressions. Full report and logs are in .dart_tool/issue-511/audit/."
    }
  },
  "independent_audit": {
    "auditor_identity": "codex-adversarial:pr516:20260917:audit-pr516",
    "audit_kind": "codex-adversarial",
    "audit_head_sha": "719ec4eba1e94953d29b7085c6ee6014f760a351",
    "audit_base_sha": "21135e37dadf60882ea427db6078ccc90f84a28a",
    "decision": "accepted",
    "unresolved_review_threads": 0,
    "known_pr_caused_p1_regressions": 0,
    "summary": "Accepted independent blocking-only audit by fresh agent /root/audit_pr516. Exact GitHub head/base verified separately. Production engine/router/worker/service path inspected. Exact-head tokenizer 8/8 and targeted tokenizer/inference/service 126/126 pass. Isolated Int8 mutation causes 5 Unicode failures with 3 controls passing; isolated strict UTF-8 mutation causes only the incomplete-byte test to fail. Restored source is clean. Two verified GGUFs on CPU and Metal produce 20/20 exact public-engine round-trips. All 13 exact-head CI checks success, zero unresolved review threads, zero known PR-caused P1 regressions. Full report and logs are in .dart_tool/issue-511/audit/."
  },
  "structured_output_evidence": null,
  "affected_test_paths": [
    "test/integration/backends/llama_cpp/tokenizer_integration_test.dart"
  ],
  "evaluation": null
}

[github-actions]

Chat app preview removed for leehack/llamadart-chat-pr-516.