LLM Bench — On-device vs Cloud LLM in Flutter

Reference implementation and measurement harness for the MSc thesis "On-device versus Cloud LLM Integration in Flutter Mobile Applications: A Comparative Empirical Study via a Reference Chatbot Implementation."

A single Flutter chatbot runs four language models behind one common interface and measures them identically, so on-device and cloud deployments can be compared under the same workload.

Systems under comparison

Deployment	Model	How
Cloud	Claude Haiku 4.5 (claude-haiku-4-5)	Anthropic Messages API (raw HTTP + SSE)
Cloud	Claude Sonnet 4.6 (claude-sonnet-4-6)	same code path, different model
On-device	Gemma 3 1B (int4, .task)	flutter_gemma (MediaPipe / LiteRT)
On-device	Gemma 3 270M (q8, .task)	same code path, smaller model

Single-provider cloud (both Anthropic) is a deliberate design choice: it holds the API, tokenizer, and infrastructure constant so the Haiku↔Sonnet difference isolates model capability rather than provider effects.

Metrics (RQ1 performance, RQ2 quality)

Every generation is timed by one shared MetricsRecorder, regardless of deployment:

• Latency — time to first token (TTFT) and total time to last token
• Throughput — output tokens/second (decode window)
• Peak memory — process phys_footprint sampled during generation (iOS native)
• Thermal state — ProcessInfo.thermalState at end of run (iOS native)
• Cost — per-query USD from token counts × model pricing (0 for on-device)
• MMLU accuracy — auto-scored multiple-choice
• MT-Bench quality — 1–10 LLM-as-judge (single-answer grading)

Architecture

lib/
  core/
    llm/          LlmProvider abstraction + request/event/usage models
    telemetry/    MetricsRecorder, RunMetrics, NativeMetrics, pricing
    config/       AppConfig (.env), OnDeviceModelSpec
  data/providers/ CloudClaudeProvider (SSE + retry), OnDeviceGemmaProvider (flutter_gemma)
  features/
    chat/         reference chatbot UI with live per-message telemetry
    benchmark/    headless runner (prompt × provider × repeats), MMLU scorer,
                  MT-Bench judge, JSON/CSV exporter, control panel
ios/Runner/       AppDelegate.swift MethodChannel `llm_bench/native` (memory + thermal)
assets/prompts/   mmlu.json (50) + mtbench.json (20)

Setup

1. Copy .env.example to .env and fill in:
   • ANTHROPIC_API_KEY — from the Anthropic Console (prepaid credits; separate from any Claude subscription)
   • HF_TOKEN, GEMMA_MODEL_URL, GEMMA_MODEL_FILE — a small quantized Gemma .task model from HuggingFace (accept the Gemma license first)
   • JUDGE_MODEL — MT-Bench judge (default claude-opus-4-8; leave empty to disable)
2. flutter pub get
3. Run on a physical iOS device — the on-device stack (MediaPipe / TFLite) has no iOS-simulator binaries.

flutter run -d <device-id>

iOS requirements are already configured: minimum iOS 16, static pod linking, file sharing, and the increased-memory-limit entitlement.

Usage

• Chat: pick a model, send a prompt, read the per-message telemetry chips. For on-device, tap Download & load once to fetch and load the model.
• Benchmark: the Benchmark button opens the control panel. Select models, set repeats, Run benchmark. Watch the live per-provider summary, then Export & Share (JSON + CSV via the iOS share sheet) or Save only.

Datasets

• MMLU — 50 items across 10 subjects, from cais/mmlu (test split).
• MT-Bench — 20 turn-1 prompts across all 8 categories, from lm-sys/FastChat.

Known limitations (thesis scope)

• Cloud comparison is single-provider (Anthropic) by design — see above.
• On-device Gemma 3 1B int4 runs on CPU/XNNPACK on iOS (no GPU slice for this file); the actual backend is recorded.
• Native memory/thermal sampling is iOS-only; other platforms report null.
• MT-Bench uses a single judge (self-preference mitigated by using a model distinct from those under test).
• Statistical care: modest sample sizes — report confidence intervals; the raw per-run dataset is exported for offline analysis.