On-device Gemma in Flutter: flutter_gemma and LiteRT-LM
Cloud AI is not the only path. On-device models keep data local, work offline, and cut per-request cost. Flutter’s story here is flutter_gemma plus Google’s LiteRT-LM inference runtime.
Why LiteRT-LM matters
It abstracts hardware differences and aims at GPU/NPU acceleration across Android, iOS, Web, Windows, Linux, and macOS — the same six targets Flutter ships.
Product patterns that work
- Vision assist — camera frames → short structured descriptions (Gemma Vision style).
- Task coaching — local multi-step planning without a round trip.
- Privacy-sensitive dictation/notes — text never leaves the device.
Engineering checklist
- Model download UX (multi-hundred MB) with resume.
- Thermal/battery budgets on mid-tier phones.
- Fallback path when NPU/driver is missing.
- Clear user messaging that processing is local.
Pitfalls
- Do not block the UI isolate on token generation.
- Quantization tradeoffs: measure quality on your tasks, not benchmarks alone.