litert_dart #

On-device LLM inference in pure Dart (dart:ffi), no Flutter. Load a LiteRT-LM model (.litertlm) — Gemma, Qwen, DeepSeek… — and generate text 100% locally, no cloud.

import 'package:litert_dart/litert_dart.dart';

final model = await LiteRtModel.load('/path/gemma-4-E2B-it.litertlm');
print(await model.ask('What is 2 + 2? Answer briefly.'));
await model.close();

Why it exists #

flutter_gemma is a Flutter plugin: it requires the Flutter engine (a window) and bundles the native binaries inside a .app. That makes it unusable from a plain dart run (a CLI, an HTTP server, a worker).

litert_dart takes the FFI layer (bindings + client, derived from flutter_gemma_litertlm, MIT) and runs it without Flutter, against the official LiteRT-LM native binaries. Same engine, no Flutter, no UI.

The inference is done by the native LiteRT-LM (C++) engine. Dart only orchestrates over FFI — it does not change generation speed.

Install #

In the consuming project's pubspec.yaml:

dependencies:
  litert_dart: ^0.1.0

Then, once, download the native binaries into your project:

dart pub get
dart run litert_dart:fetch_natives   # → ./native/<platform>/

fetch_natives downloads the platform archive from this project's own GitHub release (JhonaCodes/litert_dart, tag native-v0.13.1), verifies SHA256, and extracts the libraries into ./native/<platform>/. They are not committed (they go in .gitignore). Hosting them on our own release means the package does not depend on any third party for the binaries.

Models #

• Supported: .litertlm files (LiteRT-LM format), e.g. gemma-4-E2B-it.litertlm, Qwen2.5-1.5B-...litertlm, etc.
• Not supported: .gguf (llama.cpp/ollama), .task/.bin (MediaPipe).

API #

Calls are serialized (the engine allows one live conversation at a time). Each call uses a fresh conversation (no context carried over between prompts); for conversational memory, build the history into the prompt yourself.

Backend: CPU vs GPU #

Default is CPU (always available). The GPU (Metal on macOS) backend does not load in a pure-Dart process: LiteRT-LM looks up the accelerator as a .framework under @executable_path/../Frameworks/…, a path that only exists inside a Flutter .app. Enabling GPU would require placing that framework on the loader's search path (roadmap).

Native library resolution #

nativeLibDir() (synchronous) searches in this order:

1. LITERTLM_LIB_DIR env var (always wins).
2. ./native/<platform> — what fetch_natives populates.
3. The flutter_gemma cache (~/Library/Caches/flutter_gemma/native/…), if present.

Platforms: macos_arm64, linux_x86_64, linux_arm64.

Logs / noise #

The native engine writes diagnostics (absl/glog) to stderr. litert_dart does NOT hijack your file descriptors: if the noise bothers you, run with 2>/dev/null or route stderr to your logger. Our own logs are off by default; raise them with LITERT_DART_VERBOSE=1 (or 2).

Compiling to an executable #

dart compile exe bin/your_server.dart -o build/your_server
LITERTLM_LIB_DIR=/path/native/<platform> ./build/your_server

Under AOT there is no package-relative resolution: pass LITERTLM_LIB_DIR pointing at the directory with the libraries (or keep ./native/<platform> next to the binary).

Status and credits #

Status: working (verified with Gemma-4 E2B on macOS arm64, CPU). Linux is wired and analyzer-clean; run it on a real Linux host to confirm.

The FFI layer is derived from flutter_gemma (MIT, © Sasha Denisov), adapted to run without Flutter. The prebuilt native libraries are built from LiteRT-LM (Apache-2.0) and re-hosted on this project's releases. See LICENSE.

Architecture and decisions: doc/architecture.md.