flutter_local_llm #

A high-performance, production-grade Flutter plugin for on-device local Large Language Model (LLM) inference using llama.cpp via Dart FFI and Dart's Native Assets system (hooks/build.dart).

Key Features #

• ⚡ Hardware Acceleration:
  • iOS / macOS: Apple Silicon GPU acceleration via Metal (GGML_USE_METAL).
  • Android: NDK with OpenMP / Vulkan backend options.
  • Desktop (macOS / Linux / Windows): Native multi-threading, AVX2, and Vulkan backends.
• 🚀 Non-Blocking Token Streaming: Direct async token delivery from native worker threads to Dart StreamController<String> via NativeCallable<TokenCallbackNative>.listener with zero main UI thread stutter.
• 💬 Conversational Context Management: Multi-turn chat session with automatic KV-cache sliding window truncation to preserve context budget without exceeding limits (n_ctx).
• 📝 Pre-Built Chat Templates: Built-in formatters for ChatML (SmolLM, Qwen), Llama-3, Gemma, Mistral, and customizable templates.
• 🎯 Structured Outputs (GBNF Grammar): Grammar constraints enforcing strict JSON output adherence to JSON schemas.
• 📥 Robust Resumable Model Downloader: Background downloader for GGUF models from Hugging Face or custom URLs with HTTP Range resume, progress/speed/ETA streams, and SHA-256 integrity verification.
• 🛠️ Dual Build Pipeline: Supports modern Dart Native Assets (hooks/build.dart) as well as standard Flutter plugin build toolchains (CMake and CocoaPods).

Installation #

Add flutter_local_llm to your pubspec.yaml:

dependencies:
  flutter_local_llm: ^0.1.0

Quick Start #

1. Download a GGUF Model #

import 'package:flutter_local_llm/flutter_local_llm.dart';

final downloader = ModelDownloader();
final downloadStream = downloader.download(
  url: 'https://huggingface.co/HuggingFaceTB/SmolLM-135M-Instruct-GGUF/resolve/main/smollm-135m-instruct-q4_k_m.gguf',
  destinationPath: '/path/to/local/smollm-135m.gguf',
);

downloadStream.listen((progress) {
  print('Progress: ${(progress.progress * 100).toStringAsFixed(1)}% '
        'Speed: ${progress.speedFormatted}');
});

2. Initialize the Engine & Session #

// 1. Load the model with GPU layer offload and context window
final engine = await LocalLlmEngine.loadModel(
  modelPath: '/path/to/local/smollm-135m.gguf',
  params: const ModelParams(
    contextSize: 2048,
    gpuLayers: 99, // 99 offloads all layers to Metal / Vulkan
  ),
);

// 2. Create an interactive chat session
final session = engine.createSession(
  defaultTemplate: const ChatMlTemplate(),
);

// 3. Stream a multi-turn chat response
final stream = session.chat(
  [
    ChatMessage.system('You are a helpful coding assistant.'),
    ChatMessage.user('How do I create a stream in Dart?'),
  ],
  params: const SamplingParams(
    temperature: 0.7,
    topP: 0.9,
    maxTokens: 512,
  ),
  onMetrics: (metrics) {
    print('Generation speed: ${metrics.tokensPerSecond.toStringAsFixed(1)} tok/s');
    print('Time to first token (TTFT): ${metrics.timeToFirstToken.inMilliseconds}ms');
  },
);

await for (final token in stream) {
  stdout.write(token);
}

// Clean up resources when done
session.dispose();
engine.dispose();

Structured Output with GBNF Grammar #

To constrain the model to output strict JSON according to a JSON Schema:

final userProfileSchema = {
  'type': 'object',
  'properties': {
    'name': {'type': 'string'},
    'age': {'type': 'integer'},
    'skills': {
      'type': 'array',
      'items': {'type': 'string'},
    },
    'role': {
      'type': 'string',
      'enum': ['engineer', 'designer', 'manager'],
    },
  },
  'required': ['name', 'role'],
};

final gbnfGrammar = GrammarHelper.jsonSchemaToGbnf(userProfileSchema);

final stream = session.promptStream(
  'Generate a JSON profile for a Senior Flutter Developer named Alice.',
  jsonSchemaGrammar: gbnfGrammar,
);

await for (final chunk in stream) {
  stdout.write(chunk);
}

Supported Chat Templates #

Architecture Overview #

flutter_local_llm/
├── hooks/
│   └── build.dart                 # Dart Native Assets CLI hook (code_assets/cbuilder)
├── native/
│   ├── CMakeLists.txt             # Cross-platform CMake build configuration
│   ├── llama_wrapper.h            # Minimal C ABI export signatures
│   └── llama_wrapper.cpp          # High-performance C++ worker & llama.cpp engine bridge
├── lib/
│   ├── flutter_local_llm.dart     # Public umbrella export
│   └── src/
│       ├── ffi/bindings.dart      # Native bindings with NativeCallable.listener
│       ├── core/
│       │   ├── engine.dart        # LocalLlmEngine lifecycle & hardware control
│       │   ├── session.dart       # LlmSession with sliding window truncation
│       │   └── models.dart        # ChatMessage, ModelParams, SamplingParams
│       ├── templates/             # ChatML, Llama-3, Gemma, Mistral templates
│       └── utils/
│           ├── model_downloader.dart # Resumable chunked downloader & SHA-256
│           └── grammar_helper.dart   # JSON Schema to GBNF converter
├── example/                       # Complete Flutter Chat UI + Model Hub HUD
└── test/                          # Comprehensive unit & mock test suites

License #

MIT License.