flutter_llama

Flutter plugin for running LLM inference with llama.cpp and GGUF models on Android, iOS, and macOS.

🎉 Version 1.0.0 - Production Ready with Full GPU Acceleration!

Features

• 🚀 High-performance LLM inference using llama.cpp
• 📱 Native support for Android, iOS, and macOS
• ⚡ Full GPU acceleration:
  • Metal for iOS/macOS (3-10x faster)
  • Vulkan for Android (4-8x faster)
  • OpenCL fallback (2-5x faster)
  • Automatic GPU detection and fallback
• 🔄 Streaming and blocking text generation
• 🎯 Full control over generation parameters
• 📦 GGUF model format (industry standard)
• 🛠 Easy to integrate and production-ready
• ✅ 71 unit tests, fully tested

Installation

Add this to your package's pubspec.yaml file:

dependencies:
  flutter_llama:
    path: ../flutter_llama  # Adjust path as needed

Then run:

flutter pub get

Usage

1. Load a Model

We recommend using the braindler model from Ollama - a compact and efficient model perfect for mobile devices.

import 'package:flutter_llama/flutter_llama.dart';

final llama = FlutterLlama.instance;

// Using braindler Q4_K_M quantization (88MB - optimal balance)
final config = LlamaConfig(
  modelPath: '/path/to/braindler-q4_k_s.gguf',  // braindler from ollama.com/nativemind/braindler
  nThreads: 4,
  nGpuLayers: 0,  // 0 = CPU only, -1 = all layers on GPU
  contextSize: 2048,
  batchSize: 512,
  useGpu: true,
  verbose: false,
);

final success = await llama.loadModel(config);
if (success) {
  print('Braindler model loaded successfully!');
}

2. Generate Text (Blocking)

final params = GenerationParams(
  prompt: 'Hello, how are you?',
  temperature: 0.8,
  topP: 0.95,
  topK: 40,
  maxTokens: 512,
  repeatPenalty: 1.1,
);

try {
  final response = await llama.generate(params);
  print('Generated: ${response.text}');
  print('Tokens: ${response.tokensGenerated}');
  print('Speed: ${response.tokensPerSecond.toStringAsFixed(2)} tok/s');
} catch (e) {
  print('Error: $e');
}

3. Generate Text (Streaming)

final params = GenerationParams(
  prompt: 'Tell me a story',
  maxTokens: 1000,
);

try {
  await for (final token in llama.generateStream(params)) {
    print(token); // Print each token as it's generated
  }
} catch (e) {
  print('Error: $e');
}

4. Get Model Info

final info = await llama.getModelInfo();
if (info != null) {
  print('Model path: ${info['modelPath']}');
  print('Parameters: ${info['nParams']}');
  print('Layers: ${info['nLayers']}');
  print('Context size: ${info['contextSize']}');
}

5. Unload Model

await llama.unloadModel();
print('Model unloaded');

Configuration Options

LlamaConfig

• modelPath (String, required): Path to the GGUF model file
• nThreads (int, default: 4): Number of CPU threads to use
• nGpuLayers (int, default: 0): Number of layers to offload to GPU (0 = CPU only, -1 = all)
• contextSize (int, default: 2048): Context size in tokens
• batchSize (int, default: 512): Batch size for processing
• useGpu (bool, default: true): Enable GPU acceleration
• verbose (bool, default: false): Enable verbose logging

GenerationParams

• prompt (String, required): The prompt for text generation
• temperature (double, default: 0.8): Sampling temperature (0.0 - 2.0)
• topP (double, default: 0.95): Top-P sampling (0.0 - 1.0)
• topK (int, default: 40): Top-K sampling
• maxTokens (int, default: 512): Maximum tokens to generate
• repeatPenalty (double, default: 1.1): Penalty for repeating tokens
• stopSequences (List

Example App

See the example directory for a complete example application.

Performance Tips

🚀 GPU Acceleration (NEW in v1.0.0!)

GPU acceleration is now fully enabled and automatic!

1. iOS/macOS (Metal):

   nGpuLayers: -1,  // -1 = all layers on Metal GPU
   useGpu: true,    // Metal automatically detected
   

   • Performance: 3-10x faster than CPU
   • Devices: iPhone 8+, iPad Pro, MacBook Pro
   • Expected: ~45-50 tok/s on iPhone 14 Pro

2. Android (Vulkan/OpenCL):

   nGpuLayers: -1,  // -1 = all layers on GPU
   useGpu: true,    // Vulkan auto-detected, fallback to OpenCL
   

   • Performance: 4-8x faster than CPU (Vulkan), 2-5x (OpenCL)
   • Devices: Android 7.0+ with Vulkan support
   • Expected: ~18-25 tok/s on flagship devices
   • Fallback: Automatically uses OpenCL if Vulkan unavailable

3. Threading (CPU fallback): Adjust nThreads based on device:

   • Mobile devices: 4-6 threads
   • High-end devices: 6-8 threads

4. Model Size: Use braindler quantized models for better performance:

   • braindler:q2_k (72MB): Smallest, fastest, good quality
   • braindler:q4_k_s (88MB): ⭐ Recommended - Optimal balance
   • braindler:q5_k_m (103MB): Higher quality, larger size
   • braindler:q8_0 (140MB): Best quality, largest size

   Get from: https://ollama.com/nativemind/braindler

5. Context Size: Reduce if memory is limited:

   • Small devices: 1024-2048
   • Medium devices: 2048-4096
   • High-end devices: 4096-8192

Requirements

iOS

• iOS 13.0 or later
• Xcode 14.0 or later
• Metal support for GPU acceleration

Android

• Android API level 24 (Android 7.0) or later
• NDK r25 or later
• Vulkan support for GPU acceleration (optional)

Building

The plugin includes native C++ code for llama.cpp integration.

iOS

The iOS framework will be built automatically when you build your Flutter app.

Android

The Android native library will be built automatically using CMake/NDK.

GGUF Models

This plugin supports GGUF model format. We recommend using the braindler model:

Recommended: Braindler Model

Get the braindler model from Ollama:

Available quantizations:

• braindler:q2_k - 72MB - Fastest, good quality
• braindler:q3_k_s - 77MB - Better quality
• braindler:q4_k_s - 88MB - ⭐ Recommended - Optimal balance
• braindler:q5_k_m - 103MB - Higher quality
• braindler:q8_0 - 140MB - Best quality
• braindler:f16 - 256MB - Maximum quality

How to get braindler models:

1. Install Ollama: https://ollama.com
2. Pull the model: ollama pull nativemind/braindler:q4_k_s
3. Export to GGUF: ollama export nativemind/braindler:q4_k_s -o braindler-q4_k_s.gguf
4. Copy the GGUF file to your app's assets or documents directory

Other model sources:

• TheBloke on Hugging Face
• Hugging Face GGUF models

Limitations

• Model file must be accessible on device storage
• Large models require significant RAM
• Generation speed depends on device capabilities
• Some older devices may not support GPU acceleration

License

This plugin is released under the NativeMindNONC License. See LICENSE file for details.

llama.cpp is released under the MIT License.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Acknowledgments

• llama.cpp by Georgi Gerganov
• The Flutter team for the excellent framework
• Braindler model from Ollama - recommended model for mobile devices