Llamafu #

A Flutter package for running language models on device with support for completion, instruct mode, tool calling, streaming, constrained generation, LoRA, and multi-modal inputs (images, audio).

Features #

• 🚀 Run language models directly on device (Android and iOS)
• 💬 Support for text completion
• 🤖 Instruct mode for chat-like interactions
• 🛠️ Tool calling capabilities
• 🌊 Streaming output
• 🔒 Constrained generation (GBNF grammars)
• 🧬 LoRA adapter support
• 🖼️🎧 Multi-modal support (images, audio)

Prerequisites #

• Flutter 3.0 or higher
• Android SDK/NDK for Android development
• Xcode for iOS development
• Pre-built llama.cpp libraries

Installation #

Add llamafu as a dependency in your pubspec.yaml file:

dependencies:
  llamafu: ^0.0.1

Then run:

flutter pub get

Usage #

Text Completion #

import 'package:llamafu/llamafu.dart';

// Initialize the model
final llamafu = await Llamafu.init(
  modelPath: '/path/to/your/model.gguf',
  threads: 4,
  contextSize: 512,
);

// Generate text
final result = await llamafu.complete(
  prompt: 'The quick brown fox',
  maxTokens: 128,
  temperature: 0.8,
);

print(result);

// Clean up resources
llamafu.close();

Multi-modal Inference #

import 'package:llamafu/llamafu.dart';

// Initialize the model with multi-modal projector
final llamafu = await Llamafu.init(
  modelPath: '/path/to/your/model.gguf',
  mmprojPath: '/path/to/your/mmproj.gguf', // Multi-modal projector file
  threads: 4,
  contextSize: 512,
  useGpu: false, // Set to true to use GPU for multi-modal processing
);

// Generate text with image input
final mediaInputs = [
  MediaInput(
    type: MediaType.image,
    data: '/path/to/your/image.jpg', // Path to image file
  ),
];

final result = await llamafu.multimodalComplete(
  prompt: 'Describe this image: <image>',
  mediaInputs: mediaInputs,
  maxTokens: 128,
  temperature: 0.8,
);

print(result);

// Clean up resources
llamafu.close();

LoRA Adapter Support #

import 'package:llamafu/llamafu.dart';

// Initialize the model
final llamafu = await Llamafu.init(
  modelPath: '/path/to/your/model.gguf',
  threads: 4,
  contextSize: 512,
);

// Load a LoRA adapter
final loraAdapter = await llamafu.loadLoraAdapter('/path/to/your/lora.gguf');

// Apply the LoRA adapter with a scale factor
await llamafu.applyLoraAdapter(loraAdapter, scale: 0.5);

// Generate text with the LoRA adapter applied
final result = await llamafu.complete(
  prompt: 'Write a story about space exploration',
  maxTokens: 128,
  temperature: 0.8,
);

print(result);

// Remove the LoRA adapter
await llamafu.removeLoraAdapter(loraAdapter);

// Or clear all LoRA adapters
await llamafu.clearAllLoraAdapters();

// Clean up resources
llamafu.close();

Constrained Generation #

import 'package:llamafu/llamafu.dart';

// Initialize the model
final llamafu = await Llamafu.init(
  modelPath: '/path/to/your/model.gguf',
  threads: 4,
  contextSize: 512,
);

// Define a JSON grammar
final jsonGrammar = '''
root   ::= object
value  ::= object | array | string | number | ("true" | "false" | "null") ws

object ::=
  "{" ws (
            string ":" ws value
    ("," ws string ":" ws value)*
  )? "}" ws

array  ::=
  "[" ws (
            value
    ("," ws value)*
  )? "]" ws

string ::=
  "\"" (
    [^\"\\\\\x7F\x00-\x1F] |
    "\\\\" (["\\\\bfnrt] | "u" [0-9a-fA-F]{4}) # escapes
  )* "\"" ws

number ::= ("-"? ([0-9] | [1-9] [0-9]{0,15})) ("." [0-9]+)? ([eE] [-+]? [0-9] [1-9]{0,15})? ws

# Optional space: by convention, applied in this grammar after literal chars when allowed
ws ::= | " " | "\n" [ \t]{0,20}
''';

// Generate text constrained to JSON format
final result = await llamafu.completeWithGrammar(
  prompt: 'Generate a JSON object describing a person:',
  grammarStr: jsonGrammar,
  grammarRoot: 'root',
  maxTokens: 256,
  temperature: 0.8,
);

print(result);

// Clean up resources
llamafu.close();

Supported Multi-modal Models #

Llamafu supports various multi-modal models through the llama.cpp MTMD library:

Vision Models #

• Gemma 3
• SmolVLM
• Pixtral 12B
• Qwen 2 VL
• Qwen 2.5 VL
• Mistral Small 3.1 24B
• InternVL 2.5 and 3
• Llama 4 Scout
• Moondream2

Audio Models #

• Ultravox 0.5
• Qwen2-Audio
• SeaLLM-Audio
• Voxtral Mini

Mixed Modalities #

• Qwen2.5 Omni (audio + vision)

Building #

Android #

1. Ensure you have the Android NDK installed
2. Build the native libraries:

   cd android/src/main/cpp
   mkdir build
   cd build
   cmake .. -DLLAMA_CPP_DIR=/path/to/llama.cpp
   make
   

iOS #

1. Ensure you have Xcode installed
2. Build the native libraries using Xcode or CMake

API Reference #

For detailed API documentation, please refer to the API documentation.

License #

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments #

• This project uses the excellent llama.cpp library for running language models.
• Multi-modal support is provided by the MTMD library in llama.cpp.
• LoRA support is provided by the native LoRA adapter functionality in llama.cpp.
• Constrained generation support is provided by the grammar sampler functionality in llama.cpp.