dart_vector_embedding

Cross-platform Dart Native GGUF embeddings through vendored llama.cpp with one shared native runtime per model path, native queued execution, and async multi-isolate access.

Features

  • Native llama.cpp embeddings through dart:ffi
  • Async Embedder.open(...), embed(...), embedBatch(...), and close()
  • One shared native llama_model + llama_context per canonical model path inside a Dart process
  • One native worker thread per shared model path with queued request execution
  • Symmetric multi-isolate access without each isolate loading a duplicate model
  • GPU offload when the host/backend supports it
  • Works with dart run, dart test, and dart build cli
  • Cross-platform native assets: macOS, Linux, Windows

Why This Package

Most Dart-side local embedding wrappers either assume a single isolate, require callers to manage a native sidecar manually, or expose a much broader llama.cpp surface than an embedding workload needs.

dart_vector_embedding exists for the case where multiple isolates in the same Dart process need to generate embeddings against the same local GGUF model efficiently.

Native efficiency. Tokenization and embedding execution run in C++ through dart:ffi, not in the Dart VM. This means:

  • No duplicate model load for every isolate in the same process
  • No Dart-side queue or coordinator isolate required
  • Native request serialization around one shared llama.cpp runtime
  • Async completion back into Dart without Isolate.run

Core concurrency model:

  • opening the same model path from multiple isolates attaches them to one shared native runtime
  • requests for that model path are queued in native code
  • one dedicated native worker thread executes requests one at a time
  • up to 64 requests may wait per model; excess requests fail with VeQueueFullException
  • callers await Futures and remain free to do other work

This lets you build isolate-heavy CLI or server workloads without loading the same GGUF model separately in every isolate.

Native Build Requirements

The Dart build hook runs CMake automatically, but the host must provide:

  • CMake 3.19 or newer
  • A C/C++17 toolchain: Xcode Command Line Tools on macOS, GCC/Clang on Linux, or Visual Studio Build Tools on Windows

CUDA and Vulkan SDKs are optional. Without them, the package builds with Metal on macOS or CPU support on other platforms.

Getting Started

1. Add dependency

dependencies:
  dart_vector_embedding: ^0.1.1

2. Import the package

import 'package:dart_vector_embedding/dart_vector_embedding.dart';

3. Open the embedder and generate embeddings

final embedder = await Embedder.open(
  modelPath: '/absolute/path/to/model.gguf',
  outDim: 256,
);

try {
  final query = await embedder.embed('Which planet is known as the Red Planet?');

  final docs = await embedder.embedBatch([
    'Mars is the Red Planet.',
    'Saturn has prominent rings.',
  ]);

  print('query dim: ${query.length}');
  print('docs: ${docs.length}');
} finally {
  await embedder.close();
}

See example/example.dart.

How Should I Run It

dart run

Use this for normal development.

dart run bin/your_app.dart

The native library is built and loaded automatically through Dart hooks/code assets. No manual cmake step or libraryPath argument is required.

This mode preserves the shared native runtime behavior for multiple isolates inside the same Dart process.

Build a distributable CLI

Packages with Dart build hooks, including dart_vector_embedding, must be compiled with dart build cli. It produces a self-contained application bundle with the executable in bin/ and its native libraries in lib/.

dart build cli --target bin/your_app.dart --output dist

For example, on macOS the output is:

dist/bundle/bin/your_app
dist/bundle/lib/libdart_vector_embedding.dylib

Ship the entire dist/bundle/ directory. On Linux the library is named libdart_vector_embedding.so; on Windows it is dart_vector_embedding.dll.

At runtime, dart_vector_embedding resolves native code in this order:

  1. Dart native-asset mapping, if available
  2. DART_VECTOR_EMBEDDING_LIBRARY_PATH, if set
  3. the executable directory

Example override:

DART_VECTOR_EMBEDDING_LIBRARY_PATH=/opt/dve/libdart_vector_embedding.so ./your_app

Only use an override path controlled by your application deployment.

Models

  • This package does not bundle model files.
  • Use any compatible GGUF embedding model and pass its path into Embedder.open(...).
  • Development-only local model setup notes may exist in the repository, but they are not part of the published package.

Concurrency Model

  • Same process, same model path: one shared native model/context slot
  • Same process, multiple isolates: all isolates can attach to that same slot
  • Requests for a shared model path: queued in native code and executed sequentially on one native worker thread
  • Queue limit: 64 waiting requests per shared model path; excess requests fail immediately with VeQueueFullException
  • Request limits: 1 MiB of UTF-8 text per item, 4 MiB total batch text, and 256 items per batch
  • Model paths and embedding text must not contain NUL characters
  • Completion back to Dart: async callback -> Future

Shared-runtime attachment rules:

  • outDim, nCtx, pooling, and normalize must match the slot's effective configuration
  • outDim is compared after native clamping, so 0 and values above the model embedding dimension both resolve to full model dimension
  • nThreads uses a first-opener-wins policy; later attaches reuse the existing runtime thread count

This is one of the package's core design goals: isolate-friendly shared native embeddings, not just local single-isolate inference.

GPU Behavior

GPU offload is enabled when a supported build toolchain is detected.

  • macOS: Metal
  • Linux/Windows with a CUDA compiler: CUDA
  • Linux/Windows with a Vulkan SDK, glslc, and SPIR-V headers: Vulkan

CUDA and Vulkan detection happens during the native hook build. Missing GPU toolchains are not errors; the package falls back to CPU automatically.

Supported Platforms

The hook and native source are configured for:

  • macOS
  • Linux
  • Windows

Web is unsupported because this package depends on native FFI.

API Overview

Embedder

Method / Property Description
Embedder.open(...) Open or attach to a shared native embedding runtime
embed(text) Embed one string
embedBatch(texts) Embed multiple strings
close() Release this Dart handle
outDim Effective output dimension of this slot

PoolingType

  • PoolingType.mean
  • PoolingType.cls
  • PoolingType.last

Third-party Source

  • Native sources are vendored under third_party/llama.cpp
  • No consumer git submodule update step is required
  • Upstream pin and update notes live in third_party/llama.cpp/UPSTREAM.md

Running Tests

The tests use the 300M Q4_0 EmbeddingGemma model and do not download it automatically. From the package root:

mkdir -p models
curl -L \
  https://huggingface.co/ggml-org/embeddinggemma-300M-qat-q4_0-GGUF/resolve/main/embeddinggemma-300M-qat-Q4_0.gguf \
  -o models/embeddinggemma-300M-qat-Q4_0.gguf

The model is approximately 278 MB. It comes from the ggml-org EmbeddingGemma GGUF repository.

Then run:

dart analyze
dart test
bash test/run_native_regression.sh

To use an existing model at another location:

DART_VECTOR_EMBEDDING_TEST_MODEL=/path/to/model.gguf dart test

bash test/run_all.sh remains an equivalent convenience wrapper around dart test.

Release Check

Maintainers can run all publish checks with:

bash tools/check_release.sh

It runs formatting, analysis, model-backed Dart and native regression tests when a local test model is available, and dart pub publish --dry-run. It never downloads a model automatically.

Benchmark

The benchmark opens the same model with outDim: 256, performs five warmup embeddings, then reports the average sequential embedding duration. Model load time is excluded.

dart run benchmark/embed_benchmark.dart

Pass a model path and iteration count explicitly if needed:

dart run benchmark/embed_benchmark.dart /path/to/model.gguf 100

Libraries

dart_vector_embedding
Cross-platform Dart Native GGUF embeddings through vendored llama.cpp.