flutter_llm_memory #

A memory layer that decides which conversation messages to send to an LLM, so requests stay within the context window and cost stays down.

The problem #

LLMs are stateless. To keep a conversation coherent you resend the history on every request. As the conversation grows, each request gets larger, costs rise, and eventually you hit the model's context limit and the request fails. This package sits between your chat code and your LLM client and manages what gets sent: it tracks history, estimates tokens, prunes or summarizes old messages, and reports how many tokens you saved.

Install #

dependencies:
  flutter_llm_memory: ^0.1.0

dart pub add flutter_llm_memory

Quick start #

import 'package:flutter_llm_memory/flutter_llm_memory.dart';

final memory = ConversationMemory(
  strategy: SlidingWindowStrategy(maxTokens: 4000),
);

await memory.add(Message.user('Hello, I need help with my order'));
await memory.add(Message.assistant('Sure, what is your order number?'));
await memory.add(Message.user('It is 48213'));

// Build the messages to send to your LLM.
final (context, stats) = await memory.buildContext();

// `context` is a List<Message>. Map it to your provider's request shape.
final payload = context
    .map((message) => {'role': message.role.name, 'content': message.content})
    .toList();

print('Sending ${stats.currentTokens} tokens, saved ${stats.tokensSaved}');

The package never calls an LLM. You map context to your provider and make the request yourself.

Strategies #

SlidingWindowStrategy #

Keeps the most recent messages that fit in the budget. System messages are always kept.

final strategy = SlidingWindowStrategy(maxTokens: 4000);

SummarizationStrategy #

Once the conversation exceeds the budget, older messages are summarized by your function and replaced with a single summary message. The most recent keepRecentCount messages stay in full.

final strategy = SummarizationStrategy(
  maxTokens: 4000,
  keepRecentCount: 10,
  summarizer: (olderMessages) => summarizeWithMyLlm(olderMessages),
);

HybridMemoryStrategy #

Keeps a running summary that grows incrementally plus a window of recent messages. It summarizes only the oldest overflow messages per event, so it makes one summarization call per overflow instead of resummarizing everything.

final strategy = HybridMemoryStrategy(
  maxTokens: 4000,
  keepRecentCount: 8,
  summarizeChunkSize: 4,
  summarizer: (olderMessages) => summarizeWithMyLlm(olderMessages),
);

If your summarizer throws, all three summarizing strategies fall back to sliding window behavior for that call instead of crashing.

Persistence #

History is in memory by default. Provide a MemoryStorage adapter to persist it, then restore on startup.

final memory = await ConversationMemory.restore(
  storage: SharedPreferencesStorage(),
  strategy: HybridMemoryStrategy(
    maxTokens: 4000,
    summarizer: summarizeWithMyLlm,
  ),
);

MemoryStorage is a three-method interface (save, load, clear). The package ships InMemoryStorage. A SharedPreferencesStorage reference implementation lives in the example app; copy it or adapt it for Hive or SQLite.

Stats #

Every buildContext() returns a MemoryStats, also available as memory.stats after the call.

final (context, stats) = await memory.buildContext();

stats.currentTokens;        // tokens in the context that will be sent
stats.totalTokens;          // tokens in the full untrimmed history
stats.tokensSaved;          // totalTokens - currentTokens
stats.messageCount;         // messages in the full history
stats.contextMessageCount;  // messages in the returned context
stats.compressionRatio;     // currentTokens / totalTokens, 1.0 means no compression
stats.isSummarized;         // whether a summary block is present

Bring your own LLM #

This package does not depend on any provider and never makes a network call. You pass a summarizer and you send the built context yourself. Here is a summarizer backed by OpenAI:

Future<String> summarizeWithMyLlm(List<Message> messages) async {
  final transcript =
      messages.map((m) => '${m.role.name}: ${m.content}').join('\n');
  final response = await http.post(
    Uri.parse('https://api.openai.com/v1/chat/completions'),
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer $openAiKey',
    },
    body: jsonEncode({
      'model': 'gpt-4o-mini',
      'messages': [
        {'role': 'system', 'content': 'Summarize this conversation concisely.'},
        {'role': 'user', 'content': transcript},
      ],
    }),
  );
  final data = jsonDecode(response.body) as Map<String, dynamic>;
  return data['choices'][0]['message']['content'] as String;
}

Model token limits reference #

Set maxTokens to about 80% of the model limit to leave room for the response.

Contributing #

PRs are welcome. For anything larger than a bug fix, open an issue first so we can agree on the approach before you write code.

License #

MIT. See LICENSE.