flutter_llm_memory

A memory layer that decides which conversation messages to send to an LLM, so requests stay within the context window and cost stays down.

The problem

LLMs are stateless. To keep a conversation coherent you resend the history on every request. As the conversation grows, each request gets larger, costs rise, and eventually you hit the model's context limit and the request fails. This package sits between your chat code and your LLM client and manages what gets sent: it tracks history, estimates tokens, prunes or summarizes old messages, and reports how many tokens you saved.

Install

dependencies:
  flutter_llm_memory: ^0.1.0

dart pub add flutter_llm_memory

Quick start

import 'package:flutter_llm_memory/flutter_llm_memory.dart';

final memory = ConversationMemory(
  strategy: SlidingWindowStrategy(maxTokens: 4000),
);

await memory.add(Message.user('Hello, I need help with my order'));
await memory.add(Message.assistant('Sure, what is your order number?'));
await memory.add(Message.user('It is 48213'));

// Build the messages to send to your LLM.
final (context, stats) = await memory.buildContext();

// `context` is a List<Message>. Map it to your provider's request shape.
final payload = context
    .map((message) => {'role': message.role.name, 'content': message.content})
    .toList();

print('Sending ${stats.currentTokens} tokens, saved ${stats.tokensSaved}');

The package never calls an LLM. You map context to your provider and make the request yourself.

Strategies

Strategy	Best for	LLM calls needed
SlidingWindowStrategy	Short sessions, no summarization	No
SummarizationStrategy	Long sessions, fresh summary each time	Yes
HybridMemoryStrategy	Long sessions, incremental memory	Yes

SlidingWindowStrategy

Keeps the most recent messages that fit in the budget. System messages are always kept.

final strategy = SlidingWindowStrategy(maxTokens: 4000);

SummarizationStrategy

Once the conversation exceeds the budget, older messages are summarized by your function and replaced with a single summary message. The most recent keepRecentCount messages stay in full.

final strategy = SummarizationStrategy(
  maxTokens: 4000,
  keepRecentCount: 10,
  summarizer: (olderMessages) => summarizeWithMyLlm(olderMessages),
);

HybridMemoryStrategy

Keeps a running summary that grows incrementally plus a window of recent messages. It summarizes only the oldest overflow messages per event, so it makes one summarization call per overflow instead of resummarizing everything.

final strategy = HybridMemoryStrategy(
  maxTokens: 4000,
  keepRecentCount: 8,
  summarizeChunkSize: 4,
  summarizer: (olderMessages) => summarizeWithMyLlm(olderMessages),
);

If your summarizer throws, all three summarizing strategies fall back to sliding window behavior for that call instead of crashing.

Persistence

History is in memory by default. Provide a MemoryStorage adapter to persist it, then restore on startup.

final memory = await ConversationMemory.restore(
  storage: SharedPreferencesStorage(),
  strategy: HybridMemoryStrategy(
    maxTokens: 4000,
    summarizer: summarizeWithMyLlm,
  ),
);

MemoryStorage is a three-method interface (save, load, clear). The package ships InMemoryStorage. A SharedPreferencesStorage reference implementation lives in the example app; copy it or adapt it for Hive or SQLite.

Stats

Every buildContext() returns a MemoryStats, also available as memory.stats after the call.

final (context, stats) = await memory.buildContext();

stats.currentTokens;        // tokens in the context that will be sent
stats.totalTokens;          // tokens in the full untrimmed history
stats.tokensSaved;          // totalTokens - currentTokens
stats.messageCount;         // messages in the full history
stats.contextMessageCount;  // messages in the returned context
stats.compressionRatio;     // currentTokens / totalTokens, 1.0 means no compression
stats.isSummarized;         // whether a summary block is present

Bring your own LLM

This package does not depend on any provider and never makes a network call. You pass a summarizer and you send the built context yourself. Here is a summarizer backed by OpenAI:

Future<String> summarizeWithMyLlm(List<Message> messages) async {
  final transcript =
      messages.map((m) => '${m.role.name}: ${m.content}').join('\n');
  final response = await http.post(
    Uri.parse('https://api.openai.com/v1/chat/completions'),
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer $openAiKey',
    },
    body: jsonEncode({
      'model': 'gpt-4o-mini',
      'messages': [
        {'role': 'system', 'content': 'Summarize this conversation concisely.'},
        {'role': 'user', 'content': transcript},
      ],
    }),
  );
  final data = jsonDecode(response.body) as Map<String, dynamic>;
  return data['choices'][0]['message']['content'] as String;
}

Model token limits reference

Model	Context window
GPT-4o	128k
Claude Sonnet	200k
Gemini 1.5 Pro	1M
Llama 3 70B	8k

Set maxTokens to about 80% of the model limit to leave room for the response.

Contributing

PRs are welcome. For anything larger than a bug fix, open an issue first so we can agree on the approach before you write code.

License

MIT. See LICENSE.