token_text_controller

A token-aware TextEditingController for Flutter. It detects and styles semantic text while preserving a normal editable string, cursor, selection, clipboard behavior, and IME composition.

Use it with any TextField, TextFormField, or EditableText. No custom input widget is required.

Features

  • Every built-in detector enabled by default, with fine-grained activation.
  • Reusable token groups for messaging, social, contact, commerce, structured data, suggestions, links, and Markdown.
  • Stable light/dark token colors independent of theme brand colors.
  • A dedicated optional TextStyle for every built-in token kind.
  • Regex, function, trigger, and fully custom token detectors.
  • Priority-based overlap resolution.
  • Unicode grapheme handling for joined, modified, keycap, and flag emoji.
  • Current-token queries for mention, hashtag, command, and cashtag suggestions.
  • Token list, newly detected token, and active query callbacks.
  • Semantic token insertion with arbitrary custom metadata.
  • A reusable TokenText renderer for non-editable content.
  • Correct composing-range rendering for international keyboards and IMEs.

Installation

Add the published package to pubspec.yaml:

dependencies:
  token_text_controller: ^0.3.1

Minimal usage

final controller = TokenTextController();

TextField(
  controller: controller,
  decoration: const InputDecoration(hintText: 'Message'),
)

By default, every built-in token kind is active, including links, contact values, social tokens, structured values, lightweight Markdown, commands, and emoji. controller.text remains an ordinary string.

Markdown rendering

Recognized inline formatting renders without its delimiters by default:

final controller = TokenTextController(
  text: '**Important** _note_ ~~old~~ ||spoiler|| `code` [Flutter](https://flutter.cn)',
);

Bold supports **text** and __text__; italic supports _text_ and *text*. Markdown links show only their label. Their destination is available in token.data; token.displayText and token.contentRange expose the visible portion. token.text, token.range, controller.text, and serialized documents always retain the original source, including syntax.

In an editable field, hidden syntax has zero visual width but keeps its UTF-16 positions. It is not deleted or replaced by widget placeholders, so selections and the IME keep working with the original value. Composing ranges underline the visible content without exposing the hidden syntax. Copying from the editable field keeps the source; read-only/selectable TokenText copies rendered text.

Use showMarkdownSyntax: true in either constructor to show delimiters. The controller property can also change while editing:

controller.showMarkdownSyntax = true;

Disabling Markdown token kinds leaves their syntax literal. Inline code does not interpret nested links or formatting; escaped and incomplete markers are not treated as formatting. Prefixes such as @, #, $, and template braces remain visible. These are lightweight, non-overlapping inline detectors, not a full CommonMark/block editor. Formatting-like punctuation inside detected URLs, email addresses, handles, hashtags, and variables stays literal. TokenTextSpanBuilder preserves source length by default; only read-only consumers should set preserveSourceLength: false.

Styling

Every built-in token accepts a complete Flutter TextStyle. Unspecified styles fall back to theme-aware defaults.

The default palette adapts to light/dark brightness without using the primary, secondary, or tertiary brand colors:

Tokens Light Dark
Links, email, phone, mentions, hashtags, references #1565C0 #90CAF9
Cashtags, currency, percentages #2E7D32 #A5D6A7
Dates, times #00695C #80CBC4
Commands, template variables #6A1B9A #CE93D8

Blue follows the familiar link convention. The other colors are package defaults, not universal token standards. Links and contact details are underlined; mentions and hashtags have extra weight. Surfaces, ordinary text, and code text still follow ColorScheme. Check contrast when using custom backgrounds.

Emoji use a stable 20 font size by default in both mixed text and emoji-only values. Set emoji or emojiOnly to customize the size.

final controller = TokenTextController(
  styles: TokenTextStyles(
    base: TextStyle(fontSize: 16),
    url: TextStyle(decorationThickness: 2),
    email: TextStyle(fontWeight: FontWeight.w500),
    mention: TextStyle(
      color: Colors.blue,
      fontWeight: FontWeight.w700,
    ),
    hashtag: TextStyle(color: Colors.green),
    command: TextStyle(color: Colors.deepPurple),
    emoji: TextStyle(fontSize: 20),
    emojiOnly: TextStyle(fontSize: 30),
  ),
);

TokenTextStyles also provides styles for phone numbers, references, dates, times, currencies, percentages, UUIDs, IP addresses, variables, Markdown links, inline code, bold, italic, strikethrough, spoilers, and custom kinds.

Use TokenTextStyles.unstyled() when no built-in visual treatment is desired. For per-value or metadata-driven styling, provide a resolver:

TokenTextController(
  styleResolver: (details) {
    if (details.token.data == 'verified') {
      return details.suggestedStyle.copyWith(
        decoration: TextDecoration.underline,
      );
    }
    return details.suggestedStyle;
  },
)

Suggestions and active queries

onActiveTokenChanged and activeQueryListenable report the trigger token at the caret, including incomplete values such as @lu, #flu, or /he.

final controller = TokenTextController(
  onActiveTokenChanged: (query) {
    if (query?.kind == TokenKind.mention) {
      loadProfiles(query!.query);
    }
  },
);

A suggestion can replace the complete active range and retain domain data:

controller.replaceActiveQuery(
  '@Luis',
  kind: TokenKind.mention,
  data: {'profileId': 'profile-123'},
);

The inserted token appears in controller.semanticTokens and remains linked to its metadata until an edit intersects that token.

Serialization

Persist plain text and resolved semantic metadata together:

final json = controller.document.toJson();
final document = TokenTextDocument.fromJson(json);
final restoredController = TokenTextController.fromDocument(document);

Use encodeData and decodeData when token metadata needs conversion to a JSON-compatible representation.

Token events

TokenTextController(
  onTokensChanged: (tokens) {
    // Current non-overlapping token snapshot.
  },
  onTokenDetected: (token) {
    // A token has newly appeared in the editable value.
  },
  onActiveTokenChanged: (query) {
    // The token query at the current collapsed selection changed.
  },
)

The same state is available through tokensListenable and activeQueryListenable for widget composition without callbacks.

Active tokens and groups

Use activeTokens to enable only the token kinds needed by a field. The default is TokenGroups.all.

final controller = TokenTextController(
  activeTokens: TokenGroups.chat,
);

Groups are intentionally scoped to a concrete kind of field:

Group Intended use
links Web URLs and deep links
contact Email addresses and phone numbers
social Posts and comments with links, mentions, hashtags, and emoji
chat Human chat without commands, finance, or structured tokens
messaging Alias for chat
assistantChat AI or bot chat with commands and Markdown
commerce Cashtags, currencies, and percentages
temporal Dates and times
identifiers References, UUIDs, IP addresses, and variables
structured All built-in identifiers, dates, and times
markdown Lightweight Markdown syntax
developerContent Links, identifiers, and markup in technical text
socialSuggestions Mention and hashtag suggestion triggers
financeSuggestions Cashtag suggestion triggers
commandSuggestions Slash-command suggestion triggers
suggestions / triggers Every built-in suggestion trigger
all Every detector included by the package

Groups are regular constant lists, so they can be used directly or combined:

final controller = TokenTextController(
  activeTokens: TokenGroups.combine(
    [TokenGroups.social, TokenGroups.markdown],
    include: const [TokenKind.reference],
    exclude: const [TokenKind.cashtag],
  ),
);

Activation can change while the field is alive. Existing text, styles, token events, and the active suggestion query are recalculated immediately.

controller.activeTokens = TokenGroups.suggestions;
controller.activateTokens(const [TokenKind.emoji]);
controller.deactivateTokens(const [TokenKind.cashtag]);
controller.activateAllTokens();

availableTokens reports the kinds backed by the controller's detector catalog, while isTokenActive(kind) checks the current selection.

Detector presets

TokenTextController(detectors: TokenDetectors.social());
TokenTextController(detectors: TokenDetectors.chat());
TokenTextController(detectors: TokenDetectors.messaging());
TokenTextController(detectors: TokenDetectors.assistantChat());
TokenTextController(detectors: TokenDetectors.commerce());
TokenTextController(detectors: TokenDetectors.structured());
TokenTextController(detectors: TokenDetectors.markdown());
TokenTextController(detectors: TokenDetectors.all());

Use detectors to replace or customize the detector catalog. All supplied detectors are active unless activeTokens is also provided. TokenDetectors also exposes forKind and forKinds for building a catalog from token kinds.

Custom tokens

Token kinds are value objects rather than a closed enum:

const orderKind = TokenKind('order');

final controller = TokenTextController(
  detectors: [
    ...TokenDetectors.defaults(),
    RegexTokenDetector(
      kind: orderKind,
      id: 'order',
      priority: 150,
      pattern: RegExp(r'ORD-\d+'),
    ),
  ],
  styles: TokenTextStyles(
    custom: {
      orderKind: TextStyle(
        color: Colors.orange,
        fontWeight: FontWeight.bold,
      ),
    },
  ),
);

Implement TokenDetector directly or use FunctionTokenDetector when a regular expression is not sufficient. Higher priorities win when ranges overlap.

Rendering outside a field

TokenText(
  message,
  activeTokens: TokenGroups.chat,
  styles: controller.styles,
  semanticTokens: controller.semanticTokens,
)

This package performs token-aware styling and semantic range management. It is not a document editor and does not alter the source string with hidden markup.

Libraries

token_text_controller
Token-aware text editing, parsing, styling, and rendering for Flutter.