toonX #

[TOON Format Diagram]

Dart implementation of TOON (Token-Oriented Object Notation) - a compact, human-readable format for LLMs that uses ~40% fewer tokens than JSON while maintaining better accuracy.

🎯 What is TOON? #

TOON combines YAML's indentation-based structure with CSV-style tabular arrays to create the most efficient format for passing structured data to Large Language Models. It's a lossless, drop-in replacement for JSON designed specifically for LLM inputs.

Think: JSON for machines, TOON for AI models.

📦 What This Package Supports #

Core Functionality:

• ✅ encode() - Convert Dart objects to TOON format
• ✅ decode() - Parse TOON strings back to Dart objects
• ✅ Lossless round-trip conversion with full JSON data model support

Encoding Options:

• ✅ Custom delimiters - comma (,), tab (\t), or pipe (|) for arrays
• ✅ Length markers - optional # prefix for array validation (e.g., tags[#3])
• ✅ Custom indentation - configurable spaces per level (default: 2)
• ✅ Tabular arrays - automatic CSV-style format for uniform objects
• ✅ Flat map mode - flatten nested structures with custom separators
• ✅ Smart quoting - minimal quoting, only when necessary

Decoding Options:

• ✅ Strict mode - validates array lengths, delimiters, and syntax (default)
• ✅ Lenient mode - best-effort parsing for malformed input
• ✅ Unflatten support - reconstruct nested structures from flat maps
• ✅ Escape sequence handling - proper handling of special characters

Additional Features:

• ✅ CLI tool - command-line interface for file/stream conversion
• ✅ Comprehensive tests - 46+ tests including large datasets
• ✅ Full documentation - detailed API docs for all functions
• ✅ Type-safe - strongly typed Dart implementation

💡 Why TOON? #

Token Efficiency #

TOON significantly reduces token usage compared to other formats:

LLM Accuracy #

Benchmark results across 4 models on 209 data retrieval questions:

• TOON: 73.9% accuracy
• JSON: 69.7% accuracy
• YAML: 69.0% accuracy
• XML: 67.1% accuracy

Result: TOON achieves higher accuracy while using fewer tokens.

🚀 Quick Start #

Installation #

dependencies:
    toonx: ^1.0.0

Basic Usage #

import 'package:toonx/toonx.dart';

final data = {
  'users': [
    {'id': 1, 'name': 'Alice', 'role': 'admin'},
    {'id': 2, 'name': 'Bob', 'role': 'user'},
  ]
};

// Encode to TOON
final toon = encode(data);
print(toon);
// users[2]{id,name,role}:
//   1,Alice,admin
//   2,Bob,user

// Decode back to Dart
final decoded = decode(toon);
print(decoded); // Original data restored

💡 Why TOON? #

Token Efficiency #

TOON significantly reduces token usage compared to other formats:

LLM Accuracy #

Benchmark results across 4 models on 209 data retrieval questions:

• TOON: 73.9% accuracy
• JSON: 69.7% accuracy
• YAML: 69.0% accuracy
• XML: 67.1% accuracy

Result: TOON achieves higher accuracy while using fewer tokens.

🎨 Advanced Features #

Custom Delimiters #

Use tabs or pipes for maximum efficiency:

// Tab delimiter - best for token efficiency
encode(data, options: EncodeOptions(delimiter: '\t'));

// Pipe delimiter - human-readable
encode(data, options: EncodeOptions(delimiter: '|'));

Flat Map #

Flatten deeply nested structures:

final config = {
  'database': {'connection': {'host': 'localhost', 'port': 5432}}
};

// Flatten
final flat = encode(
  config,
  options: EncodeOptions(enforceFlatMap: true, flatMapSeparator: '.'),
);
// database.connection.host: localhost
// database.connection.port: 5432

// Unflatten on decode
final restored = decode(
  flat,
  options: DecodeOptions(enforceFlatMap: true, flatMapSeparator: '.'),
);

Strict vs Lenient Parsing #

// Strict mode (default) - validates array lengths and structure
decode(toon, options: DecodeOptions(strict: true));

// Lenient mode - best-effort parsing
decode(toon, options: DecodeOptions(strict: false));

Length Markers #

Explicit array size validation:

encode(data, options: EncodeOptions(lengthMarker: '#'));
// tags[#3]: admin,ops,dev

🔧 CLI Tool #

Encode and decode via command line:

# Encode JSON to TOON
dart run toonx encode input.json -o output.toon

# Decode TOON to JSON
dart run toonx decode data.toon -o output.json

# Pipe from stdin
cat data.json | dart run toonx encode

# Show help
dart run toonx --help

📚 When to Use TOON #

Perfect for:

• ✅ Uniform arrays of objects (e-commerce orders, user records, logs)
• ✅ LLM prompts with structured data
• ✅ API responses for AI models
• ✅ Time-series data and analytics
• ✅ Configuration files for AI systems

Consider alternatives:

• ❌ Deeply nested, non-uniform structures (JSON may be better)
• ❌ Pure tabular data (CSV is smaller but less structured)
• ❌ Binary data or extremely large payloads

📖 Examples #

See comprehensive examples in example/example.dart:

1. Basic encoding and decoding
2. Tabular arrays for uniform data
3. Custom delimiters (tab, pipe)
4. Length markers
5. Flat map for nested structures
6. Strict vs lenient parsing
7. Real-world e-commerce example

🧪 Benchmarks #

This package includes extensive tests with real-world data:

• 100+ employee records
• E-commerce orders with nested structures
• GitHub repositories data
• Analytics time-series
• Configuration files

Run tests: dart test

📄 Specification #

This implementation follows the TOON Specification v2.0 for cross-language compatibility.

🤝 Contributing #

Contributions welcome! Please check the official TOON repository for the full specification and conformance tests.

📜 License #

MIT License - see LICENSE file for details.

🔗 Links #

• GitHub Repository
• pub.flutter-io.cn Package
• API Documentation
• Issue Tracker
• TOON Official Specification
• Benchmarks & Accuracy Tests
• Interactive Playground

Made with ❤️ for the Dart & Flutter community