🚀 Flutter AI Scrapper 2.0 #

On-device AI web scraping for Flutter. Parses real HTML with an HTML5 DOM parser, harvests JSON-LD, Microdata, and OpenGraph for free with zero AI calls, and falls back to an on-device Gemma model or any OpenAI/Claude-compatible endpoint for schema-typed extraction.

⚡ The Four-Tier Extraction Architecture #

Web scraping on mobile must balance accuracy, battery life, token expenditure, and user privacy. flutter_ai_scrapper uses a four-tier pipeline that prioritizes deterministic execution and only invokes language models when necessary:

[Web Page]
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│ Tier 1 — Deterministic Foundation (0 AI Tokens, <2ms)       │
│ HTML5 DOM Parser • Readability Scoring • JSON-LD • Microdata│
└───────────────────────────┬─────────────────────────────────┘
                            │ (missing fields or unstructured)
                            ▼
┌─────────────────────────────────────────────────────────────┐
│ Tier 4 — Site Selector Recipes (0 AI Tokens, <0.05ms)       │
│ Cached CSS Selectors • Pure-CSS Runner • Drift Detection    │
└───────────────────────────┬─────────────────────────────────┘
                            │ (new domain or mutated layout)
                            ▼
┌─────────────────────────────────────────────────────────────┐
│ Tier 2 — Typed Schema Extraction (On-Device or Cloud)       │
│ Schema-as-Tool • BM25 Chunk Ranking • Map-Reduce Collections│
└───────────────────────────┬─────────────────────────────────┘
                            │ (free-form queries)
                            ▼
┌─────────────────────────────────────────────────────────────┐
│ Tier 3 — Natural-Language Planner (`page.ask`)              │
│ Prompt → Schema Inference • Auto-Recipe Lifecycle           │
└─────────────────────────────────────────────────────────────┘

📊 Measured Benchmarks #

Measured on a standard mobile/desktop development machine using the included benchmark suite:

📱 Platform Support #

🚀 Quick Start #

1. Installation #

dependencies:
  flutter_ai_scrapper: ^2.0.0

2. Tier 1: Zero-AI Deterministic Extraction #

Scrape articles and extract metadata with zero tokens, zero model downloads, and zero platform weight:

import 'package:flutter_ai_scrapper/flutter_ai_scrapper.dart';

// Fetch and parse with full HTML5 DOM
final page = await AiScrapper.open('https://news.ycombinator.com');

print('Title: ${page.title}');
print('First link: ${page.document.selectFirst("a")?.attributes["href"]}');

// Automatic JSON-LD, OpenGraph, and Microdata harvesting
final productSchema = Schema.object({
  'name': const Field.string(),
  'price': const Field.money(),
});
final harvest = page.extract(productSchema);
if (harvest.isComplete) {
  print('Harvested deterministically: ${harvest.data}');
}

3. Tier 2: Typed Schema Extraction (Gemma & Cloud Providers) #

When websites lack structured metadata, the schema extraction pipeline reduces the page via BM25 chunk ranking and uses local Gemma or cloud providers:

// Option A: On-Device Gemma (Zero Network, Zero Cost)
final gemmaProvider = GemmaProvider(manager: ModelManager());

// Option B: Resilient Fallback Chain with Cloud
final chain = ProviderChain(
  providers: [
    gemmaProvider,
    OpenAiProvider(baseUrl: 'http://localhost:11434/v1', model: 'llama3'), // Ollama
    AnthropicProvider(apiKey: 'sk-ant-...', model: 'claude-3-5-sonnet-latest'),
  ],
  allowCloudEgress: false, // Strict zero-egress privacy default!
);

final result = await page.extractWithAi(productSchema, provider: chain);
print('Extracted by: ${result.providerId}');
print('Data: ${result.data}');

4. Tier 3: Natural-Language Asking (page.ask) #

Ask questions in plain English. The library plans a typed schema and executes the extraction:

final result = await page.ask(
  'extract all book titles and prices in USD',
  provider: chain,
);

print(result.data);

5. Tier 4: Site-Level Selector Recipes (AI Once, CSS Forever) #

Synthesize CSS selector recipes with an AI model on page 1, and run with pure CSS for free on pages 2–N:

final recipeStore = RecipeStore();

// Page 1: Synthesizes recipe using AI
final res1 = await page1.ask('extract product name and price', provider: chain, recipeStore: recipeStore);

// Page 2+: Runs pure CSS with ZERO tokens and sub-millisecond execution!
final res2 = await page2.ask('extract product name and price', provider: chain, recipeStore: recipeStore);

🔒 Privacy & Secret Safety #

1. allowCloudEgress: false by default: Remote cloud providers are skipped unless callers explicitly opt in.
2. Secret Redaction: KeySanitizer redacts API keys and bearer tokens from all logs and error messages.
3. Local Ollama Support: Connect to local desktop Ollama instances (http://localhost:11434/v1) without sending data to any cloud provider.

🎨 Built-In Material 3 UI Components #

Zero external state-management dependencies (ValueNotifier only):

• ModelManagerSheet: Download, delete, and inspect on-device Hugging Face models.
• ProviderSettingsSheet: Configure fallback chains, test connections, and toggle cloud egress.
• ResultViewer: Tabbed viewer (Table, JSON, Markdown, Raw) with field-level Provenance Badges ([JSON-LD], [Recipe], [AI]).
• StreamingTextView: Live token streaming with collapsible reasoning/thinking blocks.
• ExtractionConsole: Interactive scraping console with real-time stage progress.

📚 Documentation Links #

• Migration Guide (1.1.0 to 2.0.0)
• On-Device Model Selection & Architecture
• Cloud Providers & Fallback Setup
• Ethics & Responsible Scraping

📄 License #

MIT License. See LICENSE for details.