🚀 Flutter Scrapper #

A lightweight, production-ready HTML scraper designed specifically for Flutter mobile apps (Android/iOS). Extract data from public websites with intelligent content detection, automatic caching, and beautiful formatting - all on-device with zero backend dependencies.

⭐ Key Features #

🧠 Smart Content Extraction #

• Auto-detect titles, descriptions, images, prices, and more
• Zero-configuration - works on 90% of websites out-of-the-box
• Fallback strategies - multiple selectors ensure reliable extraction
• E-commerce ready - detect prices, products, contact info

⚡ High-Performance Caching #

• 50x faster repeated requests with intelligent caching
• Persistent storage - works offline after first load
• Smart expiration - automatic cache invalidation
• Memory efficient - configurable size limits

📝 Professional Content Formatting #

• Clean text extraction with HTML tag removal
• Markdown conversion for documentation apps
• Readability mode - removes ads, navigation, clutter
• Reading time estimation and word counting

🎯 Custom Extraction #

• Tag-based extraction with class and ID filtering
• Regex-powered content matching
• Flexible querying for any HTML structure
• Multiple fallback strategies

🛡️ Production Features #

• Retry logic with exponential backoff
• Timeout protection and cancellation support
• Error handling with detailed exception types
• Encoding support (UTF-8, Latin-1, fallback)
• Resource management with proper disposal

📱 Platform Support #

🚀 Quick Start #

Installation #

dependencies:
  flutter_scrapper: ^0.1.0

Basic Usage #

import 'package:flutter_scrapper/mobile_scraper.dart';

final scraper = MobileScraper(url: 'https://example.com');
await scraper.load();

// Smart extraction (NEW!)
final smartContent = scraper.extractSmartContent();
print('Title: ${smartContent.title}');
print('Description: ${smartContent.description}');
print('Images: ${smartContent.images}');
print('Prices: ${smartContent.prices}');

// Traditional extraction
final headings = scraper.queryAll(tag: 'h1');
print('Headings: $headings');

🧠 Smart Content Extraction #

Revolutionary auto-detection - no more guessing selectors!

// Extract everything automatically
final content = scraper.extractSmartContent();

// Access structured data
print('📰 Title: ${content.title}');
print('📄 Description: ${content.description}'); 
print('👤 Author: ${content.author}');
print('📅 Date: ${content.publishDate}');
print('🖼️ Images: ${content.images.length} found');
print('🔗 Links: ${content.links.length} found');
print('💰 Prices: ${content.prices}'); // E-commerce ready!

// Open Graph metadata
if (content.openGraph != null) {
  print('🌐 Site: ${content.openGraph!.siteName}');
  print('🖼️ OG Image: ${content.openGraph!.image}');
}

// Extract specific content types
final title = scraper.extractTitle();
final images = scraper.extractImages();
final emails = scraper.extractEmails();
final phones = scraper.extractPhoneNumbers();

🎯 Custom Tag Extraction #

Powerful tag-based extraction with CSS class and ID filtering:

await scraper.load();

// Basic tag extraction
final allHeadings = scraper.queryAll(tag: 'h1');
final allParagraphs = scraper.queryAll(tag: 'p');
final allLinks = scraper.queryAll(tag: 'a');

// Extract with CSS class filtering
final headlines = scraper.queryAll(tag: 'h1', className: 'headline');
final articles = scraper.queryAll(tag: 'div', className: 'article-content');
final prices = scraper.queryAll(tag: 'span', className: 'price');

// Extract with ID filtering
final mainContent = scraper.queryAll(tag: 'div', id: 'main-content');
final sidebar = scraper.queryAll(tag: 'div', id: 'sidebar');

// Combine class and ID filtering
final specificElement = scraper.queryAll(
  tag: 'div', 
  className: 'content', 
  id: 'article-123'
);

// Get only the first match
final firstHeading = scraper.query(tag: 'h1');
final firstPrice = scraper.query(tag: 'span', className: 'price');

// Real-world examples
final newsHeadlines = scraper.queryAll(tag: 'h2', className: 'news-title');
final productPrices = scraper.queryAll(tag: 'div', className: 'price-box');
final authorNames = scraper.queryAll(tag: 'span', className: 'author');
final publishDates = scraper.queryAll(tag: 'time', className: 'publish-date');

🔍 Regex-Based Extraction #

Advanced pattern matching for complex content extraction:

await scraper.load();

// Extract prices with regex
final prices = scraper.queryWithRegex(pattern: r'\$(\d+\.\d{2})');
print('Prices found: $prices'); // ['29.99', '199.99', '5.49']

// Extract email addresses
final emails = scraper.queryWithRegex(
  pattern: r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
);

// Extract phone numbers
final phones = scraper.queryWithRegex(
  pattern: r'\+?1?[-.\s]?\(?(\d{3})\)?[-.\s]?(\d{3})[-.\s]?(\d{4})'
);

// Extract dates in various formats
final dates = scraper.queryWithRegex(
  pattern: r'(\d{1,2}[/-]\d{1,2}[/-]\d{4})'
);

// Extract specific content between tags
final quotations = scraper.queryWithRegex(
  pattern: r'<blockquote[^>]*>(.*?)</blockquote>',
  group: 1  // Extract content inside the tags
);

// Extract URLs from href attributes
final links = scraper.queryWithRegex(
  pattern: r'href=["\'](https?://[^"\']+)["\']'
);

// Get only the first match
final firstPrice = scraper.queryWithRegexFirst(pattern: r'\$(\d+\.\d{2})');

// Complex patterns for structured data
final productInfo = scraper.queryWithRegex(
  pattern: r'Product:\s*([^<]+)<br>Price:\s*\$(\d+\.\d{2})'
);

// Extract content with lookahead/lookbehind
final socialLinks = scraper.queryWithRegex(
  pattern: r'(?:twitter|facebook|instagram)\.com/([a-zA-Z0-9_]+)'
);

// Extract numbers with optional formatting
final statistics = scraper.queryWithRegex(
  pattern: r'(\d{1,3}(?:,\d{3})*(?:\.\d+)?)\s*(?:views|likes|shares)'
);

⚡ High-Performance Caching #

10-50x faster with automatic caching:

// Automatic caching (default: enabled)
await scraper.load(); // First load: network request
await scraper.load(); // Second load: instant from cache!

// Cache management
final isCached = await scraper.isCached();
await scraper.removeFromCache();

// Global cache operations
await MobileScraper.clearAllCache();
final stats = MobileScraper.getCacheStats();
print('📊 Cache: ${stats.entryCount} entries, ${stats.totalSizeMB.toStringAsFixed(2)}MB');

// Configure caching
await CacheManager.instance.initialize(
  config: CacheConfig(
    maxSizeMB: 100,
    defaultExpiry: Duration(hours: 2),
    maxEntries: 500,
  ),
);

📝 Professional Content Formatting #

Beautiful, clean content in any format:

// Plain text (clean, readable)
final cleanText = scraper.toPlainText();

// Markdown format
final markdown = scraper.toMarkdown();
print(markdown);
// Output:
// # Main Title
// ## Subtitle
// This is **bold** and *italic* text.
// - List item 1
// - List item 2
// [Link text](https://example.com)

// Readability mode (removes ads, navigation)
final readable = scraper.getReadableContent();

// Custom formatting
final formatted = scraper.formatContent(ContentFormat.markdown);

// Content analysis
final wordCount = scraper.getWordCount();
final readingTime = scraper.estimateReadingTime();
print('📖 ${wordCount} words, ~${readingTime.inMinutes} min read');

// Extract specific elements
final specificContent = scraper.extractSpecificContent();
print('📑 Headings: ${specificContent['headings']}');
print('🔗 Links: ${specificContent['links']}');
print('📊 Tables: ${specificContent['tables']}');

// Clean content with specific formatting
final cleanHeadings = scraper.getCleanContent(
  tag: 'h1',
  className: 'title', 
  format: ContentFormat.markdown
);

⚙️ Advanced Configuration #

final scraper = MobileScraper(
  url: 'https://example.com',
  config: ScraperConfig(
    timeout: Duration(seconds: 15),
    maxContentSize: 10 * 1024 * 1024, // 10MB limit
    userAgent: 'MyApp/1.0',
    headers: {
      'Accept-Language': 'en-US,en;q=0.9',
      'Accept-Encoding': 'gzip, deflate',
    },
    retryConfig: RetryConfig(
      maxAttempts: 3,
      baseDelay: Duration(seconds: 1),
      maxDelay: Duration(seconds: 10),
      backoffMultiplier: 2.0,
    ),
  ),
);

🎯 Complete Example #

import 'package:flutter/material.dart';
import 'package:flutter_scrapper/mobile_scraper.dart';

class WebScrapingPage extends StatefulWidget {
  @override
  _WebScrapingPageState createState() => _WebScrapingPageState();
}

class _WebScrapingPageState extends State<WebScrapingPage> {
  final _scraper = MobileScraper(url: 'https://httpbin.org/html');
  SmartContent? _content;
  bool _loading = false;
  String? _error;

  Future<void> _scrapeContent() async {
    setState(() {
      _loading = true;
      _error = null;
    });

    try {
      // Load with caching
      await _scraper.load(useCache: true);
      
      // Smart extraction
      final content = _scraper.extractSmartContent();
      
      // Custom tag extraction
      final headlines = _scraper.queryAll(tag: 'a', className: 'story-link');
      final scores = _scraper.queryWithRegex(pattern: r'(\d+)\s*point');
      
      setState(() {
        _content = content;
        _loading = false;
      });
    } on ScraperException catch (e) {
      setState(() {
        _error = e.message;
        _loading = false;
      });
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: Text('Smart Web Scraper')),
      body: Column(
        children: [
          ElevatedButton(
            onPressed: _loading ? null : _scrapeContent,
            child: _loading 
              ? CircularProgressIndicator() 
              : Text('Extract Content'),
          ),
          if (_error != null)
            Container(
              color: Colors.red[100],
              padding: EdgeInsets.all(16),
              child: Text('Error: $_error', style: TextStyle(color: Colors.red)),
            ),
          if (_content != null)
            Expanded(
              child: ListView(
                padding: EdgeInsets.all(16),
                children: [
                  _buildInfoCard('📰 Title', _content!.title),
                  _buildInfoCard('📄 Description', _content!.description),
                  _buildInfoCard('👤 Author', _content!.author),
                  _buildInfoCard('📅 Date', _content!.publishDate),
                  _buildListCard('🖼️ Images', _content!.images),
                  _buildListCard('💰 Prices', _content!.prices),
                  _buildListCard('📧 Emails', _content!.emails),
                  _buildListCard('📞 Phones', _content!.phoneNumbers),
                ],
              ),
            ),
        ],
      ),
    );
  }

  Widget _buildInfoCard(String title, String? content) {
    if (content == null || content.isEmpty) return SizedBox.shrink();
    
    return Card(
      margin: EdgeInsets.only(bottom: 8),
      child: ListTile(
        title: Text(title, style: TextStyle(fontWeight: FontWeight.bold)),
        subtitle: Text(content),
      ),
    );
  }

  Widget _buildListCard(String title, List<String> items) {
    if (items.isEmpty) return SizedBox.shrink();
    
    return Card(
      margin: EdgeInsets.only(bottom: 8),
      child: ExpansionTile(
        title: Text('$title (${items.length})'),
        children: items.take(5).map((item) => 
          ListTile(
            dense: true,
            leading: Icon(Icons.arrow_right),
            title: Text(item, maxLines: 2, overflow: TextOverflow.ellipsis),
          )
        ).toList(),
      ),
    );
  }

  @override
  void dispose() {
    _scraper.dispose();
    super.dispose();
  }
}

🎨 Content Formatting Examples #

HTML Input: #

<article>
  <h1>Breaking News</h1>
  <p>This is <strong>important</strong> news about <em>technology</em>.</p>
  <ul>
    <li>Point 1</li>
    <li>Point 2</li>
  </ul>
</article>

Plain Text Output: #

Breaking News

This is important news about technology.

• Point 1
• Point 2

Markdown Output: #

# Breaking News

This is **important** news about *technology*.

- Point 1
- Point 2

🛡️ Error Handling #

try {
  await scraper.load();
  final content = scraper.extractSmartContent();
} on NetworkException catch (e) {
  print('Network error: ${e.message}');
} on TimeoutException catch (e) {
  print('Request timed out after ${e.timeout}');
} on ParseException catch (e) {
  print('Failed to parse content: ${e.message}');
} on UnsupportedPlatformException catch (e) {
  print('Platform ${e.platform} not supported');
} on ScraperException catch (e) {
  print('Scraping error: ${e.message}');
}

📊 Performance Comparison #

🔧 Complete API Reference #

Smart Content Extraction #

• extractSmartContent() → SmartContent - Extract all content types automatically
• extractTitle() → String? - Page title with fallbacks
• extractDescription() → String? - Meta description
• extractImages() → List<String> - All image URLs found
• extractLinks() → List<String> - All external links
• extractEmails() → List<String> - Email addresses found
• extractPhoneNumbers() → List<String> - Phone numbers found
• extractPrices() → List<String> - Prices (e-commerce)

Custom Tag Extraction #

• queryAll({required String tag, String? className, String? id}) → List<String> - Extract all matching elements
• query({required String tag, String? className, String? id}) → String? - Extract first matching element

Regex-Based Extraction #

• queryWithRegex({required String pattern, int group = 1}) → List<String> - Extract all regex matches
• queryWithRegexFirst({required String pattern, int group = 1}) → String? - Extract first regex match

Content Formatting #

• toPlainText() → String - Clean text without HTML
• toMarkdown() → String - Markdown formatted content
• getReadableContent() → String - Readability.js-style extraction
• formatContent(ContentFormat) → String - Custom formatting
• getCleanContent({required String tag, String? className, String? id, ContentFormat format}) → String - Format specific elements
• getWordCount() → int - Word count analysis
• estimateReadingTime() → Duration - Reading time estimation
• extractSpecificContent() → Map<String, List<String>> - Extract headings, links, images, etc.

Cache Management #

• isCached() → Future<bool> - Check if URL is cached
• removeFromCache() → Future<void> - Remove from cache
• MobileScraper.clearAllCache() → Future<void> - Clear all cache
• MobileScraper.getCacheStats() → CacheStats - Cache statistics

Core Methods #

• load({bool useCache = true}) → Future<bool> - Load webpage content
• cancel() → void - Cancel ongoing operations
• dispose() → void - Clean up resources
• rawHtml → String? - Get raw HTML content
• isLoaded → bool - Check if content is loaded

💡 Use Cases & Examples #

News & Blog Scraping #

// Extract articles automatically
final content = scraper.extractSmartContent();
final headlines = scraper.queryAll(tag: 'h2', className: 'article-title');
final bylines = scraper.queryAll(tag: 'span', className: 'author');

E-commerce Data Extraction #

// Product information
final prices = scraper.extractPrices();
final productTitles = scraper.queryAll(tag: 'h1', className: 'product-title');
final descriptions = scraper.queryAll(tag: 'div', className: 'product-description');
final ratings = scraper.queryWithRegex(pattern: r'(\d+\.\d+)\s*stars?');

Contact Information Mining #

// Business directory data
final emails = scraper.extractEmails();
final phones = scraper.extractPhoneNumbers();
final addresses = scraper.queryWithRegex(
  pattern: r'\d+\s+[A-Za-z\s]+(?:Street|St|Avenue|Ave|Road|Rd)'
);

Social Media & Forums #

// Forum posts and comments
final posts = scraper.queryAll(tag: 'div', className: 'post-content');
final usernames = scraper.queryAll(tag: 'span', className: 'username');
final timestamps = scraper.queryAll(tag: 'time');
final upvotes = scraper.queryWithRegex(pattern: r'(\d+)\s*upvotes?');

🤝 Contributing #

We welcome contributions! Please see our Contributing Guide for details.

📄 License #

MIT License - see LICENSE file for details.

Made with ❤️ for the Flutter community

🌟 Star this repo if it helped you build amazing mobile apps!