Scraper class

Handles HTTP requests and HTML fetching for web scraping operations.

This class is responsible for:

  • Building HTTP headers with appropriate user agents and cookies
  • Cleaning URLs based on configuration
  • Fetching HTML content from web pages
  • Handling proxy requests when configured
  • Managing HTML parsing and document creation

The scraper can work with pre-fetched HTML or fetch it automatically based on the scraper configuration settings.

Constructors

Scraper.new()

Properties

hashCode → int
The hash code for this object.
no setterinherited
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited

Methods

noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
scrape({required Uri url, required ScraperConfig scraperConfig, Document? html, Map<String, String>? overrideCookies, String? overrideUserAgent, Map<String, String>? overrideHeaders, ProxyAPIConfig? overrideProxyAPIConfig, bool debug = false}) → Future<Data>
Fetches HTML content from a URL or uses provided HTML document.
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited