ScraperConfig class

Configuration for targeting and scraping specific types of URLs.

This class defines how to scrape a particular website or URL pattern. It contains all the necessary settings for:

  • URL pattern matching to determine when to use this configuration
  • Parser definitions for extracting data from the page
  • HTTP request settings (user agent, headers, etc.)
  • URL preprocessing and cleaning rules
  • HTML fetching behavior control

Multiple ScraperConfig instances can be used together to handle different pages on the same website or different websites entirely.

Constructors

ScraperConfig.new({List<String> pathPatterns = const [], bool requiresHtml = true, UrlCleaner? urlCleaner, required List<Parser> parsers, ProxyAPIConfig? proxyAPIConfig, Map<String, String>? cookies, Map<String, String>? headers, bool forceRefresh = false, UserAgentDevice userAgent = UserAgentDevice.mobile})
Creates a new ScraperConfig instance.
ScraperConfig.fromJson(String json)
Creates a ScraperConfig instance from a JSON string.
factory
ScraperConfig.fromMap(Map<String, dynamic> map)
Creates a ScraperConfig instance from a Map.
factory

Properties

cookies ↔ Map<String, String>?
Cookies to include in the HTTP request.
getter/setter pair
forceRefresh ↔ bool
Whether to force a fresh HTTP request even if HTML is provided.
getter/setter pair
hashCode → int
The hash code for this object.
no setterinherited
headers ↔ Map<String, String>?
Headers to include in the HTTP request.
getter/setter pair
parsers ↔ List<Parser>
List of parsers that define how to extract data from the page.
getter/setter pair
pathPatterns ↔ List<String>
List of URL path patterns that this configuration should handle.
getter/setter pair
proxyAPIConfig ↔ ProxyAPIConfig?
Proxy API configuration.
getter/setter pair
requiresHtml ↔ bool
Whether HTML content needs to be fetched from the URL.
getter/setter pair
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
urlCleaner ↔ UrlCleaner?
URL preprocessing and cleaning configuration.
getter/setter pair
userAgent ↔ UserAgentDevice
User agent device type for HTTP requests.
getter/setter pair

Methods

noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
toJson() → String
Converts the ScraperConfig instance to a JSON string.
toMap() → Map<String, dynamic>
Converts the ScraperConfig instance to a Map.
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited