fetchHtml method

Future<String> fetchHtml({
  1. required String url,
  2. Map<String, String>? headers,
  3. int? timeout,
  4. int? retries,
  5. int priority = 0,
})

Fetches HTML content from the given URL

url is the URL to fetch headers are additional headers to send with the request timeout is the timeout for the request in milliseconds retries is the number of retry attempts priority is the priority of the request (higher values = higher priority)

Implementation

Future<String> fetchHtml({
  required String url,
  Map<String, String>? headers,
  int? timeout,
  int? retries,
  int priority = 0,
}) async {
  // Prepare headers with user agent
  final effectiveHeaders = {
    'User-Agent': _userAgentRotator.getRandomUserAgent(),
    ...?headers,
  };

  final userAgent = effectiveHeaders['User-Agent'];

  return _rateLimiter.execute(
    url: url,
    fn:
        () => _fetchWithRetry(
          url: url,
          headers: effectiveHeaders,
          timeout: timeout ?? _defaultTimeout,
          retries: retries ?? _maxRetries,
        ),
    userAgent: userAgent,
    priority: priority,
  );
}