ARE YOU A SLOTH? - SLOTH LINK CRAWLER

This package is WIP! It will crawl an url and list all links. We try to respect the robots.txt file. See example on how to implement into your codebase.

Features

• Crawl URL
• List all links (internal OR all)
• Delay crawl requests
• Custom user agent

Getting started

With dart:

 $ dart pub add sloth_link_crawler

With Flutter:

 $ flutter pub add sloth_link_crawler

Usage

Import the package

import 'package:sloth_link_crawler/sloth_link_crawler.dart';

Access the extensions:

  final crawler = SlothLinkCrawler(
      // Domain to crawl
      baseUrl: 'https://example.com',
      // if true only urls of the domain will get listed
      onlyInternalDomainLinks: true,
      // if true some debugging information will be displayed in the console
      debugMode: true,
      // set the duration between crawl calls
      duration: const Duration(seconds: 2),
      // set a custom user agent
      userAgent: 'SlothLinkCrawler');

  // List of all crawled urls
  List<String> result = await crawler.crawl();