sloth_link_crawler 0.0.2 copy "sloth_link_crawler: ^0.0.2" to clipboard
sloth_link_crawler: ^0.0.2 copied to clipboard

Published 2 years ago • verified publisherynnob.com
SDKDartFlutter
PlatformAndroidiOSLinuxmacOSWindows
→

Metadata

(WIP!) Web crawler to crawl all links of a website. Looking foward to respect robots.txt and use custom user agents.

More...

0.0.2 #

  • Crawl website and return found links
  • ADD optional parameters
  • ADD custom agent
  • ADD respect robots.txt of domain
  • ADD Filter to only return internal domain links
  • ADD Debug mode to print results on console
  • ADD Delay bewteen scrape calls (GET-Requests)
  • ADD Retry of max 5 on failed requests

0.0.1 #

  • Initial version.

← Metadata

3
likes
140
points
38
downloads

Publisher

verified publisherynnob.com

Weekly Downloads

Metadata

(WIP!) Web crawler to crawl all links of a website. Looking foward to respect robots.txt and use custom user agents.

Homepage
Repository (GitHub)
View/report issues

Documentation

API reference

License

BSD-3-Clause (license)

Dependencies

beautiful_soup_dart, http, http_interceptor, queue, robots_txt

More

Packages that depend on sloth_link_crawler

Back