Fetchbot alternatives for AI agents.

Elegant Scraper and Crawler Framework for Golang

A next-generation crawling and spidering framework.

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

Polite, slim and concurrent web crawler.

👾 Fast and simple video download library and CLI tool written in Go

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Declarative web scraping

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Gospider - Fast web spider written in Go

Python ProxyPool for web spider

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

A scalable web crawler framework for Java.

Web Crawler/Spider for NodeJS + server-side jQuery ;-)