Go Dork alternatives for AI agents.

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

Elegant Scraper and Crawler Framework for Golang

👾 Fast and simple video download library and CLI tool written in Go

Declarative web scraping

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

A next-generation crawling and spidering framework.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A powerful browser crawler for web vulnerability scanners

Open source SEO audit tool.

Gospider - Fast web spider written in Go

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

Python ProxyPool for web spider

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

A scalable web crawler framework for Java.

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )