AI Agent Skills

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Distributed crawler powered by Headless Chrome

ACHE is a web crawler for domain-specific search.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A scalable web crawler framework for Java.

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM…

Incredibly fast crawler designed for OSINT.

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Polite, slim and concurrent web crawler.

:zap: A distributed crawler for weibo, building with celery and requests.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

a new crawler based on python with more function including Network fingerprint search

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.