AI Agent Skills

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Distributed crawler powered by Headless Chrome

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A scalable web crawler framework for Java.

Incredibly fast crawler designed for OSINT.

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Polite, slim and concurrent web crawler.

:zap: A distributed crawler for weibo, building with celery and requests.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Turn websites into clean markdown or structured data for retrieval and agents.

An Unofficial REST API for vlr.gg, a site for Valorant Pro Esports match results and news.

Elegant Scraper and Crawler Framework for Golang

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量…

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you ca…