AI Agent Skills

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B站视频 | 评论爬虫、微博帖子 | 评论爬虫、百度贴吧帖子 | 知乎问答文章 | 评论爬虫。支持多平台社交媒体内容抓取，提供完整的数据采集解决方案。

A multi-thread crawler framework with many builtin image crawlers provided.

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Distributed crawler powered by Headless Chrome

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A scalable web crawler framework for Java.

news-please - an integrated web crawler and information extractor for news that just works

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Polite, slim and concurrent web crawler.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Turn websites into clean markdown or structured data for retrieval and agents.

Web crawling built for AI

Elegant Scraper and Crawler Framework for Golang

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM…

AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recomme…