AI Agent Skills

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL…

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B站视频 | 评论爬虫、微博帖子 | 评论爬虫、百度贴吧帖子 | 知乎问答文章 | 评论爬虫。支持多平台社交媒体内容抓取，提供完整的数据采集解决方案。

The only open-source toolkit that can download SEC EDGAR financial reports and extract textual data from specific item sections into nice & clean structured JSON files.…

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

A powerful browser crawler for web vulnerability scanners

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

SiteOne Crawler GUI is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engi…

AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recomme…

Collect Bilibili video and creator evidence with MediaCrawler through search, exact BV detail, comments, dynamics, contacts, and optional media. Use for topic, format, t…

Collect Douyin competitive evidence with MediaCrawler using keyword search, exact video detail, comments, media, and creator profiles. Use for trend, hook, format, audie…

Collect Kuaishou competitive evidence with MediaCrawler using keyword search, exact video detail, comments, media, and creator profiles. Use for trend, format, audience-…

Collect Baidu Tieba thread and user evidence with MediaCrawler using keyword or bar discovery, exact thread detail, replies, and creator pages. Use for community pain-po…

Collect Weibo posts and creator evidence with MediaCrawler using search, exact post IDs, comments, optional media, and creator IDs. Use for discourse, trend, messaging,…