AI Agent Skills

A high performance web crawler / scraper in Elixir.

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

A collection of awesome web crawler,spider in different languages

A scalable, mature and versatile web crawler based on Apache Storm

A lightweight web crawler framework.（Java爬虫框架）

Elasticsearch File System Crawler (FS Crawler)

Run a high-fidelity browser-based web archiving crawler in a single Docker container

A multi-thread crawler framework with many builtin image crawlers provided.

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.