AI Agent Skills

Run a high-fidelity browser-based web archiving crawler in a single Docker container

A powerful browser crawler for web vulnerability scanners

Scrapy, a fast high-level web crawling & scraping framework for Python.

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频

Random User-Agent middleware based on fake-useragent

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineer…

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B站视频 | 评论爬虫、微博帖子 | 评论爬虫、百度贴吧帖子 | 知乎问答文章 | 评论爬虫。支持多平台社交媒体内容抓取，提供完整的数据采集解决方案。

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Distributed crawler powered by Headless Chrome

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量…

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

A scalable web crawler framework for Java.

news-please - an integrated web crawler and information extractor for news that just works

Easy to use lightweight web crawler（易用的轻量化网络爬虫）