AI Agent Skills

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

This email crawler will visit all pages of a provided website and parse and save emails found to a csv file.

Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are per…

带你了解一下Golang的市场行情

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A scalable web crawler framework for Java.

news-please - an integrated web crawler and information extractor for news that just works

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Polite, slim and concurrent web crawler.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Web crawling built for AI

Elegant Scraper and Crawler Framework for Golang

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM…

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量…