AI Agent Skills

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Crawl and extract (regular or onion) webpages through TOR network

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you ca…

Crawl BookCorpus

API of DouYin for Humans used to Crawl Popular Videos and Musics

GitHub Search: Platform used to crawl, store and present projects from GitHub, as well as any statistics related to them

A command-line tool to crawl websites using puppeteer.

CrawlAI RAG is an AI-powered website intelligence platform that allows users to crawl entire websites, index their content, and ask natural-language questions using Retr…

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

Automatically crawl arXiv papers daily and summarize them using AI. Illustrating them using GitHub Pages.

Crawl GitHub trending repositories, analyze with LLM for Chinese insights, categorize by themes, compute diffs against history, and generate Markdown reports. Default br…