AI Agent Skills

Open-source LLM-friendly web crawler and scraper for agent workflows.

Turn websites into clean markdown or structured data for retrieval and agents.

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Crawl and extract (regular or onion) webpages through TOR network

Flexible Node.js AI-assisted crawler library

Crawl telegra.ph searching for nudes!

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recomme…

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you ca…

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

Acquire a page or site into the shared PageSnapshot every audit module reads — real HTTP status chain, headers, robots directives, raw HTML on disk, parsed document, opt…

Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are per…

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

Crawl BookCorpus