AI Agent Skills

Flexible Node.js AI-assisted crawler library

Turn websites into clean markdown or structured data for retrieval and agents.

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

Open-source LLM-friendly web crawler and scraper for agent workflows.

Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subsc…

Crawl and extract (regular or onion) webpages through TOR network

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Crawl telegra.ph searching for nudes!

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recomme…

Acquire a page or site into the shared PageSnapshot every audit module reads — real HTTP status chain, headers, robots directives, raw HTML on disk, parsed document, opt…

Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are per…

CrawlAI RAG is an AI-powered website intelligence platform that allows users to crawl entire websites, index their content, and ask natural-language questions using Retr…

Full-site crawling, scraping, and site mapping via Firecrawl MCP. Use when user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site stru…

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more