AI Agent Skills

Flexible Node.js AI-assisted crawler library

Turn websites into clean markdown or structured data for retrieval and agents.

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

Open-source LLM-friendly web crawler and scraper for agent workflows.

Crawl and extract (regular or onion) webpages through TOR network

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Crawl telegra.ph searching for nudes!

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recomme…

Master orchestrator for 33 Firecrawl skills. Routes to the optimal specialized skill based on task context. Use for any web scraping, research, extraction, or F

Acquire a page or site into the shared PageSnapshot every audit module reads — real HTTP status chain, headers, robots directives, raw HTML on disk, parsed document, opt…

Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are per…

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you ca…

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.