AI Agent Skills

Scrapy, a fast high-level web crawling & scraping framework for Python.

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

Crawly, a high-level web crawling & scraping framework for Elixir.

a reliable high-level web crawling & scraping framework for Node.js.

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

A next-generation crawling and spidering framework.

Web crawling built for AI

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Web crawling framework based on asyncio.

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM…

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Full-site crawling, scraping, and site mapping via Firecrawl MCP. Use when user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site stru…

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

Golang Crawling and scraping framework