Open-source LLM-friendly web crawler and scraper

Scrapy, a fast high-level web crawling & scraping framework for Python.

A next-generation crawling and spidering framework.

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Web crawling framework based on asyncio.

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

TuShare is a utility for crawling historical data of China stocks

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio python SDK for intelligent web data gathering.

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.