Open-source LLM-friendly web crawler and scraper

Python scraper based on AI

CLI proxy that reduces LLM token consumption by 60-90% on common dev commands. Single Rust binary, zero dependencies

The Cloud Native Application Proxy

Elegant Scraper and Crawler Framework for Golang

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Python ProxyPool for web spider

A lightweight Node.js private proxy registry

Proxy server to bypass Cloudflare protection

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Turn any webpage into structured data using LLMs