Adaptive web scraping for agent data collection

Scrapy, a fast high-level web crawling & scraping framework for Python.

Turn any website into LLM-ready markdown or structured data

🕵️‍♂️ All-in-one OSINT tool for analysing any website

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!

Make Any Website into CLI & Use your logged-in browser by AI agent.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Clone any website with one command using AI coding agents

Official repository for IPython itself. Other repos in the IPython organization contain things like the website, documentation builds, etc.

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

The headless browser for AI agents and web scraping

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.