Open-source LLM-friendly web crawler and scraper

🌐 Make websites accessible for AI agents. Automate tasks online with ease.

Deploy applications and websites to Vercel, including preview deployments and shareable deployment links.

Adaptive web scraping for agent data collection

Scrapy, a fast high-level web crawling & scraping framework for Python.

Make AI agents interact with websites using natural language

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Empowering People Ethically 🚀 — Matomo is hiring! Join us → https://matomo.org/jobs Matomo is the leading open-source alternative to Google Analytics, giving you complete control and built-in privacy. Easily collect, visualise, and analyse data from websites & apps. Star us on GitHub ⭐️ – Pull Requests welcome!

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

The headless browser for AI agents and web scraping

WPScan WordPress security scanner. Written for security professionals and blog maintainers to test the security of their WordPress websites. Contact us via contact@wpscan.com

🪐 Markdown with superpowers: from ideas to papers, presentations, websites, books, and knowledge bases.