Best web scraping skills for AI agents

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Excellent quality, 24K stars, and a 37 use-case fit score.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Excellent quality, 9.2K stars, and a 36 use-case fit score.

The API to search, scrape, and interact with the web at scale. 🔥

Excellent quality, 139K stars, and a 34 use-case fit score.

Python scraper based on AI

Excellent quality, 27K stars, and a 33 use-case fit score.

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

Excellent quality, 16K stars, and a 31 use-case fit score.

Adaptive web scraping for agent data collection

Excellent quality, 68K stars, and a 30 use-case fit score.

Elegant Scraper and Crawler Framework for Golang

Excellent quality, 25K stars, and a 30 use-case fit score.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

Excellent quality, 15K stars, and a 29 use-case fit score.

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

Excellent quality, 44K stars, and a 27 use-case fit score.

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

Excellent quality, 6.9K stars, and a 25 use-case fit score.

The headless browser for AI agents and web scraping

Excellent quality, 16K stars, and a 25 use-case fit score.

Make AI agents interact with websites using natural language

Excellent quality, 101K stars, and a 25 use-case fit score.

List of libraries, tools and APIs for web scraping and data processing.

Excellent quality, 7.9K stars, and a 24 use-case fit score.

Python ProxyPool for web spider

Excellent quality, 23K stars, and a 23 use-case fit score.

A next-generation crawling and spidering framework.

Excellent quality, 17K stars, and a 23 use-case fit score.

为所有人准备的AI搞钱团队，帮你把经验和方法跑成一门生意。

Excellent quality, 8.3K stars, and a 22 use-case fit score.

Turn any webpage into structured data using LLMs

Excellent quality, 6.8K stars, and a 22 use-case fit score.

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

Excellent quality, 91K stars, and a 22 use-case fit score.

SeleniumBase is a framework for UI Testing, Web Scraping, and Stealth. Passes every bot-detection test with CDP Mode, and extends Playwright.

Excellent quality, 13K stars, and a 22 use-case fit score.

Get your documents ready for gen AI

Excellent quality, 63K stars, and a 21 use-case fit score.

Scrapy, a fast high-level web crawling & scraping framework for Python.

Excellent quality, 63K stars, and a 21 use-case fit score.

Create agents that monitor and act on your behalf. Your agents are standing by!

Excellent quality, 49K stars, and a 21 use-case fit score.

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!

Excellent quality, 32K stars, and a 21 use-case fit score.

Lightpanda: the headless browser designed for AI and automation

Excellent quality, 31K stars, and a 21 use-case fit score.

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed.

Excellent quality, 27K stars, and a 21 use-case fit score.

Make Any Website into CLI & Use your logged-in browser by AI agent.

Excellent quality, 26K stars, and a 21 use-case fit score.

Build browser agents with natural language actions

Excellent quality, 23K stars, and a 20 use-case fit score.

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

Excellent quality, 15K stars, and a 20 use-case fit score.

Python version of the Playwright testing and automation library.

Excellent quality, 15K stars, and a 20 use-case fit score.

QuestPDF is a modern library for PDF document generation. Its fluent C# API lets you design complex layouts with clean, readable code. Create documents using a flexible, component-based approach.

Excellent quality, 14K stars, and a 20 use-case fit score.