Best web scraping skills for AI agents

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Excellent quality, 24K stars, and a 37 use-case fit score.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Excellent quality, 9.2K stars, and a 36 use-case fit score.

The API to search, scrape, and interact with the web at scale. 🔥

Excellent quality, 132K stars, and a 34 use-case fit score.

Python scraper based on AI

Excellent quality, 27K stars, and a 33 use-case fit score.

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

Excellent quality, 16K stars, and a 31 use-case fit score.

Open-source LLM-friendly web crawler and scraper

Excellent quality, 66K stars, and a 31 use-case fit score.

👾 Fast and simple video download library and CLI tool written in Go

Excellent quality, 31K stars, and a 30 use-case fit score.

Elegant Scraper and Crawler Framework for Golang

Excellent quality, 25K stars, and a 30 use-case fit score.

Adaptive web scraping for agent data collection

Excellent quality, 54K stars, and a 29 use-case fit score.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

Excellent quality, 15K stars, and a 29 use-case fit score.

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

Excellent quality, 3.2K stars, and a 27 use-case fit score.

Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.

Excellent quality, 3.0K stars, and a 27 use-case fit score.

The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.

Excellent quality, 2.9K stars, and a 27 use-case fit score.

Node.js scraper to get data from Google Play

Excellent quality, 2.9K stars, and a 27 use-case fit score.

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Excellent quality, 2.7K stars, and a 27 use-case fit score.

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

Excellent quality, 44K stars, and a 27 use-case fit score.

Declarative web scraping

Excellent quality, 6.0K stars, and a 27 use-case fit score.

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

Excellent quality, 2.1K stars, and a 27 use-case fit score.

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

Excellent quality, 6.9K stars, and a 25 use-case fit score.

The headless browser for AI agents and web scraping

Excellent quality, 16K stars, and a 25 use-case fit score.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Excellent quality, 6.1K stars, and a 25 use-case fit score.

Make AI agents interact with websites using natural language

Excellent quality, 97K stars, and a 25 use-case fit score.

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B站视频 | 评论爬虫、微博帖子 | 评论爬虫、百度贴吧帖子 | 知乎问答文章 | 评论爬虫。支持多平台社交媒体内容抓取，提供完整的数据采集解决方案。

Excellent quality, 50K stars, and a 25 use-case fit score.

Every web site provides APIs.

Excellent quality, 3.6K stars, and a 24 use-case fit score.

List of libraries, tools and APIs for web scraping and data processing.

Excellent quality, 7.9K stars, and a 24 use-case fit score.

Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.

Excellent quality, 2.4K stars, and a 24 use-case fit score.

Python ProxyPool for web spider

Excellent quality, 23K stars, and a 23 use-case fit score.

A next-generation crawling and spidering framework.

Excellent quality, 17K stars, and a 23 use-case fit score.

📊 Python's all-in-one framework for web crawling, scraping, and testing. Supports pytest. CDP Mode provides stealth. Includes many tools.

Excellent quality, 13K stars, and a 23 use-case fit score.

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

Excellent quality, 11K stars, and a 22 use-case fit score.