Awesome Web Scraping
lorien
List of libraries, tools and APIs for web scraping and data processing.
7.9KGitHub
OPENAGENTSKILL / DIRECTORY
AI Agent Skills
Find a skill for your next task. Explore tools for Codex, Claude Code, Cursor and more.
Find a skill for your next task. Preview examples where available.
Results · “web-crawling”
1–16 / 32
Results: 32
Crawl4AI
unclecode
Web crawling built for AI
73.1KGitHub
Scrapy
scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
62.5KGitHub
Rod
go-rod
A Chrome DevTools Protocol driver for web automation and scraping.
7KGitHub
Crawlee
apify
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Down…
25.7KGitHub
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
2.9KGitHub
Autoscraper
alirezamika
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
7.9KGitHub
Helium
mherrmann
Lighter web automation with Python
8.3KGitHub
Crawlee Python
apify
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o…
9.5KGitHub
Katana
projectdiscovery
A next-generation crawling and spidering framework.
17.1KGitHub
Maxun
getmaxun
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
16.2KGitHub
Browser Use
browser-use
Give your AI agent a web browser
100.5KGitHub
Midscene
web-infra-dev
AI-powered, vision-driven UI automation for every platform.
14.2KGitHub
Trafilatura
adbar
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
6.4KGitHub
Scrapegraph AI
ScrapeGraphAI
27.2KGitHub
Colly
gocolly
Elegant Scraper and Crawler Framework for Golang
25.3KGitHub