AI Agent Skills

List of libraries, tools and APIs for web scraping and data processing.

Web crawling built for AI

Scrapy, a fast high-level web crawling & scraping framework for Python.

A Chrome DevTools Protocol driver for web automation and scraping.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Down…

The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Lighter web automation with Python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o…

A next-generation crawling and spidering framework.

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

Give your AI agent a web browser

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Python scraper based on AI

Elegant Scraper and Crawler Framework for Golang

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs: