AI Agent Skills

List of libraries, tools and APIs for web scraping and data processing.

Open-source LLM-friendly web crawler and scraper for agent workflows.

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Adaptive web scraping for agent data collection

Scrapy, a fast high-level web crawling & scraping framework for Python.

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask,…

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient wa…

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

A Chrome DevTools Protocol driver for web automation and scraping.

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Down…

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM…

Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere

Web Scraping Framework

Simple web scraping for R

Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username