AI Agent Skills

List of libraries, tools and APIs for web scraping and data processing.

Open-source LLM-friendly web crawler and scraper for agent workflows.

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and…

Cloud-Native distributed storage built on and for Kubernetes

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Adaptive web scraping for agent data collection

Scrapy, a fast high-level web crawling & scraping framework for Python.

Distributed reliable key-value store for the most critical data of a distributed system

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

SeaweedFS is a distributed storage system for object storage (S3), file systems, and Iceberg tables, designed to handle billions of files with O(1) disk access and effor…

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

A scalable, distributed, collaborative, document-graph database, for the realtime web

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask,…

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）

CNCF Jaeger, a Distributed Tracing Platform