AI Agent Skills

Scrapy, a fast high-level web crawling & scraping framework for Python.

A next-generation crawling and spidering framework.

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Web crawling framework based on asyncio.

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Returns latest research results by crawling arxiv papers and summarizing abstracts. Helps you stay afloat with so many new papers everyday.

Hydra: a multithreaded site-crawling link checker in Python standard library