Scrapy
scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
62.5KGitHub
OPENAGENTSKILL / DIRECTORY
AI Agent Skills
Find a skill for your next task. Explore tools for Codex, Claude Code, Cursor and more.
Find a skill for your next task. Preview examples where available.
Results · “crawling”
1–7 / 7
Results: 7
Katana
projectdiscovery
A next-generation crawling and spidering framework.
17.1KGitHub
DotnetSpider
dotnetcore
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
4.1KGitHub
Gain
elliotgao2
Web crawling framework based on asyncio.
2KGitHub
Geziyor
geziyor
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
2.8KGitHub
Sotawhat
chiphuyen
Returns latest research results by crawling arxiv papers and summarizing abstracts. Helps you stay afloat with so many new papers everyday.
1.4KGitHub
Hydra Link Checker
victoriadrake
Hydra: a multithreaded site-crawling link checker in Python standard library
125GitHub