#1
A collection of awesome web crawler,spider in different languages
7.2K starsJun 16, 2024 pushweb-automationCrawlerClaude Code
$ npx skills add BruceDone/awesome-crawlerAlternatives
Spidr alternatives for AI agents.
Compare similar skills by workflow fit, trust score, quality, GitHub adoption, maintenance, and install readiness.
Current skill
Spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
77
Quality
85
Trust
837
Stars
#2
Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.
1.1K starsJan 27, 2026 pushweb-automationRubyCrawler
$ npx skills add vifreefly/kimuraframework#3
Elegant Scraper and Crawler Framework for Golang
25K starsMay 25, 2026 pushweb-automationGoCrawler
$ npx skills add gocolly/colly#4
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
1.4K starsApr 7, 2026 pushweb-automationRubyCrawler
$ npx skills add felipecsl/wombat#5
Every web site provides APIs.
3.6K starsJun 4, 2026 pushweb-automationPythonCrawler
$ npx skills add elliotgao2/toapi#6
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
2.7K starsJun 10, 2026 pushweb-automationPHPCrawler
$ npx skills add jae-jae/QueryList#7
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
11K starsJun 14, 2023 pushweb-automationJavaCrawler
$ npx skills add ssssssss-team/spider-flow#8
👾 Fast and simple video download library and CLI tool written in Go
31K starsMar 29, 2026 pushweb-automationGoCrawler
$ npx skills add iawia002/lux#9
Python ProxyPool for web spider
23K starsJun 9, 2026 pushweb-automationPythonCrawler
$ npx skills add jhao104/proxy_pool#10
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
15K starsMay 13, 2026 pushweb-automationPythonCrawler
$ npx skills add codelucas/newspaper#11
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
12K starsFeb 10, 2026 pushweb-automationGoCrawler
$ npx skills add crawlab-team/crawlab#12
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
15K starsJun 28, 2025 pushweb-automationHTMLCrawler
$ npx skills add shengqiangzhang/examples-of-web-crawlers#13
Crawly, a high-level web crawling & scraping framework for Elixir.
1.1K starsJul 16, 2025 pushweb-automationElixirCrawler
$ npx skills add elixir-crawly/crawly#14
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
2.8K starsApr 16, 2025 pushweb-automationGoCrawler
$ npx skills add geziyor/geziyor#15
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
6.8K starsJun 4, 2026 pushweb-automationTypeScriptCrawler
$ npx skills add bda-research/node-crawler#16
A lightweight web crawler framework.（Java爬虫框架）
756 starsJun 13, 2026 pushweb-automationJavaCrawler
$ npx skills add xuxueli/xxl-crawlerHow to choose
When should you switch?
Use an alternative when it has a clearer install path, higher trust score, fresher maintenance, or better platform fit for your current agent stack. Keep Spidr if it already passes your workflow test and repository review.
Next step
Compare top candidates side by side
Open the compare page, test the install commands in a sandbox, and check each repository before using a skill in production.