Crawler
https://spatie.be/docs/crawler
Install with one command
$ npx skills add spatie/crawlerBest for
Web scraping
Find skills for crawling websites, extracting structured data, monitoring pages, and turning messy web content into agent-ready inputs.
Choose it when
- You want a GitHub-backed skill with 2.8K stars.
- You need a reusable install command for agents.
- You want to compare it with related marketplace skills.
Check before install
- Pushed 4d ago
- License: MIT
- Review the repository README and examples.
Quality profile
Excellent candidate for agent workflows
High-confidence pick with strong adoption and healthy maintenance signals.
Workflow fit
Use this skill in these scenarios
Collect structured data
Web scraping
I need my agent to scrape websites and extract structured data from pages.
Build and ship code
Coding agents
I need a coding agent that can understand a repository, edit code, and review pull requests.
Manage repositories
GitHub automation
I need my agent to triage GitHub issues, review pull requests, and summarize repository changes.
Stack fit
Add it to a complete workflow
Scrape, clean, and reuse web data
Web data pipeline
A practical stack for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.
Inspect, patch, and verify code
Coding review agent
A stack for software agents that inspect repositories, review pull requests, generate tests, and turn findings into shippable patches.
Turn skills into distribution
Content growth agent
A stack for turning newly indexed skills into SEO briefs, social drafts, comparison pages, and reusable publishing workflows.
Overview
https://spatie.be/docs/crawler
Imported by the skill-only GitHub discovery pipeline because it matches agent skill, automation, RAG, or developer-tool signals. Protocol-server projects are excluded from automated imports.
Platform Compatibility
Technical Details
- Version
- 1.0.0
- License
- MIT
- Last Updated
- 5/23/2026
- Published
- 5/23/2026
Frameworks & Tools
Author
spatie✓
@spatie
Platform Fit
Health Signals
- GitHub stars
- 2.8K
- Quality score
- 66/100
- Last GitHub push
- May 18, 2026
- Framework hints
- 2
Community Signal
Share whether this skill looks useful for your agent workflow. Aggregated feedback improves rankings over time.
Trust & Safety
- —Open source (public GitHub repo)
- —AI static analysis passed
- —License: MIT
- —Manually verified by team
Related Skills
Crawl4AI
Open-source LLM-friendly web crawler and scraper
66.1K stars · 31.0K installsScrapy
High-throughput crawling and scraping for agent data pipelines
61.8K stars · 0 installsScrapling
Adaptive web scraping for agent data collection
53.2K stars · 0 installsEasySpider
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
43.9K stars · 0 installs