A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Supply asset profile
Deep research, source comparison, literature review, RAG, knowledge search, and reports.
Scenario
RAG and knowledge
I need my agent to build a RAG workflow over documents and retrieve reliable context.
Agent fit
Claude Code + CLI + Codex
Codex, Claude Code, Cursor, CLI, or custom agents.
Install
Ready
npx skills add postmodern/spidr
Maintenance
stable
6mo since push
Risk
Needs review
Quality score needs review
GitHub quality
836
73/100 quality · 80/100 trust
Coverage tags
Review notes
Quality score needs review · Needs review
Agent adoption scorecard
These scores combine public repository metadata, OpenAgentSkill review signals, maintenance freshness, and install readiness. They are a shortlist signal, not a replacement for human review.
Quality
StrongSolid option that is likely worth shortlisting for production workflows.
Trust
Sandbox onlyUseful candidate with missing or mixed trust signals. Keep it in an isolated workspace until the outcome loop proves task fit.
Audit
Needs reviewInstall readiness, security metadata, maintenance, and adoption risk.
Trust Score v5
Run only in a sandbox and compare close alternatives before using it for real work.
Stars
836 GitHub stars
Repo activity
836 stars, 107 forks
Maintenance
6mo since push
License
MIT
Install
npx skills add postmodern/spidr
Install safety
standard package or runtime install path
Permission surface
filesystem or document access
Agent outcomes
No agent outcome data yet
Docs
Strong README/SKILL.md context
Risk summary
Install readiness
Agent-readable metadata
Use this block or the embedded JSON to decide whether an agent should install this skill, choose an alternative, or ask for human review first.
Suited tasks
Suited agents
Install decision
Trust and risk
Outcome loop
Do not use when
Agent safety v2
Usable candidate, but the agent should surface permission and audit notes before installation.
Require human approval before installing into a real workspace.
medium
Skill likely fetches remote pages, APIs, repositories, or external services.
medium
Skill may read or write project files, documents, generated artifacts, or local workspace state.
Install targets
Copy the registry command or an agent-specific install prompt for Codex, Claude Code, and Cursor.
Use the registry command when your workflow supports the OpenAgentSkill installer.
$ npx skills add postmodern/spidrAgent resolve plan
The Resolve API returns the selected skill, alternatives, safety policy, audit notes, install target, and copy-paste prompt an agent can follow without scraping this page.
Resolve JSON
/api/agent/resolve?task=Use%20Spidr%20for%20an%20agent%20workflow&agent=codex&max_risk=medium
Resolve text
/api/agent/resolve?task=Use%20Spidr%20for%20an%20agent%20workflow&agent=codex&max_risk=medium&format=text
Install handoff
/api/skills/postmodern-spidr/install
Agent should check
Copy prompt
Task: Use Spidr in this workspace.
Resolve first: https://www.openagentskill.com/api/agent/resolve?task=Use%20Spidr%20for%20an%20agent%20workflow&agent=codex&max_risk=medium
Review install handoff: https://www.openagentskill.com/api/skills/postmodern-spidr/install
Install command: npx skills add postmodern/spidr
Before running it, summarize audit warnings, required permissions, and the fallback skill if install is risky.Agent handoff
Use the public install endpoint to fetch the command, safety checklist, target prompts, and canonical links for this skill.
Install handoff
/api/skills/postmodern-spidr/install
LLM text format
/api/skills/postmodern-spidr/install?format=text
Find alternatives
/api/skills/search?q=Spidr&limit=3
Agent prompt
Use Spidr for this task. Review https://www.openagentskill.com/api/skills/postmodern-spidr/install, then install with: npx skills add postmodern/spidrRegistry metadata
This page exposes the same decision, trust, audit, use-case, and install signals through the Registry API, so agents can rank this skill without scraping the UI.
Manifest
/api/registry/manifest/postmodern-spidr
LLM text
/api/registry/manifest/postmodern-spidr?format=text
Install alias
/api/registry/install/postmodern-spidr
Recommend
/api/registry/recommend?task=Use%20Spidr%20in%20an%20agent%20workflow&limit=3
Agent fit
Web scraping
Use-case tags
Platforms
Ruby, Crawler, Claude Code
Audit report
Review install readiness, maintenance, trust, quality, and metadata warnings before adding this skill to an agent workflow.
Agent decision cockpit
Shortlist this skill and compare it with close alternatives before production adoption.
Role in stack
Companion skill
Primary fit
Web scraping
Trust label
Strong shortlist
Install path
Command ready
Use when
Evidence
Review first
Implementation path
Trust profile
Useful candidate with missing or mixed trust signals. Keep it in an isolated workspace until the outcome loop proves task fit.
GitHub adoption
INFO836 GitHub stars
Stars/forks activity
INFO836 stars, 107 forks; issue activity unavailable in current metadata
Recent maintenance
INFO6mo since push
License clarity
PASSMIT
Good signals
Review before install
Recommended action
Run only in a sandbox and compare close alternatives before using it for real work.
Quality profile
Solid option that is likely worth shortlisting for production workflows.
Workflow fit
Collect structured data
I need my agent to scrape websites and extract structured data from pages.
Build and ship code
I need a coding agent that can understand a repository, edit code, and review pull requests.
Search private knowledge
I need my agent to build a RAG workflow over documents and retrieve reliable context.
Stack fit
Scrape, clean, and reuse web data
A practical stack for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.
Turn skills into distribution
A stack for turning newly indexed skills into SEO briefs, social drafts, comparison pages, and reusable publishing workflows.
Inspect, patch, and verify code
A stack for software agents that inspect repositories, review pull requests, generate tests, and turn findings into shippable patches.
Alternative shortlist
Similar skills in this category, ranked with the same readiness and quality signals.
Open-source LLM-friendly web crawler and scraper
Adaptive web scraping for agent data collection
Scrapy, a fast high-level web crawling & scraping framework for Python.
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Imported by the skill-only GitHub discovery pipeline because it matches agent skill, automation, domain workflow, RAG, document-processing, data, finance, security, or developer-tool signals. Protocol-server projects are excluded from automated imports.
Frameworks & Tools
Decision snapshot
836 GitHub stars
Audit snapshot
Install and adoption review
Agent-proven evidence
Outcome reports after resolve, review, install, and one narrow run.
No agent outcome data yet. The first agent run can report success, setup needs, risk blocks, failure, or not-relevant through /api/agent/outcome.
Install
Free and open source. Review the audit before production use.
Growth loop
Scenario-led draft for Spidr, ready for a manual X post.
Most web agents fail in the boring part: messy pages, missing context, repeatable extraction. Spidr gives agents a cleaner path to browse, extract, and monitor web pages. 836 stars https://www.openagentskill.com/skills/postmodern-spidr?ref=x #AIAgents
Listing + install path for Spidr: https://www.openagentskill.com/skills/postmodern-spidr?ref=x Install: npx skills add postmodern/spidr
Listing source
This listing was indexed from public sources and is not marked official until a maintainer claim is approved.
Attribution links to the public repository or creator profile. Creators can claim the listing to update ownership signals.
Claim this skillOwner claim
This community indexed listing is attributed to postmodern but is not marked official yet. Claim it to add a verified owner signal and make future launch, install, and audit updates easier to trust.
Creator backlink kit
Show the canonical listing, current trust and audit signals, and real Agent Proven evidence where developers evaluate the repository.
[](https://www.openagentskill.com/skills/postmodern-spidr)
[](https://www.openagentskill.com/skills/postmodern-spidr)
[](https://www.openagentskill.com/skills/postmodern-spidr/audit)
[](https://www.openagentskill.com/skills/postmodern-spidr)postmodern
@postmodern
Share whether this skill looks useful for your agent workflow. Aggregated feedback improves rankings over time.
Sandbox only
Crawl4AI
Open-source LLM-friendly web crawler and scraper
73.1K stars · 31.0K installsScrapling
Adaptive web scraping for agent data collection
70.0K stars · 0 installsScrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
62.5K stars · 0 installsEasySpider
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
44.1K stars · 0 installs