Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Supply asset profile
Deep research, source comparison, literature review, RAG, knowledge search, and reports.
Scenario
RAG and knowledge
I need my agent to build a RAG workflow over documents and retrieve reliable context.
Agent fit
Claude Code + CLI + Codex
Codex, Claude Code, Cursor, CLI, or custom agents.
Install
Ready
npx skills add Norconex/crawler
Maintenance
fresh
7d since push
Risk
Safe to try
Quality score needs review
GitHub quality
202
80/100 quality · 80/100 trust
Coverage tags
Review notes
Quality score needs review
Agent adoption scorecard
These scores combine public repository metadata, OpenAgentSkill review signals, maintenance freshness, and install readiness. They are a shortlist signal, not a replacement for human review.
Quality
StrongSolid option that is likely worth shortlisting for production workflows.
Trust
Sandbox onlyUseful candidate with missing or mixed trust signals. Keep it in an isolated workspace until the outcome loop proves task fit.
Audit
Safe to tryInstall readiness, security metadata, maintenance, and adoption risk.
Trust Score v5
Run only in a sandbox and compare close alternatives before using it for real work.
Stars
202 GitHub stars
Repo activity
202 stars, 71 forks
Maintenance
7d since push
License
Apache-2.0
Install
npx skills add Norconex/crawler
Install safety
standard package or runtime install path
Permission surface
filesystem or document access, network or browser access
Agent outcomes
No agent outcome data yet
Docs
Strong README/SKILL.md context
Risk summary
Install readiness
Agent-readable metadata
Use this block or the embedded JSON to decide whether an agent should install this skill, choose an alternative, or ask for human review first.
Suited tasks
Suited agents
Install decision
Trust and risk
Outcome loop
Do not use when
Alternative
112.2K stars
npx skills add microsoft/generative-ai-for-beginners
Alternative
91.0K stars
npx skills add safishamsi/graphify
Alternative
77.0K stars
npx skills add elastic/elasticsearch
Alternative
75.1K stars
npx skills add Egonex-AI/Understand-Anything
Agent safety v2
Good audit and safety signals with no high-risk permission hints in public metadata.
Review the audit page, then allow agent install in a sandboxed workflow.
medium
Skill likely fetches remote pages, APIs, repositories, or external services.
medium
Skill may read or write project files, documents, generated artifacts, or local workspace state.
Install targets
Copy the registry command or an agent-specific install prompt for Codex, Claude Code, and Cursor.
Use the registry command when your workflow supports the OpenAgentSkill installer.
$ npx skills add Norconex/crawlerAgent resolve plan
The Resolve API returns the selected skill, alternatives, safety policy, audit notes, install target, and copy-paste prompt an agent can follow without scraping this page.
Resolve JSON
/api/agent/resolve?task=Use%20Crawler%20for%20an%20agent%20workflow&agent=codex&max_risk=medium
Resolve text
/api/agent/resolve?task=Use%20Crawler%20for%20an%20agent%20workflow&agent=codex&max_risk=medium&format=text
Install handoff
/api/skills/norconex-crawler/install
Agent should check
Copy prompt
Task: Use Crawler in this workspace.
Resolve first: https://www.openagentskill.com/api/agent/resolve?task=Use%20Crawler%20for%20an%20agent%20workflow&agent=codex&max_risk=medium
Review install handoff: https://www.openagentskill.com/api/skills/norconex-crawler/install
Install command: npx skills add Norconex/crawler
Before running it, summarize audit warnings, required permissions, and the fallback skill if install is risky.Agent handoff
Use the public install endpoint to fetch the command, safety checklist, target prompts, and canonical links for this skill.
Install handoff
/api/skills/norconex-crawler/install
LLM text format
/api/skills/norconex-crawler/install?format=text
Find alternatives
/api/skills/search?q=Crawler&limit=3
Agent prompt
Use Crawler for this task. Review https://www.openagentskill.com/api/skills/norconex-crawler/install, then install with: npx skills add Norconex/crawlerRegistry metadata
This page exposes the same decision, trust, audit, use-case, and install signals through the Registry API, so agents can rank this skill without scraping the UI.
Manifest
/api/registry/manifest/norconex-crawler
LLM text
/api/registry/manifest/norconex-crawler?format=text
Install alias
/api/registry/install/norconex-crawler
Recommend
/api/registry/recommend?task=Use%20Crawler%20in%20an%20agent%20workflow&limit=3
Agent fit
RAG and knowledge
Use-case tags
Platforms
Java, Search, Claude Code
Audit report
Review install readiness, maintenance, trust, quality, and metadata warnings before adding this skill to an agent workflow.
Agent decision cockpit
Shortlist this skill and compare it with close alternatives before production adoption.
Role in stack
Companion skill
Primary fit
RAG and knowledge
Trust label
Strong shortlist
Install path
Command ready
Use when
Evidence
Review first
Implementation path
Trust profile
Useful candidate with missing or mixed trust signals. Keep it in an isolated workspace until the outcome loop proves task fit.
GitHub adoption
INFO202 GitHub stars
Stars/forks activity
INFO202 stars, 71 forks; issue activity unavailable in current metadata
Recent maintenance
PASS7d since push
License clarity
PASSApache-2.0
Good signals
Review before install
Recommended action
Run only in a sandbox and compare close alternatives before using it for real work.
Quality profile
Solid option that is likely worth shortlisting for production workflows.
Workflow fit
Search private knowledge
I need my agent to build a RAG workflow over documents and retrieve reliable context.
Automate repeated work
I need my agent to automate a repeated workflow across tools and files.
Operate local tools
I need my agent to operate local files and desktop apps in a repeatable workflow.
Stack fit
Ingest, retrieve, and cite
A stack for document-heavy agents that ingest files, create searchable knowledge, retrieve relevant context, and answer with grounded sources.
Turn skills into distribution
A stack for turning newly indexed skills into SEO briefs, social drafts, comparison pages, and reusable publishing workflows.
Scrape, clean, and reuse web data
A practical stack for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.
Alternative shortlist
Similar skills in this category, ranked with the same readiness and quality signals.
21 Lessons, Get Started Building with Generative AI
AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, and more). Turn any folder of code, SQL schemas, R scripts, shell scripts, docs, papers, images, or videos into a queryable knowledge graph. App code + database schema + infrastructure in one graph.
Free and Open Source, Distributed, RESTful Search Engine
Graphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Imported by the skill-only GitHub discovery pipeline because it matches agent skill, automation, domain workflow, RAG, document-processing, data, finance, security, or developer-tool signals. Protocol-server projects are excluded from automated imports.
Frameworks & Tools
Decision snapshot
recent repository activity
Audit snapshot
Install and adoption review
Agent-proven evidence
Outcome reports after resolve, review, install, and one narrow run.
No agent outcome data yet. The first agent run can report success, setup needs, risk blocks, failure, or not-relevant through /api/agent/outcome.
Install
Free and open source. Review the audit before production use.
Growth loop
Scenario-led draft for Crawler, ready for a manual X post.
Most web agents fail in the boring part: messy pages, missing context, repeatable extraction. Crawler gives agents a cleaner path to browse, extract, and monitor web pages. 202 stars https://www.openagentskill.com/skills/norconex-crawler?ref=x #AIAgents
Listing + install path for Crawler: https://www.openagentskill.com/skills/norconex-crawler?ref=x Install: npx skills add Norconex/crawler
Listing source
This listing was indexed from public sources and is not marked official until a maintainer claim is approved.
Attribution links to the public repository or creator profile. Creators can claim the listing to update ownership signals.
Claim this skillOwner claim
This community indexed listing is attributed to Norconex but is not marked official yet. Claim it to add a verified owner signal and make future launch, install, and audit updates easier to trust.
Creator backlink kit
Show the canonical listing, current trust and audit signals, and real Agent Proven evidence where developers evaluate the repository.
[](https://www.openagentskill.com/skills/norconex-crawler)
[](https://www.openagentskill.com/skills/norconex-crawler)
[](https://www.openagentskill.com/skills/norconex-crawler/audit)
[](https://www.openagentskill.com/skills/norconex-crawler)Norconex
@norconex
Share whether this skill looks useful for your agent workflow. Aggregated feedback improves rankings over time.
Sandbox only
Generative AI For Beginners
21 Lessons, Get Started Building with Generative AI
112.2K stars · 0 installsGraphify
AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, and more). Turn any folder of code, SQL schemas, R scripts, shell scripts, docs, papers, images, or videos into a queryable knowledge graph. App code + database schema + infrastructure in one graph.
91.0K stars · 0 installsElasticsearch
Free and Open Source, Distributed, RESTful Search Engine
77.0K stars · 0 installsUnderstand Anything
Graphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
75.1K stars · 0 installs