Collect structured data
Crawl a documentation site
Find skills for crawling docs, converting HTML to markdown, preserving links, and preparing agent-ready source material.
Agent prompt
Find the best skill for crawling a documentation website and converting pages into clean markdown with useful metadata.
Best first install
Firecrawl
Turn any website into LLM-ready markdown or structured data
Install with one command
$ npx skills add firecrawl/firecrawlInstall targets
Install this skill in your agent workflow
Use the public install endpoint to fetch the command, safety checklist, target prompts, and canonical links for this skill.
OpenAgentSkill CLI
Resolve policy, run the source installer safely, and report a verified install receipt.
$ npx --yes https://github.com/Leon-Drq/openagentskill/releases/download/cli-v0.2.1/openagentskill-0.2.1.tgz install firecrawlDecision guide
Use and avoid conditions
Success criteria
- Preserves source URLs
- Produces clean markdown
- Can limit crawl scope
Do not use when
- Docs block crawling
- The content is private without authorization
- You need pixel-perfect browser state
Alternatives
Compare before installing
Crawlee
823Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Crawl4AI
806Open-source LLM-friendly web crawler and scraper
Colly
783Elegant Scraper and Crawler Framework for Golang
Scrapegraph AI
776Python scraper based on AI
Scrapy
736Scrapy, a fast high-level web crawling & scraping framework for Python.
Changedetection.Io
718Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!