Workflow recipe
A practical workflow for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.
Growth, research, and data teams building repeatable web collection workflows.
Expected outcome
Workflow map
Start with a crawler or browser skill that can discover and fetch target pages.
Use extraction skills to turn HTML, tables, and page metadata into structured text.
Add checks for freshness, duplicates, blocked pages, and schema consistency.
Send clean output into reports, databases, or knowledge-base ingestion.
Suggested capabilities
Ranked by relevance to this workflow, quality, GitHub adoption, and maintenance freshness. This is a decision guide, not a single install command.
Repository description
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawleeRepository description
The API to search, scrape, and interact with the web at scale. 🔥
$ npx skills add firecrawl/firecrawlRepository description
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawlee-pythonRepository description
Python scraper based on AI
$ npx skills add ScrapeGraphAI/Scrapegraph-aiRepository description
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
$ npx skills add getmaxun/maxunRepository description
Open-source LLM-friendly web crawler and scraper
$ npx skills add unclecode/crawl4aiRepository description
👾 Fast and simple video download library and CLI tool written in Go
$ npx skills add iawia002/luxRepository description
Elegant Scraper and Crawler Framework for Golang
$ npx skills add gocolly/collyGood fit
Not the right route when
Need a runnable bundle?
Skill packs include an install order, audit links, and a machine-readable Agent plan.
Workflow recipe
A practical workflow for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.
Growth, research, and data teams building repeatable web collection workflows.
Expected outcome
Workflow map
Start with a crawler or browser skill that can discover and fetch target pages.
Use extraction skills to turn HTML, tables, and page metadata into structured text.
Add checks for freshness, duplicates, blocked pages, and schema consistency.
Send clean output into reports, databases, or knowledge-base ingestion.
Suggested capabilities
Ranked by relevance to this workflow, quality, GitHub adoption, and maintenance freshness. This is a decision guide, not a single install command.
Repository description
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawleeRepository description
The API to search, scrape, and interact with the web at scale. 🔥
$ npx skills add firecrawl/firecrawlRepository description
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawlee-pythonRepository description
Python scraper based on AI
$ npx skills add ScrapeGraphAI/Scrapegraph-aiRepository description
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
$ npx skills add getmaxun/maxunRepository description
Open-source LLM-friendly web crawler and scraper
$ npx skills add unclecode/crawl4aiRepository description
👾 Fast and simple video download library and CLI tool written in Go
$ npx skills add iawia002/luxRepository description
Elegant Scraper and Crawler Framework for Golang
$ npx skills add gocolly/collyGood fit
Not the right route when
Need a runnable bundle?
Skill packs include an install order, audit links, and a machine-readable Agent plan.
Workflow recipe
A practical workflow for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.
Growth, research, and data teams building repeatable web collection workflows.
Expected outcome
Workflow map
Start with a crawler or browser skill that can discover and fetch target pages.
Use extraction skills to turn HTML, tables, and page metadata into structured text.
Add checks for freshness, duplicates, blocked pages, and schema consistency.
Send clean output into reports, databases, or knowledge-base ingestion.
Suggested capabilities
Ranked by relevance to this workflow, quality, GitHub adoption, and maintenance freshness. This is a decision guide, not a single install command.
Repository description
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawleeRepository description
The API to search, scrape, and interact with the web at scale. 🔥
$ npx skills add firecrawl/firecrawlRepository description
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawlee-pythonRepository description
Python scraper based on AI
$ npx skills add ScrapeGraphAI/Scrapegraph-aiRepository description
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
$ npx skills add getmaxun/maxunRepository description
Open-source LLM-friendly web crawler and scraper
$ npx skills add unclecode/crawl4aiRepository description
👾 Fast and simple video download library and CLI tool written in Go
$ npx skills add iawia002/luxRepository description
Elegant Scraper and Crawler Framework for Golang
$ npx skills add gocolly/collyGood fit
Not the right route when
Need a runnable bundle?
Skill packs include an install order, audit links, and a machine-readable Agent plan.
Workflow recipe
A practical workflow for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.
Growth, research, and data teams building repeatable web collection workflows.
Expected outcome
Workflow map
Start with a crawler or browser skill that can discover and fetch target pages.
Use extraction skills to turn HTML, tables, and page metadata into structured text.
Add checks for freshness, duplicates, blocked pages, and schema consistency.
Send clean output into reports, databases, or knowledge-base ingestion.
Suggested capabilities
Ranked by relevance to this workflow, quality, GitHub adoption, and maintenance freshness. This is a decision guide, not a single install command.
Repository description
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawleeRepository description
The API to search, scrape, and interact with the web at scale. 🔥
$ npx skills add firecrawl/firecrawlRepository description
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawlee-pythonRepository description
Python scraper based on AI
$ npx skills add ScrapeGraphAI/Scrapegraph-aiRepository description
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
$ npx skills add getmaxun/maxunRepository description
Open-source LLM-friendly web crawler and scraper
$ npx skills add unclecode/crawl4aiRepository description
👾 Fast and simple video download library and CLI tool written in Go
$ npx skills add iawia002/luxRepository description
Elegant Scraper and Crawler Framework for Golang
$ npx skills add gocolly/collyGood fit
Not the right route when
Need a runnable bundle?
Skill packs include an install order, audit links, and a machine-readable Agent plan.