Scrape, clean, and reuse web data
A practical stack for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.
Built for Growth, research, and data teams building repeatable web collection workflows.
Outcomes
Workflow map
Start with a crawler or browser skill that can discover and fetch target pages.
Use extraction skills to turn HTML, tables, and page metadata into structured text.
Add checks for freshness, duplicates, blocked pages, and schema consistency.
Send clean output into reports, databases, or knowledge-base ingestion.
Recommended stack
Ranked by workflow relevance, quality score, GitHub adoption, and maintenance freshness.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawleeThe API to search, scrape, and interact with the web at scale. 🔥
$ npx skills add firecrawl/firecrawlCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ npx skills add apify/crawlee-python$ npx skills add ScrapeGraphAI/Scrapegraph-aiOpen-source LLM-friendly web crawler and scraper
$ npx skills add unclecode/crawl4ai🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
$ npx skills add getmaxun/maxunElegant Scraper and Crawler Framework for Golang
$ npx skills add gocolly/collynewspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
$ npx skills add codelucas/newspaperIdeal for
Avoid when