Browser & automation
Fetchbot
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
PuerkitoBio · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add PuerkitoBio/fetchbotAbout this skill
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Imported by the skill-only GitHub discovery pipeline because it matches agent skill, automation, domain workflow, RAG, document-processing, data, finance, security, or developer-tool signals. Protocol-server projects are excluded from automated imports.