Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Automatically crawl arXiv papers daily and summarize them using AI. Illustrating them using GitHub Pages.

Flexible Node.js AI-assisted crawler library

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!

Crawl and extract (regular or onion) webpages through TOR network

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

SEO/GEO toolkit to analyse, crawl, parse and optimise websites & logs (Nginx & Apache)

A collection of reusable CLI tool skills for AI coding assistants like Claude Code, Codex, and Gemini CLI, covering search, thinking, code understanding, and task delegation.

A Claude Skill that automates post-launch website QA, auditing SEO, security, accessibility, performance, and more, then generates fixes and re-tests.