AI Agent Skills

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Audit and generate robots.txt and general crawl access for a page — verify robots.txt reachability and syntax, detect Disallow rules that block CSS/JS or important conte…

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you ca…

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

Crawl BookCorpus

API of DouYin for Humans used to Crawl Popular Videos and Musics

GitHub Search: Platform used to crawl, store and present projects from GitHub, as well as any statistics related to them

🕸️ Crawl in the web network

A command-line tool to crawl websites using puppeteer.

Crawl one public website from a supplied URL, select up to ten representative page types, score evidence-lined GEO readiness, and generate a standalone visual HTML repor…

Index and access layer audit for AI search -- the hard prerequisites. Verifies Bing indexation (the ChatGPT entry ticket), audits robots.txt and CDN-level AI-crawler acc…

BenedictKing's Agent Skills collection for Claude Code and compatible agents

Automate a real browser from the terminal for navigation, form filling, snapshots, screenshots, extraction, and UI-flow debugging.

Browser automation layer for agents that need to interact with websites.

Reliable browser automation and testing engine for web agent tasks.

Use when creating or maintaining OpenCLI site sitemaps: agent-facing navigation, page-state, action, workflow, API-reference, pitfall, and fallback knowledge for a websi…