AI Agent Skills

Audit and generate robots.txt and general crawl access for a page — verify robots.txt reachability and syntax, detect Disallow rules that block CSS/JS or important conte…

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you ca…

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

Crawl BookCorpus

API of DouYin for Humans used to Crawl Popular Videos and Musics

GitHub Search: Platform used to crawl, store and present projects from GitHub, as well as any statistics related to them

🕸️ Crawl in the web network

A command-line tool to crawl websites using puppeteer.

Crawl one public website from a supplied URL, select up to ten representative page types, score evidence-lined GEO readiness, and generate a standalone visual HTML repor…

Index and access layer audit for AI search -- the hard prerequisites. Verifies Bing indexation (the ChatGPT entry ticket), audits robots.txt and CDN-level AI-crawler acc…

BenedictKing's Agent Skills collection for Claude Code and compatible agents

Automate a real browser from the terminal for navigation, form filling, snapshots, screenshots, extraction, and UI-flow debugging.

Technical SEO audit with GEO-specific checks — crawlability, indexability, security, performance, SSR, and AI crawler access

Fetch, inspect, crawl, and capture live, JavaScript-rendered websites with Moli. Use when Codex needs current web content, web research, fact lookup, link following, a b…

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

SEO/GEO toolkit to analyse, crawl, parse and optimise websites & logs (Nginx & Apache)