AI Agent Skills

Open-source LLM-friendly web crawler and scraper for agent workflows.

Turn websites into clean markdown or structured data for retrieval and agents.

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

Crawl and extract (regular or onion) webpages through TOR network

Flexible Node.js AI-assisted crawler library

AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recomme…

Every web site provides APIs.

Use when driving a website with opencli browser and sitemap context is available, requested, or needed to avoid blind navigation. Guides agents to consume site sitemap f…

Use when creating or maintaining OpenCLI site sitemaps: agent-facing navigation, page-state, action, workflow, API-reference, pitfall, and fallback knowledge for a websi…

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you ca…

Run Google Lighthouse on your entire site.

Jekyll-based static site for The Programming Historian

Browser automation layer for agents that need to interact with websites.

Automate a real browser from the terminal for navigation, form filling, snapshots, screenshots, extraction, and UI-flow debugging.