AI Agent Skills

Turn websites into clean markdown or structured data for retrieval and agents.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

SiteOne Crawler GUI is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engi…

HTML to Markdown converter and crawler.

A very simple news crawler with a funny name

带你了解一下Golang的市场行情

Elegant Scraper and Crawler Framework for Golang

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量…

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

This is a wencai crawler.（i问财的策略回测接口的Pythonic工具包）

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you ca…

The unix-way web crawler

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realis…

Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are per…

A Facebook crawler

Crawler (Bot) searching for credential leaks on paste sites.