浏览器与自动化
agent-ready-ai-rules
Sub-skill de agent-ready-cloudflare: Implement AI Bot Rules in robots.txt
fabricioctelles · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add fabricioctelles/skills --skill agent-ready-ai-rules技能说明
name: agent-ready-ai-rules description: > Sub-skill de agent-ready-cloudflare: Implement AI Bot Rules in robots.txt
Implement AI Bot Rules in robots.txt
Add explicit User-agent entries for AI crawlers in your robots.txt per
RFC 9309.
Requirements
- Add
User-agentblocks for AI-specific bots:GPTBot,OAI-SearchBot,Claude-Web,Google-Extended,Amazonbot,anthropic-ai,Bytespider,CCBot,Applebot-Extended - Set
Allowand/orDisallowrules matching your content policy - A wildcard
User-agent: *block alone is not sufficient — explicit AI bot entries are required
Cloudflare
AI Crawl Control lets you manage AI bot rules from the dashboard without editing robots.txt manually.
Validate
POST https://isitagentready.com/api/scan
Content-Type: application/json
{"url": "https://YOUR-SITE.com"}
Check that checks.botAccessControl.robotsTxtAiRules.status is "pass".