跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

agent-ready-robots-txt

Sub-skill de agent-ready-cloudflare: Implement robots.txt

fabricioctelles · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add fabricioctelles/skills --skill agent-ready-robots-txt

技能说明


name: agent-ready-robots-txt description: > Sub-skill de agent-ready-cloudflare: Implement robots.txt

Implement robots.txt

Publish a valid robots.txt at your site root per RFC 9309.

Requirements

  • Serve /robots.txt as text/plain with HTTP 200
  • Include User-agent directives with Allow/Disallow rules
  • Reference your sitemap if one exists: Sitemap: https://example.com/sitemap.xml

Cloudflare

AI Crawl Control can manage your robots.txt from the dashboard, including AI-specific bot rules.

Validate

POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}

Check that checks.discoverability.robotsTxt.status is "pass".