Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Browser & automation

agent-ready-robots-txt

Sub-skill de agent-ready-cloudflare: Implement robots.txt

fabricioctelles · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add fabricioctelles/skills --skill agent-ready-robots-txt

About this skill


name: agent-ready-robots-txt description: > Sub-skill de agent-ready-cloudflare: Implement robots.txt

Implement robots.txt

Publish a valid robots.txt at your site root per RFC 9309.

Requirements

  • Serve /robots.txt as text/plain with HTTP 200
  • Include User-agent directives with Allow/Disallow rules
  • Reference your sitemap if one exists: Sitemap: https://example.com/sitemap.xml

Cloudflare

AI Crawl Control can manage your robots.txt from the dashboard, including AI-specific bot rules.

Validate

POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}

Check that checks.discoverability.robotsTxt.status is "pass".