Robots.txt Generator
Build standards-aware crawler rules with live validation, safe presets, and clear guidance.
No restrictions — every crawler may access every path.
Crawler groups
Choose who the rules apply to and which paths they may crawl.
AI crawler presets
Add a separate Disallow group for a crawler token verified against its vendor documentation. Training, search, and user-triggered access are different controls.
GPTBot (OpenAI training)
Disallowing GPTBot indicates this site's content should not be used to train OpenAI's generative AI foundation models. It has no effect on OAI-SearchBot or ChatGPT-User — each OpenAI bot is controlled independently.
Official source · checked 2026-09-02OAI-SearchBot (OpenAI search)
Disallowing OAI-SearchBot means this site will not be shown in ChatGPT's search-answer results, though pages may still appear as navigational links. It has no effect on whether content is used for model training (that's GPTBot).
Official source · checked 2026-09-02ChatGPT-User (OpenAI, user-triggered)
ChatGPT-User only fetches a page because a person directly asked ChatGPT or a Custom GPT to visit it. OpenAI's own documentation states these actions are user-initiated, so robots.txt rules may not apply the same way they do to automatic crawling — a Disallow rule here is a request, not a guarantee.
Official source · checked 2026-09-02ClaudeBot (Anthropic training)
Disallowing ClaudeBot signals that this site's future content should be excluded from Anthropic's AI model training datasets. It has no effect on Claude-SearchBot or Claude-User — each is controlled independently.
Official source · checked 2026-09-02Claude-SearchBot (Anthropic search)
Disallowing Claude-SearchBot prevents this content from being indexed for Anthropic's search-quality features, which may reduce this site's visibility and accuracy in Claude's search results. It does not affect model training.
Official source · checked 2026-09-02Claude-User (Anthropic, user-triggered)
Claude-User only fetches a page because a person asked Claude a question that required visiting it. Disallowing it prevents that on-demand retrieval, which may reduce this site's visibility for user-directed queries — it is not the same as blocking automatic crawling.
Official source · checked 2026-09-02Google-Extended (Gemini/Vertex AI training & grounding)
Google-Extended controls whether content Google already crawls (via Googlebot) may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google's own documentation states it does not impact this site's inclusion in Google Search or search ranking — that remains controlled by Googlebot alone.
Official source · checked 2026-09-02Sitemap URLs
Optional absolute URLs to XML sitemap files.
Your data is processed locally in your browser whenever supported and is not uploaded to our servers.
How it works
Create a robots.txt file in three steps
01
Choose a starting point
Start with an allow-all, directory-blocking, WordPress, or other clearly explained preset.
02
Edit crawler rules
Add user-agent groups, Allow or Disallow paths, sitemap URLs, and optional verified AI crawler rules.
03
Review and download
Fix validation errors, read the important guidance, then copy or download robots.txt.
Clear crawler control
Standards-aware rules without false security promises.
The generator checks structure, path syntax, duplicate rules, sitemap URLs, and high-impact choices. It never claims robots.txt can secure private content or guarantee removal from search results.
Safe presets
Use practical starting points while keeping every path and user-agent editable.
Verified AI tokens
Optional OpenAI, Anthropic, and Google crawler entries include scope notes and official sources.
Live validation
Invalid files cannot be copied or downloaded until configuration errors are resolved.
Local generation
Rules are generated in your browser without visiting your site or sending input to a server.
FAQ
Common questions
Can robots.txt keep a page private?
No. Robots.txt is a voluntary instruction for cooperative crawlers, not access control. Use authentication or other server-side protection for private or sensitive content.
Does Disallow remove a URL from Google?
Not necessarily. A disallowed URL may still be discovered and indexed from links elsewhere. Use a noindex meta tag or X-Robots-Tag on a crawlable page when you need to prevent indexing.
Where should I upload the file?
Serve it at the top-level /robots.txt path of the exact protocol, host, and port it controls. Subdomains and different protocols need their own file.
Does Google support Crawl-delay?
No. Crawl-delay is not part of the core robots.txt standard and Google does not support it, although some other crawlers may. The tool always warns when you add it.
Can one rule block every kind of AI use?
No. Training crawlers, search crawlers, and user-triggered retrieval use different tokens and may follow different policies. Vendor behavior can change, so each included preset shows its precise scope and verification source.
Keep working
Related tools
XML Sitemap Validator
Check XML sitemap structure, URLs, dates, limits, duplicates, and host consistency locally.
Open tool →Meta Tag Generator & Preview
Create safe search, Open Graph, and X meta tags with honest live previews.
Open tool →QR Code Generator
Create private QR codes for text, URLs, Wi-Fi, email, phone, and SMS directly in your browser.
Open tool →