Choose crawler permissions by documented purpose. Block a named training crawler while preserving the search access you actually want, and verify the public response plus edge rules before treating the change as effective.
You can object to your articles being collected for model training while still wanting people to discover those articles through search. The practical problem is that an AI service can use different crawlers for different jobs. A blanket block may remove the discovery route you intended to preserve.
Start with a written policy: which public content may be crawled for search, which collection you want to disallow, and which private content requires authentication regardless of crawler identity. Then translate that policy into the controls each operator actually documents. This guide explains a configuration and verification process; it does not promise search visibility or universal compliance from bots.
Identify the job before blocking the bot
OpenAI documents separate controls for GPTBot, used for potential model-training collection, and OAI-SearchBot, used for search. You can disallow the former and allow the latter. ChatGPT-User is different again: it can fetch a page in response to a person’s request, and robots.txt rules may not apply to that user-triggered action. Search eligibility should therefore be managed using the documented search crawler rather than inferred from a request carrying the ChatGPT-User name.
Do not extend that naming pattern to every company. Google-Extended is a robots.txt product token covering specified Gemini training and grounding uses. It has no separate HTTP user-agent string, and Google says it does not affect inclusion or ranking in Google Search. Blocking it is not a pure training-only switch: consider its grounding scope as part of your decision.
Create an inventory with one row per operator, the documented purpose, the exact robots token, the desired permission and the date checked. A label such as AI bot is too broad to be a reliable configuration specification. An unfamiliar user agent should remain unclassified until its operator and behavior are investigated.
Capture the existing public configuration first
Before editing anything, save the current robots.txt response from the canonical public hostname. Also record your CDN’s bot settings, relevant firewall rules and whether WordPress or an SEO plugin generates a virtual robots file. A file visible in a hosting panel may not be the one visitors receive.
Include www and non-www behavior in the check. Ideally one redirects consistently to your chosen hostname. If both serve content independently, examine both robots responses rather than assuming a setting on one protects the other. Keep a dated copy of the previous response and the setting you are about to change so rollback is concrete.
Useful baseline evidence includes the response status, final URL after redirects, response content type and the complete plain-text body. Record how a normal public article behaves too. If the site already returns a challenge or error, resolve that access issue separately instead of attributing it to your new policy.
Add a targeted rule without replacing unrelated rules
The following is a small illustrative group to merge into an existing policy, not a replacement robots.txt file:
User-agent: GPTBot
Disallow: /
If OAI-SearchBot is not blocked by your existing rules, a new explicit Allow group may be unnecessary. Keep the rest of the file intact, including restrictions on irrelevant crawl paths and any sitemap declaration. Where you do create a specific search-bot group, carry over the path restrictions that should apply to it. A more specific group may cause a crawler to use that group’s rules instead of the wildcard group; do not assume every existing wildcard restriction will automatically remain effective.
For example, decide explicitly whether public search-result URLs, filtered comparison URLs and generated parameter combinations should be crawled. That is a crawl-management question distinct from allowing the service to access your useful guides. Never put confidential directory names into robots.txt as a substitute for authentication: the file is publicly readable.
Google’s robots.txt guidance explains why a crawl restriction is not equivalent to securing a URL or reliably removing it from an index. Use actual access controls for private material. If removal from search is the goal, investigate the appropriate indexing controls instead of combining incompatible instructions and hoping the crawler guesses your intent.
Check the CDN response, not only the origin file
A CDN can add policy at the edge. Cloudflare’s current managed robots documentation describes Bot Preference Sync and separate Training, Search and Agent policies. Managed directives can be prepended to an existing origin file. Consequently, a correct origin edit can coexist with an unexpected public rule above it.
After any change, retrieve the final public response again and compare it with your baseline. Look for duplicate groups, a new wildcard block, a training token that also covers another use, and an unintended change to the sitemap line. Investigate the resulting policy rather than deleting unfamiliar lines from a managed service without understanding how they were generated.
The same separation applies to enforcement. robots.txt expresses preferences to cooperating crawlers; a firewall can block the request itself. An Allow rule cannot override an edge block, rate limit or JavaScript challenge. Conversely, a 200 response does not mean a crawler is permitted under your policy. Check both layers.
Use a small acceptance matrix
Choose a public guide, an image it needs, the homepage and a genuinely private URL. For each intended crawler class, write down the expected result before testing. A practical acceptance matrix is:
- The canonical robots.txt returns a readable policy without a login form or challenge.
- A public guide remains accessible to ordinary visitors.
- The search crawler’s documented token is not accidentally disallowed from the guide.
- The training crawler’s selected rule expresses the restriction you intended.
- Required public images and rendering resources remain available where needed.
- A private URL remains protected by authentication, regardless of user-agent text.
A command-line request with a copied user-agent string can help reveal a simple rule or challenge, but it does not impersonate a verified crawler. Real bot validation may also involve source IP or other documented verification. Do not report a simulated request as proof that the genuine service fetched or indexed the page.
Read logs as evidence of requests, not understanding
Review the available origin and edge logs after a reasonable observation period. Record the path, time, status and validated crawler classification where available. If the CDN answers a request without contacting the origin, the origin log alone will not show the complete journey.
When search access unexpectedly fails, distinguish three cases. A robots denial needs a policy correction. A forbidden or challenged HTTP response needs an edge or origin investigation. A successful fetch followed by no visible search result may be a discovery or selection issue rather than a block. These cases have different fixes; repeatedly toggling the training switch does not diagnose them.
A request also cannot prove that content was trained on, cited, or omitted from a future answer. Keep your evidence statement narrow: the policy served on this date, the access observed, and the remaining unknowns.
Roll back narrowly if discovery breaks
If the change accidentally blocks a wanted search crawler, restore the specific previous group or edge rule. Avoid disabling every security feature to recover one bot. Recheck public access, private access and the final robots response after rollback, and retain the failed configuration with an explanation for the next person maintaining the site.
A useful operational habit is to change one layer at a time. First adjust the documented crawler preference, verify its response, and then consider enforcement if required. Changing the CDN policy, WordPress plugin, redirects and firewall together makes a failure much harder to isolate.
Write a change record that someone else can use
A useful maintenance record states the wanted result, the exact group changed, the previous setting and the acceptance checks. For example: public guides remain available to the documented search crawler; collection by the named training crawler is disallowed; private pages still require login. Attach the before and after robots responses and identify any edge policy involved. This makes it possible to revisit the decision when a new crawler appears without reconstructing the whole configuration from memory. Keep the record with your website operations notes rather than only in a chat conversation.
What this approach can and cannot achieve
This process gives you a maintainable expression of your preferences and a way to catch accidental search blocks. It cannot make every crawler obey, undo past collection, or guarantee a place in an AI answer. Some operators combine purposes, some user-triggered fetches behave differently, and documentation evolves.
Revisit the inventory when a service changes its crawler names or policy scope. For the separate question of whether to publish an AI-oriented documentation index, see whether your site needs llms.txt. Keeping discovery, documentation and access control as separate decisions makes each one easier to verify.
Sources and verification
Editorial review 2026-10-14. Source check dates are recorded separately. Notes describe what each source supports and its limits.
View sources
- OpenAI crawler overviewChecked GPTBot and OAI-SearchBot have independent controls; user-triggered ChatGPT-User access differs from automatic crawling.
Limits: OpenAI's documented behavior; not a universal rule for other crawler operators. - Google common crawlers: Google-ExtendedChecked Google-Extended controls specified Gemini training and grounding uses without affecting Google Search inclusion or ranking.
Limits: The product token covers more than training alone and is not a separate HTTP user-agent string. - Cloudflare robots.txt settingChecked Managed directives can be prepended to the origin response; current policy separates Training, Search and Agent settings.
Limits: Dashboard behavior and availability can change. Read the response actually served on the site's hostname. - Google robots.txt introductionChecked robots.txt is a crawling control rather than a mechanism for securing private content or reliably removing indexed URLs.
Limits: This guide is about crawler configuration, not copyright enforcement or legal rights.