AI crawler permissions: search visibility vs model training
A practical AEO audit for separating search, model training and user-requested access, with OpenAI and Google primary documentation.
An AI crawler policy should start with the use you want to permit: discovery in search, model training, or a visit requested by a person. Treating these as one permission makes an AEO audit harder to interpret. A block intended for training can affect search when it targets the wrong agent.
Search and training have different controls
OpenAI documents independent controls: OAI-SearchBot supports ChatGPT search, while GPTBot collects material that may be used for model training. Its documentation also separates ChatGPT-User, which handles certain user-initiated visits rather than automatic crawling. Robots rules may not apply to those visits. Allowing access does not guarantee a citation.
Google makes a different distinction. Googlebot controls crawling for Search, including its AI features. Google-Extended manages specified training and grounding uses outside Search; Google says it does not affect inclusion or ranking in Search. These controls are not interchangeable across providers.
Write a permission matrix before editing robots.txt
Use one row per provider and use case. Record the desired decision, the official control, the affected URL paths, the owner and the date checked. A useful matrix distinguishes permission from observation: an allowed path, a successful fetch, an indexed page and a cited answer are separate facts.
- Search: identify the crawler or product control documented for the search experience you care about.
- Training: record the separate preference where the provider offers one.
- User-requested access: review its documented behavior separately.
- Restricted material: protect access with authentication where needed; do not use robots.txt as an access-control system.
Verify the served policy as well as the repository file
For example, an owner may allow search discovery of public guides while declining a provider’s training use. Record that as two explicit decisions. If a licensed manual must remain private, handle its access separately rather than assuming that a training preference protects the file.
Our recommended audit starts at the public origin. Read the robots.txt actually returned by the server, then compare it with the intended policy. Check whether hosting, a CDN or a bot-management rule modifies the response. A correct file in Git does not prove that a crawler receives that version.
- Inspect the response status and content of the public robots.txt.
- Check a representative article, guide and service page for access and indexability.
- Review CDN and firewall rules for challenges or blocks on permitted requests.
- Use the provider’s published verification guidance when attributing requests; a user-agent name alone is weak evidence.
- Save the observed result and the change time so a later audit can compare like with like.
Measure visibility after the technical check
For an AEO review, keep a stable set of questions and record the engine, date, answer and cited URLs. Compare repeated observations rather than interpreting one missing citation as proof of a crawler block. Content relevance, retrieval and answer construction remain separate from whether the server allowed a request.
A practical decision rule is to fix an evidenced access problem first, then evaluate citations separately. If the public response is healthy but a page is absent from an answer, investigate the question, the source page and competing evidence before changing crawler permissions again.
What to document at handover
Leave the permission matrix, official references, the public response checked, representative URLs, the responsible person and a future review date. Do not claim that a robots change caused a visibility gain without a comparison that supports it. This article is a documentary guide. It does not test every provider or recommend the same policy for every site.
Documentation reviewed on 11 October 2026. Provider behavior can change; check the linked primary sources before applying a configuration.