The robots.txt file remains the reference tool for telling crawlers which parts of a site they can explore. As AI-powered search tools and agents now visit sites alongside traditional crawlers, understanding the rules each follows is essential to avoid accidentally excluding your content.
Placed at the root of a site, this text file contains several directives. User-agent specifies the crawler concerned, Disallow requests not to crawl certain paths, Allow authorizes crawling of certain URLs even if their parent folder is blocked, and Sitemap indicates the location of the XML sitemap. The operation of these directives is described in the RFC 9309. The file does not support full regular expressions and only has two wildcards: the asterisk (*), which matches zero or more characters, and the dollar sign ($), which marks the end of a URL. Paths are case-sensitive, while directive names and crawler names are not.
Which rule takes precedence in case of conflict
When a crawler has its own specific rules, it follows those rather than the rules for User-agent: *. For Google, two principles apply within the relevant group. The most specific rule, the one that matches more characters of the URL, is applied first. Thus, an allowance for a subfolder, such as /downloads/free/, takes precedence over a disallow for /downloads/. If specificity is equal, the least restrictive rule applies.
Blocking pages that do not need to appear in search results, such as a privacy policy, helps reduce unnecessary crawling and directs the crawl budget towards useful pages. Google specifies, however, that this does not automatically transfer this budget to other pages, unless the site has already reached its crawl capacity limit. Online stores are particularly affected: filters with infinite combinations and action URLs, such as add-to-cart or add-to-wishlist, can generate a large number of pages with no inherent value. Gary Illyes of Google has repeatedly warned that these URLs can be crawled indefinitely.
The file does not, however, guarantee the removal of a page from the index. Google can index a blocked page from links pointing to it. To remove it, the source recommends the noindex tag, provided the page remains accessible to crawling so that Google can read this instruction.
AI Crawlers: Training, Search, and On-Demand Visits
The essential distinction concerns the use made of the content. A crawler may be used for training a language model, another for search or live information retrieval, and a third for a visit requested by a user. At OpenAI, OAI-SearchBot powers ChatGPT’s search responses, ChatGPT-User handles user-requested visits, and GPTBot is used for training. It is therefore possible to block GPTBot while allowing the other two.
According to a BuzzStream study, reported by Search Engine Journal in January 2026, 79% of US and UK news sites examined disallow at least one training crawler, and 71% disallow at least one search or live retrieval crawler. The guide highlights that these retrieval crawlers help AI tools find and cite pages, so blocking them is a separate decision from refusing training.
These rules have limits. A crawler that ignores robots.txt can still collect content, and blocking training does not force AI tools to cite the site. According to OpenAI, the file’s rules may not apply to ChatGPT-User, and Perplexity indicates that Perplexity-User generally ignores them. Anthropic states that its crawlers, including Claude-User, respect robots.txt. To protect private information, authentication remains necessary.
Google and Apple also offer separate tokens. Google-Extended covers training for Gemini and some answer grounding uses, without blocking Googlebot or inclusion in classic search. Applebot-Extended controls training for foundation models without blocking Applebot’s search crawling. These two tokens are not separate crawlers: they do not, by themselves, reduce the number of crawl requests.
