Blocking Google-Extended does not remove a site from Search

A Disallow for Google-Extended in robots.txt does not take a site out of Google Search and does not change a ranking signal. Google's crawling infrastructure docs give the token a different job: whether content already downloaded may be used to train future generations of Gemini models and for grounding. Google-Extended has no HTTP User-Agent line of its own. The crawl uses Google's existing agents.

What robots.txt can and cannot do is covered in robots.txt and bots. Search crawlers as traffic a site usually wants are in Good bots.

What the Googlebot token covers

The Googlebot token in robots.txt affects Search, including Discover and all search features, plus Images, the Video mode and News. That is a different token from Google-Extended.

Ordinary crawlers obey robots.txt on an automatic crawl and in general come from the ranges in the common-crawlers.json object. The reverse hostname matches the mask crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. The HTTP User-Agent string can be spoofed, so the docs check the hostname, not the agent string alone. The masks are a template for that check, not a list of addresses to block. The docs do not call the crawler list on that page exhaustive.

If the job is to keep a site out of Search, the token the docs tie to Search is Googlebot. Google-Extended does not do that job.

What Google-Extended is not

Google-Extended is not a separate robot with its own User-Agent line. The docs do not describe a disallow of the token as turning the site off in Search, and they do not describe it as a change to a ranking signal. Content that ordinary crawlers have already downloaded is what the token relates to training future Gemini models and to grounding. Search itself lives on the Googlebot token.

The mix-up usually starts from one line in the file: both tokens sit in robots.txt, and any Disallow looks like it "closes Google" in the same way. The docs split them by consequence. A blocked Google-Extended leaves the Search question on the Googlebot token. A blocked Googlebot affects Search, Discover, Images, Video and News.

How Yandex splits the index from Alice's answers

Yandex Webmaster help says Yandex robots use the autonomous systems AS13238, AS208722 and AS212066. Addresses change often, and they do not publish an IP list. The help prefers a reverse-DNS check over an address list. An address sitting in one of those systems is not, by that fact, a search robot.

Some robots download documents for something other than indexing and may not obey User-agent: *. YaDirectFetcher does not use robots.txt. Metrica robots are marked in the help's table as not following the general robots.txt rules. Metrica 4.0 fetches and caches CSS so it can reproduce a page in Webvisor.

The YandexAdditionalBot and YandexAdditional records are honored in robots.txt in order to limit content in quick answers with YandexGPT and in Search answers with Alice. They do not make indexing requests. Disallowing those records is about generated answers, not about whether a page enters the index. It is the neighboring fork to Google-Extended: a separate token limits use in a model's answers, and the indexing crawl lives apart from it.

How OpenAI splits training and search

OpenAI's docs set separate robots.txt tokens. GPTBot downloads content that may go into training generative models. A disallow means the content should not be used for that training. OAI-SearchBot is for ChatGPT search. A site closed to it is not shown in search answers, though a navigation link is possible. The tokens are independent: you can allow search and disallow training.

ChatGPT-User visits when a person asked a question or a GPT action fired. It is not an automatic crawl. Because a user initiated it, robots.txt rules may not apply, and it does not decide inclusion in search. JSON files with address lists are published for the automatic robots. Version numbers in the example agent strings can change, and those strings are a poor thing to copy into a filter as a permanent sample.

This is one operator's description. It does not measure traffic shares or a full list of other AI crawlers. For a decision about Google, Google's own section is the one that matters: Google-Extended does not touch Search. The parallel with OpenAI is only that a training token and a search token are set separately, and the consequence of blocking search is different at OpenAI: a blocked OAI-SearchBot removes the site from ChatGPT search answers.

What to do with the line in the file

1. Find the Google-Extended token and the Googlebot token in robots.txt. They are different groups.

2. If the line disallows only Google-Extended, Google's docs do not describe that as turning Search off.

3. If you need Search, Images, Video, News or Discover off, look at the Googlebot group, not Google-Extended.

4. At Yandex, the limit on answers with YandexGPT and Alice is the YandexAdditionalBot and YandexAdditional records. Those records do not make indexing requests.

5. Google's docs tie a robot check to the reverse hostname mask, because the User-Agent string can be spoofed. The mask is not a list of addresses to disallow.

A training disallow does not stand in for a crawl disallow. And a Googlebot crawl disallow is not the Google-Extended token. Keep the group whose consequence you want, and do not expect Google-Extended to drop a page out of Search.