Does bot protection affect SEO and real visitors

Bot protection and SEO split on the first question: who is allowed onto the site at all. RFC 9309 describes robots.txt as a protocol by which the service owner asks automatic clients how to crawl URIs. A crawler in that document is an automatic client. One example is a search service following links for indexing. The rules of the protocol are not an access permission. A disallow in the file does not replace a decision to admit the request, and the absence of a disallow does not grant a right to enter.

What follows is how Google and Yandex describe their robots, which page-experience thresholds Google sets, and what happens to a visit if a Binetro check does not answer.

The file asks; the server decides access

robots.txt is a request to an automatic client. RFC does not promise what a particular robot will do if the file disallows a crawl. The behavior of Googlebot and of Yandex robots is described in their own documentation, and it does not collapse to a single Disallow line.

The HTTP User-Agent string can be spoofed. A header that matches a robot’s name does not prove that robot arrived. Google’s documentation separates the header text from the network the crawl comes from.

Google’s ordinary crawlers and the Google-Extended token

Google’s crawl-infrastructure documentation says that ordinary crawlers respect robots.txt on automatic crawls and, in general, come from the ranges in the common-crawlers.json object. The reverse hostname matches the pattern crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. The list of crawlers on that page is not exhaustive. The hostname patterns are a check template, not a list of addresses to block.

The Googlebot token in robots.txt covers Search, including Discover and all search features, plus Images, video mode, and News. Google-Extended has no separate HTTP User-Agent string: crawling is done by Google’s existing agents. The token in robots.txt decides whether content already fetched may be used to train future generations of Gemini models and for grounding. It does not affect inclusion in Search or a ranking signal.

Closing Google-Extended is not the same as closing the crawl for Search. The other way around, the Googlebot line is about search features, not about whether something already fetched will be used to train models.

Yandex robots and robots.txt

Yandex Webmaster’s help says Yandex robots use the autonomous networks AS13238, AS208722, and AS212066. Addresses change often, so they do not publish an IP list. The help prefers a reverse-DNS check to an address list. An address belonging to a Yandex ASN does not mean that address is a search robot.

Some robots fetch documents for reasons other than indexing and may ignore User-agent: *. YaDirectFetcher does not use the robots.txt file. Metrica robots are marked in the help table as not following the general robots.txt rules. The Metrica 4.0 variant fetches and caches CSS so it can reproduce the page in Webvisor. The YandexAdditionalBot and YandexAdditional records are honored in robots.txt in order to limit content shown in quick answers with YandexGPT and in Search answers with Alice. They do not make indexing requests.

One robots.txt file therefore does not describe “all Yandex robots.” Indexing, the Direct fetcher, Metrica robots, and the records for Alice answers read it differently. A User-agent: * disallow does not promise that YaDirectFetcher or a Metrica robot will stop.

Page-experience thresholds are not a measurement of the protection script

The Web Vitals documentation, updated on 31 October 2024, sets three stable page-experience metrics.

The threshold for most users is the 75th percentile of loads, separately for mobile and for desktop. A page passes if all three hold at that percentile. The metrics are meant to be field metrics. A lab does not replace them: the score is affected by the device, the network, other processes, and how a person touches the page. Lighthouse in the lab does not measure INP, because there is no user input. It looks at Total Blocking Time instead.

The thresholds are Google’s recommendation for page experience as of that page’s update. The document does not measure a protection script and does not tie these numbers to an SEO position. Product documentation does not set a time budget for the Binetro script. You cannot treat 2.5 seconds, 200 milliseconds, or 0.1 as “the protection limit”: those numbers are about page experience, not about how long a check takes.

What the check does with a robot and with a real visit when it fails

A known search robot can be let through. The decision on the visit does not wait for this check to finish. A failure of the check is not treated as proof that the robot is fake. How a robot is recognized is not covered here. What follows from the fact is different: a broken check does not become a verdict of “this is a fake robot,” and the decision on the visit does not sit in a queue until this check ends.

If the script got no answer — an error, a timeout, or a 5xx response — the visit is let through. The check itself has no such mode: the pass-through on failure is done by the script on the site. If the response could not be assembled, the visitor is let through as well. A real person is not left without the page because the check stayed silent. A client the check did not have time to review gets the same pass-through.

The SEO file and the visit check answer different questions. robots.txt asks an automatic client to crawl URIs in a certain way, and it is not access control. The visit check decides on an arrival, and when it fails it does not replace silence with a refusal or treat that failure as proof of a fake robot.