Good bots: crawlers, monitoring, and link previews
Contents
Share
A good bot is a program with a clear purpose: index a page, check that the site answers, fetch a preview, follow links for search. Yandex Metrica defines a robot the same way: a program that has a purpose, with indexing and an availability check as the examples. Some identify themselves, others disguise themselves. Metrica detects them by behavior and technical signals. This is not click fraud. An ad click and a crawler walking pages are decided in different places.
You need to tell a robot that named itself from a client that only borrowed the name. Below: what to look at in the report, how to verify Google and Yandex, and who cannot be closed with one line in robots.txt.
Who is already outside ordinary statistics
Robots that identified themselves are excluded from Metrica’s ordinary statistics at once. You look at them in the Robots report, including to see indexing. If a visit disappeared from the main reports and is visible only there, that is a declared name, not a verdict of “attacker.”
A separate rule in the report is “Filtered for missing JS and Referer.” The counter code is JavaScript plus an invisible image in noscript. If JavaScript is off or unsupported, view data leaves in the Referer header when that image loads. If the header did not arrive, Metrica treats the view as robotic. That is one named rule. The report still does not spell out the behavioral rules, so “robot” in the report is not an instruction about which signal fired.
A User-Agent string proves nothing
RFC 9110 (June 2022) describes User-Agent as information about the program that sent the request. Servers use it to narrow incompatibility reports, work around a specific program, and look at browser statistics. A client should send the header unless it was deliberately disabled. The value is product identifiers with an optional version. Unnecessary fine detail and other parties’ subproducts do not belong there: a very detailed string adds latency and the risk of recognizing a user against their will. In theory the string makes it easier to aim at a known hole; in practice hole-scanning proceeds regardless of the claimed version. Example strings in the RFC illustrate the grammar. They are not a sign of a bot.
Google’s documentation says other crawlers often put on the Google crawler’s User-Agent. Verify the request with reverse DNS on the IP, or by matching the IP to published ranges, not by the header text. Google’s ordinary crawlers honor robots.txt on automatic fetches. Special-case crawlers may not. User-triggered fetchers ignore robots.txt because a person started the request.
Yandex Webmaster’s help: someone else’s robot can call itself a Yandex robot. The check is the IP from the log, reverse DNS, a name ending in yandex.ru, yandex.net, or yandex.com, then forward DNS. If the forward address does not match the original, the name is fake. The help prefers this to an IP list because the internal networks change.
robots.txt asks, it does not lock
RFC 9309 describes robots.txt as a request to automated clients about how to fetch URIs. A crawler in the document is an automated client; the example is a search engine following links. The rules are not an access-control grant. The RFC does not promise what a particular robot will do if the file disallows a fetch. Google’s and Yandex’s behavior are looked at separately.
Google’s ordinary crawlers honor robots.txt on automatic fetches and, in general, come from the ranges in common-crawlers.json. The reverse hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. The Googlebot token in robots.txt covers Search, including Discover and search features, plus Images, video mode, and News. Google-Extended has no separate HTTP User-Agent: fetching uses existing Google agents, and the token decides whether content already fetched may be used to train future Gemini models and for grounding. It does not affect inclusion in Search or a ranking signal. The crawler list is not exhaustive. The hostname patterns are a verification template, not a list of addresses to block.
Yandex robots use the autonomous systems AS13238, AS208722, and AS212066. Addresses change often, and no IP list is published. Some robots fetch documents for reasons other than indexing and may not obey User-agent: *. YaDirectFetcher does not use robots.txt. Metrica’s robots are marked as not honoring the general robots.txt rules; the 4.0 variant fetches and caches CSS so a page can be replayed in Webvisor. YandexAdditionalBot and YandexAdditional are honored in robots.txt so you can limit content in quick answers with YandexGPT and in Search answers with Alice; they do not make indexing requests. An address in a Yandex ASN is not automatically a search robot.
OpenAI sets separate tokens. GPTBot fetches content that may be used to train generative models; a disallow means the content should not be used for that training. OAI-SearchBot is for ChatGPT search; a site closed to it is not shown in search answers, though a navigation link is still possible. The tokens are independent: you can allow search and disallow training. ChatGPT-User fetches when a person asked a question or a GPT action fired. It is not an automatic crawl, robots.txt may not apply because a user initiated it, and it does not decide inclusion in search. JSON address lists are published for the automatic robots. Version numbers in sample agent strings can change. This documentation does not publish traffic shares or a full list of other companies’ AI crawlers, so one OpenAI token cannot stand in for “all bots.”
If the robot check fails
A known search robot may be allowed through. The decision on the visit does not wait for that check to finish. A failed check is not treated as proof of a spoof. How the robot is recognized is not part of this rule. The practical reading: a DNS timeout is not a reason to record the visit as a fake Googlebot.
Order:
Look at robots that identified themselves in the Robots report, not in the main statistics.
Verify a User-Agent name with reverse and forward DNS, or with published ranges.
In robots.txt, separate a search token from a training token. The file does not replace access control.
Do not read a user-triggered fetcher, or Metrica’s robot for Webvisor, as a violator of
User-agent: *.Do not treat a failed check as a spoof.
Who clicks ads, click fraud, invalid clicks, and a refund are not decided by a robot’s name string.