How modern bots work: from scripts to AI agents

A modern bot is not one mechanism. Click fraud belongs to ad measurement. Next to it sit a program with a purpose in Yandex Metrica's definition, a browser with no window, a search crawler, and a fetch that a person started by asking a chat a question. Below is how those descriptions are built, and where one does not prove another.

Mixing them is expensive. Blocking a search crawler because of a client string cuts indexing, while letting a windowless session through leaves a click the ad account may count as ordinary. Click fraud and invalid clicks are not settled by this walkthrough. The point is to read the public definitions before you decide what to do with a visit.

A program with a purpose, and traffic that is hard to see

Yandex Metrica calls a robot a program that has a purpose. The definition names indexing pages and checking that a site answers. Some robots identify themselves, others disguise themselves. Metrica detects them from behavior and technical signals. That is not an automatic label of "a click spent to drain a budget."

Who clicks ads, and why is described separately in the MRC standard for the harder cases. SIVT covers situations that are harder to notice and that need advanced analysis, a comparison of several signals, or a clear human role. The examples include automation from a dedicated device and from an infected or captured device, incentivized human activity, forced clicks and accidental clicks caused by a spoofed interface, fabricated measurement events, bots that disguise themselves, and invalid proxy traffic. A footnote gives a click farm as a case where non-robot activity is organized and directed through many users or machines.

The word IVT covers both the class that can later be recognized from standard lists (GIVT) and SIVT. Some SIVT can move into GIVT if those lists appear. This is a classification for ad measurement, not a blueprint of a farm and not a how-to. The standard asks that disclosed reward programs without an intent to substitute measurement stay outside this category. Invalid clicks in this frame include both disguised bots and living but organized activity. Whether Yandex refunds click fraud is not something the MRC standard decides.

The client string and the fingerprint

The HTTP User-Agent header is the string a server uses to learn the application, operating system, vendor, and client version. MDN writes that the string historically identified a specific browser too well, so browsers now send a reduced set: the platform version and device model are frozen, and the minor Chrome version numbers are shown as zeros. The server can ask for the rest separately, through client hints. The concrete example strings on the MDN page illustrate the format. They are not a sign of a bot, so a filter built on "looks like Chrome" does not follow from that string.

A W3C Privacy Working Group note of 25 September 2025 defines browser fingerprinting as a site's ability to recognize or re-recognize a user, a program, or a device from settings and other observable characteristics. A fingerprint can tie visits together inside a session and across sessions, including across sites, when another origin's cookie policy does not hand the identifier over. Unlike a cookie, collection often happens without an explicit signal, and a typical fingerprint cannot be cleared or reset. Clearing cookies and using a VPN does not stop later linking by fingerprint.

The note treats full removal of fingerprinting against a determined party, by widely deployed technical measures alone, as implausible: mitigations are not a solution. The surface runs across every capability of the program and into other layers, for example differences in TCP connections. This is guidance for specification authors, not a detector and not a definition of an antidetect browser. The note does not give figures for how unique a browser is, and there is nothing to paste into a visit decision.

A browser with no window

Chrome's documentation describes Headless as running the browser unattended and without a visible interface. From Chrome 112 the mode works like this: platform windows are created but not shown, and the other features, current and future, are available. Earlier, Headless was a separate browser implementation inside the same binary and did not share ordinary Chrome's code. From Chrome 132.0.6793.0 the old mode remains only as a separate chrome-headless-shell binary. In the Puppeteer example on that page, headless: true turns Headless on by default, and headless: false opens an ordinary window.

The mode is described for automation and tests. The documentation does not call a windowless launch click fraud by itself. Masking flags are not part of the public description of the mode, and a visit is not filed as SIVT from "no window" alone.

Search crawlers and a fetch from chat

Google's crawler infrastructure documentation says that ordinary crawlers, on an automatic crawl, respect robots.txt and in general come from ranges in the common-crawlers.json object. The reverse hostname matches the pattern crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. The HTTP User-Agent string can be spoofed, so a token in the string without a hostname check does not prove a crawler.

The Googlebot token in robots.txt covers Search, including Discover and all search features, plus Images, video mode, and News. Google-Extended has no separate HTTP User-Agent string: crawling uses Google's existing agents, and the robots.txt token decides whether content already fetched may be used to train future generations of Gemini models and for grounding. It does not affect inclusion in Search or a ranking signal. The crawler list in that documentation is not exhaustive. The hostname patterns are a check template, not a list of addresses to block.

OpenAI's documentation sets separate robots.txt tokens. GPTBot fetches content that may go into training generative models; a disallow means the content should not be used for that training. OAI-SearchBot is for ChatGPT search; a site closed to it is not shown in search answers, though a navigation link remains possible. The tokens are independent: you can allow search and disallow training. ChatGPT-User fetches when a user asked a question or a GPT action ran; it is not an automatic crawl, robots.txt rules may not apply because a user initiated it, and it does not decide inclusion in search. JSON address lists are published for the automatic robots. Version numbers in the sample agent strings can change, and they should not be copied into a filter as a permanent sample. This is one operator's description. The documentation does not give shares of other vendors' AI traffic.

What a TLS string does not prove

The FoxIO JA4 specification builds a client fingerprint from the TLS ClientHello packet. The fingerprint includes the transport (TLS over TCP, QUIC, or DTLS), the TLS version, whether SNI is present, the cipher count, the extension count, the ALPN marker, and hashes of the cipher and extension lists. The specification says to ignore GREASE values. Neighboring methods in the same suite are counted separately: JA4S is the server response, JA4H is the HTTP client. The specification describes a way to fold the handshake into a string. It does not claim that a particular string proves an automatic visit. Byte order, hash lengths, and sample strings are not copied into a decision. Binetro's current check does not use this signal. That is a product boundary, not a property of JA4.

How to keep the cases apart

Walk a visit in this order. Each step answers one question and does not replace the next.

1. Does the program have a stated purpose such as indexing or checking that the site answers, and does it identify itself. That is how Metrica separates a robot from disguised traffic. It does not diagnose an ad click.

2. Does the case look like SIVT: automation from a device, a captured device, incentivized people, a forced or accidental click, a fabricated event, disguise, an invalid proxy. That is a reason to compare several signals, not to apply one rule.

3. What is in User-Agent. The string names the client, and it is both reduced and spoofable. A match with a browser name does not prove a bot.

4. Is there a window. Headless since Chrome 112 is the same browser with the window not shown. For tests that is a normal mode, not a verdict on the click.

5. Whose crawl is it. Check Googlebot by reverse hostname and the published ranges. Google-Extended neither includes nor excludes Search. GPTBot and OAI-SearchBot are independent, and ChatGPT-User is a fetch a person asked for.

6. Do not take a JA4 fingerprint as proof of automation. A folded ClientHello does not prove an automatic visit, and Binetro does not use it today.

One visit, six questions

Take one suspicious click from the account and walk it through the same six questions, without mixing the answers.

Suppose the report shows a click from an ordinary browser, with no goal action, from an address you have not seen before. The first question is whether a program identified itself and had a purpose such as indexing. If the client string is a reduced Chrome, that is not yet a Metrica robot identifying itself: the minor version numbers are zeros there anyway, and the string can be spoofed. The second question is whether several SIVT signals are present at once, or only "there was no lead." The absence of a lead is not enough: the standard asks for a comparison, and living organized activity, including the click farm in the footnote, sits in the same class apart from a program.

The third question is the window. Headless since Chrome 112 creates windows and does not show them; from version 132.0.6793.0 the old mode lives in a separate binary. For tests that is normal. The fourth is whose crawl it is, if the string looks like a search robot. Google is checked by reverse name and ranges, not by a token in the string. Google-Extended neither includes nor excludes Search. ChatGPT-User fetches because a person asked, and it is not an automatic crawl. The fifth is not to replace all of this with a JA4 string. It folds the ClientHello and does not prove an automatic visit. Binetro does not use it today.

The sixth question is about money, and it is short. Even if the visit looks like SIVT, the MRC standard does not decide whether the platform returns the money. The classification measures traffic. The charge stays with the ad system.

The names do not stack into one ladder of "script, then bot, then AI." A program with a purpose in Metrica may be indexing honestly. Headless may be running a test. A Google crawler and the OpenAI tokens fetch content, and ChatGPT-User fetches because a person asked. SIVT is for the case where the browse already looks ordinary and one name in the string is not enough. JA4 does not attach to that ladder: a handshake string does not prove an automatic visit.