What User-Agent is and why you cannot trust it

User-Agent is an HTTP header, a string by which the server learns the application, the operating system, the vendor, and the client version. It cannot be trusted as an identity document: the client composes the string, it can be trimmed and spoofed, and another crawler can insert a robot’s text. Click fraud is not decided from this header alone.

RFC 9110, from June 2022, describes User-Agent as information about the program that sent the request. Servers use it to narrow incompatibility reports, work around the limits of a particular program, and look at browser and system statistics. A client should send the header on every request unless it was specifically turned off. The value is one or more product identifiers with an optional version. A client should not put fine detail and other products’ sub-products into the header without a need: an overly detailed string grows latency and the risk that the user is recognized against their will. In the security section, the RFC says that in theory the string makes it easier to aim at a known hole, while in practice holes are tried regardless of the declared version. Example strings in the RFC illustrate the grammar. They are not a sign of a bot.

The string is trimmed and substituted

MDN writes that historically the string identified a specific browser too well, so browsers send a reduced set: the platform version and the device model are frozen, and the minor Chrome version numbers are shown as zeros. The server can ask for detail separately, through client hints. Example strings of particular browsers on the MDN page do not become a sign of a bot: they illustrate the format.

Google’s documentation says other crawlers often insert the User-Agent string of a Google robot. A request has to be checked by reverse DNS on the IP, or by matching the IP to published ranges, not by the header text. Google’s ordinary crawlers respect robots.txt on an automatic crawl. Special-case crawlers may not. Fetchers that the user starts ignore robots.txt, because a person initiated the request. A text match with the name Googlebot therefore does not prove that Googlebot arrived.

In Metrica’s “Robots” report the path is “Reports → Monitoring → Robots.” A behavioral-rule hit is shown as a separate row without a decoding of the rule. The other rows are labeled with a name from User-agent. A label in the report is a name from the header, not the reverse-DNS check Google asks for its robots.

When the string is part of routine recognition

The MRC addendum to invalid-traffic detection standards, final version of June 2020, splits IVT into two classes. GIVT is traffic recognized routinely: by lists or by standard parameter checks. The examples include known data-center traffic that consistently produces invalid activity but not the route artifacts of live users; bots, spiders, and other crawlers; checks on transaction parameters; a non-browser User-Agent; a prefetch after which a live person did not open the ad; known-bad placement sizes of 0×0 and 1×1; and sessions that, by the client’s data, cannot render an image, including a headless browser.

The example list is for an MRC accreditation audit. It should not be mixed with the TAG taxonomy. Direct’s GIVT share is not set by this list. A non-browser User-Agent in the examples is one routine sign of the class, not the only ground for cutting a click and not an instruction for the account.

A W3C note splits collection into passive and active. Passive collection reads what is already on the request, with no code on the client: a cookie, the set of HTTP headers, the address, and other network data. User-Agent is given as an example of a passive channel: it usually names the browser, the engine, the version, and the operating system. Active collection runs JavaScript or other code on the client. The note says CSS techniques are already catching up with JavaScript. Examples of active collection include window size, enumerating fonts and connected devices, a performance measurement, sensors, and rendered patterns. Active collection is in principle visible on the client. Traits more often leave for the server in a batch, and the server combines them out of the page’s sight. The list illustrates classes. It is not the list of fields a visit decision is made on.

A visit check builds a decision from several independent observations: behavior on the page, the browser environment, the network and address, request frequency, and the parameters of the ad click. One observation does not know the rules of the others. The field set, the scores, and the rules of individual checks are not published. User-Agent can sit among the request data as a passive channel, but that one observation does not know the rules of the others and does not issue the decision by itself.

An invalid click is therefore not the same as “an odd User-Agent string.” The header string does not settle whether Yandex returns the money. Who clicks without a purchase is in the article on who clicks ads. The difference in terms is in the article on click fraud and sklikivanie.

The header cannot be trusted as the program’s name: the client writes it, browsers trim it, and other crawlers often insert the Googlebot text. A Google robot is checked by IP and reverse DNS. A visit decision is built from several observations, and one of them does not replace the rest.