robots.txt: what it can and cannot do against bots

robots.txt can ask an automatic client to stay off some URLs and, for clients that read the file, reduce crawling. Against a bot that does not follow the protocol it can do nothing: the rules are not an access authorization, and a disallow line does not make the page unreachable.

A forged robot name and click fraud are different jobs. Click fraud as a term lives in ad spend. A file at the root does not protect a site from clicks. Who presses the ads is covered in who clicks ads. Invalid clicks and a refund for click fraud are counted in the ad account, not in a Disallow line.

A request, not a lock

RFC 9309 defines the Robots Exclusion Protocol: the service owner asks automatic clients to follow access rules for URIs. Section 1 states outright that these rules are not a form of access authorization. A group with no rules at the end of the file implicitly allows everything. If a robot token has neither its own group nor a * group, the rules do not apply. The URI of /robots.txt itself is implicitly allowed. The RFC describes clients that follow the protocol. It does not promise that an arbitrary program will read it.

Google’s documentation separates crawling and indexing. Disallowing Googlebot in robots.txt does not stop a URL from appearing in search results. To keep a URL out of the index you need noindex. To stop both a robot and a person from opening the page you need a different measure, such as a password. Blocking Googlebot affects Search, Discover, Images, Video, and News.

At Yandex a disallow does not pull a page out of search either. Webmaster help says the file limits crawling and can reduce load. Pages restricted in the file can still take part in search. To remove them you need noindex in the HTML or in an HTTP header, and those pages should not be closed in robots.txt: otherwise the robot will not see the instruction. If the file fails the requirements (no more than 500 KB, the name robots.txt at the root, a 200 response or a redirect to another robots.txt that returns 200), the site is treated as open for indexing. Cyrillic in the file and in the server’s HTTP headers is forbidden: the domain is written in Punycode, the path in the site’s encoding. Robots of other systems may read the directives differently.

What Google supports, and what it does not

Google supports user-agent, allow, disallow, and sitemap. crawl-delay is not on the list. A file larger than 500 KiB: everything past the limit is ignored. A 4xx response, other than 429, is read as if there were no usable robots.txt, so there are no crawl restrictions. The help says not to use 401 and 403 to slow crawling: they do not change the rate. A crawl ban stops the content from being indexed, but the URL can remain in results without a snippet. Those are Googlebot rules, not a third-party parser’s.

Since 22 February 2018 Yandex has not honored Crawl-delay either. The rate is set in Webmaster on the “Crawl rate” page, separately for the main domain and each subdomain. The default is “Trust Yandex.” A value set by hand is a recommendation to the robot: the actual rate can differ.

A name in the header is not yet the robot

Google’s documentation says other crawlers often put Google’s robot User-Agent on their requests. A request has to be checked by reverse DNS on the IP, or by matching the IP to published ranges, not by the header text. Google’s ordinary crawlers follow robots.txt on an automatic crawl. Special-case crawlers may not. Fetchers a user starts themselves ignore robots.txt, because a person initiated the request.

A stranger can also name itself Yandex’s robot. Webmaster’s check: the IP from the log, reverse DNS, the name must end in yandex.ru, yandex.net, or yandex.com, then a forward DNS lookup. If the forward address does not match the original, the name is fake. The help prefers this method to an IP list, because Yandex’s internal networks change. Some Yandex robots fetch documents for reasons other than indexing and may not obey User-agent: *. The robot may bypass particular robots.txt restrictions if Yandex has an agreement with the site owner.

A closed path as a sign, not as a lock

In the Honeypots section the OWASP cheat sheet names a technique: a bait path is disallowed in robots.txt, and a request to that path is treated as a sign of a client that does not follow the directive. Clients that read the protocol are the ones the page expects not to fetch the closed path.

The rules still do not lock the page. A closed path does not by itself remove access: it can be opened. A request to the bait is a sign that the client does not follow the file, not proof that the catalog is already protected. The page gives no false-positive rate. How to hide the path, and how to get around it, is not carried here.