How Google and Yandex slow their own crawl

Google crawls less when a site slows down, returns 5xx, or signals a rate limit, including HTTP 429. Yandex reduces how often its indexing robot requests pages when it sees a 429. Google does not support a Crawl-delay field in robots.txt, and HTTP 401 and 403 do not lower Google's crawl rate.

What robots.txt can do against bots in general is covered in robots.txt and bots. How bots touch behavioral factors is in Bots and SEO. Here is only how the two search engines turn their own crawl down.

What Google's crawl budget is made of

Google's documentation calls crawl budget the set of URLs Google can and wants to crawl. It is made of a capacity limit and demand. The capacity limit caps the total time the server keeps connections open for Google.

Every site starts equally cautiously. If responses are stable or faster, the limit rises. If the site slows down, returns 5xx, or signals a rate limit, including HTTP 429, the limit falls and Google crawls less. Duplicates and extra URLs waste the crawl. One crawler with high demand shares the overall capacity limit with the others.

If the crawl is stuck on server power, the document names adding server resources among the ways to raise the budget, when that is justified for the business. The markers "1 million pages" and "10,000 pages" are what the document itself calls a rough guess, not a threshold where the crawl switches on or off. This is Google's crawl, not a bill from someone else's parser, and not a promise that extra bots will raise a hosting tariff by a named amount.

What in robots.txt will not slow Google down

In its robots.txt reading, Google supports user-agent, allow, disallow and sitemap. crawl-delay is not in the supported list. A line Crawl-delay: 10 does not set Google's crawl rate.

A file over 500 KiB: everything past the limit is ignored. A 4xx response, other than 429, is read as if there were no usable robots.txt, so there are no crawl restrictions. The help says not to use 401 and 403 to lower the rate: they do not affect it. The code this budget document ties directly to a lower limit, next to slowdown and 5xx, is 429.

A crawl disallow does not get the page content indexed, but the URL can remain in results without a snippet. Disallow cuts requests to the blocked URLs. It does not replace a 429 when the server is struggling under the crawl that is still allowed.

How Yandex turns its robot down

The help page "How to reduce the load" says the indexing robot plans its own visits and regulates the load by how many new and already known pages it has to crawl. A spike happens when the robot learns about a new section or URLs change. If loaded pages greatly outnumber pages in search, the help treats near-duplicate and duplicate pages as likely.

A Disallow in robots.txt stops requests to the named pages within a day after the file is edited. The help calls HTTP 429 a sign that the server is having a hard time: the robot will reduce the number of requests. Yandex treats subdomains as separate sites, so a crawl of each one can multiply the load on the server. This is about Yandex's robot, not an arbitrary parser, and not a hosting bill in rubles.

What to do when the crawl gets in the server's way

1. Check whether you are answering Google with 5xx, or whether response time is rising. The budget document lowers the limit on that.

2. Send 429 if the server is struggling. Both Google and the Yandex help tie that code to fewer requests. Do not use 401 and 403 to change Google's rate.

3. Do not add Crawl-delay expecting Google to honor it: it is not in the list of directives Google supports.

4. Disallow what the robot does not need. At Yandex, requests to blocked pages drop within a day. At Google, a crawl disallow does not index the content, but the URL can remain in results without a snippet.

5. Remove duplicates and extra URLs: they waste Google's crawl budget. At Yandex, a strong skew of "loaded far more than is in search" is read as near-duplicates and duplicates.

6. Yandex crawls subdomains as separate sites. The load on one server adds up.

7. Google's docs name extra server capacity as a way to raise the budget, when the crawl is stuck on power and when that is justified for the business. It is not the first step, and it is not a fee "for bots".

A teaching example, not a measurement of your server. The site answers Google more slowly than usual and closes part of the requests with 503. The capacity limit falls, on the document's terms, and the crawl becomes less frequent. A neighboring 403 for the same volume of requests does not, on the robots.txt reading, lower the rate. The difference is the status code, not the length of the Disallow list.

What these codes do not do

A 429 does not tell a search robot from someone else's price parser: the code is one code, and the docs are talking about their own crawler. Disallow does not scrub a URL out of results entirely. A higher budget after fast responses means more of Google's crawl, not less. If the goal was "let them visit less often", a steadily fast server moves the limit the other way.

Neither Google nor the Yandex help names a hosting bill in these texts. Watch status codes and duplicates. Money for the robot's traffic does not follow from this help.