How bots affect SEO and on-site behavior metrics
Contents
Share
Bots affect SEO where they crawl and index pages, spend crawl budget on duplicates, or publish someone else’s content on their own site. Behavior figures in analytics do not automatically become a ranking factor because of that: neither Yandex’s help nor Google’s documentation, in these facts, measures bounce as a weight in the results. Closing a URL in robots.txt does not pull the page out of search.
Click fraud is a separate spend. Click fraud as a term hits ads, not the index line. Who presses the ads is covered in who clicks ads. Invalid clicks and a refund for click fraud are counted in the ad account, not in rankings.
Crawling and the index are different steps
Google’s documentation separates crawling and indexing. Disallowing Googlebot in robots.txt does not stop a URL from appearing in search results. To keep a URL out of the index you need noindex. To stop both a robot and a person from opening the page you need a different measure, such as a password. Blocking Googlebot affects Search, Discover, Images, Video, and News.
At Yandex the file limits crawling and can reduce load. Pages restricted in the file can still take part in search. To remove them you need noindex in the HTML or in an HTTP header, and those pages should not be closed in robots.txt: otherwise the robot will not see the instruction. If the file fails the requirements (no more than 500 KB, the name robots.txt at the root, a 200 response or a redirect to another robots.txt that returns 200), the site is treated as open for indexing. Cyrillic in the file and in the server’s HTTP headers is forbidden: the domain is written in Punycode, the path in the site’s encoding. Robots of other systems may read the directives differently.
Some Yandex robots fetch documents for reasons other than indexing and may not obey User-agent: *. The robot may bypass particular restrictions if Yandex has an agreement with the site owner. A foreign name in the header does not replace that. The check: the IP from the log, reverse DNS, the name must end in yandex.ru, yandex.net, or yandex.com, then a forward DNS lookup. If the forward address does not match the original, the name is fake. The help prefers this method to an IP list, because Yandex’s internal networks change.
Crawl budget and a spike
Google’s documentation calls the crawl budget the set of URLs Google can and wants to crawl. It is made of a capacity limit and demand. The capacity limit caps the total time the server keeps connections open for Google. Every site starts equally cautious. If responses are stable or faster, the limit rises. If the site slows, returns 5xx, or signals a rate limit, including HTTP 429, the limit falls and Google crawls less. Duplicates and extra URLs waste crawl. One crawler with high demand shares the overall capacity limit with the others. If crawling is limited by server power, the document lists adding server resources among the ways to raise the budget, if that is justified for the business. Figures such as a million pages are a rough estimate, the document says, not a threshold.
Yandex’s robot plans its own visits. A spike happens when it learns about a new section or addresses change. If loaded pages far outnumber pages in search, the help treats lookalike pages and duplicates as likely. Disallow stops requests to the named pages within a day after the file is edited. HTTP 429 is a sign the server is struggling: the robot will reduce the number of requests. Yandex treats subdomains as separate sites, so crawling each one can multiply the load. This is not someone else’s parser bill and not a hosting tariff.
| What the owner sees | What it means for search |
|---|---|
Disallow without noindex | Crawling is asked to be limited. The URL can remain in results |
noindex on a page closed in robots.txt | The robot may not see the instruction |
| Duplicates and extra URLs | They waste Google’s crawl budget |
| HTTP 429 and 5xx | Google crawls less. Yandex’s robot reduces requests on 429 |
| A “Yandex” User-Agent with no DNS check | The name may have been forged |
Someone else’s text on your site is a different policy
In Google Search spam policies, scraping is taking content from other sites, often automatically, and publishing it on your own site to influence ranking. Examples of a violation: reprinting with no original content and no source credit, light rewriting such as swapping synonyms, reproducing someone else’s feeds with no benefit to the reader, and sites that collect other people’s video and images without substantial added value. A site that violates the spam policies may rank lower or not appear in results.
“Scaled content abuse” names the generation of many pages from other people’s feeds and search results for rankings rather than for the reader. The policy describes the site that published someone else’s material. It does not say that the shop whose prices were copied loses rankings because of that. “Machine-generated traffic” on the same page means automated requests to Google itself, not bots on someone else’s shop.
A failed check does not punish a robot for the check’s own failure
A known search robot may be let through. The decision on the visit does not wait for this check to finish. A failed check is not treated as proof that the robot was forged. How the robot is recognized is not part of this fact. For SEO that is a boundary: an answer that never arrived does not turn a search crawl into a forgery, and it is not a measurement of a behavior factor.