What website scraping is and how it hurts a business
Contents
Share
Website scraping is the collection of content that is already available, so it can be used somewhere else: copied prices, a catalog, or the rendered output of a page. The business harm starts when another shop publishes your data or keeps pulling the catalog. A disallow rule in robots.txt does not make the page unreachable.
The boundary below is between a request to a robot, indexing, and the collection OWASP describes separately from an ordinary crawl.
What counts as scraping
OAT-011 Scraping in the OWASP catalog is the collection of application content and other data for use elsewhere. The description talks about copying available data and processed output. Among the goals are exploiting the application’s own data and influencing prices. Collection can run continuously or in bursts, with a fake or taken-over account, or with no login. If another application sits between a person and the app, the page points to OAT-020. If the goal is to obtain money or goods, it points to OAT-012.
That page does not measure a drop in the source site’s rankings. Prices leaving the shop is not, by itself, a verdict on that shop’s search results.
Google Search’s spam policy uses “scraping” differently: taking content from other sites, often automatically, and publishing it on your own site to influence ranking. It hits the site that published someone else’s material. Examples of a violation: reprinting with no original content and no source credit, light rewriting such as swapping synonyms, reproducing someone else’s feeds with no benefit to the reader, and sites that collect other people’s video and images without substantial added value. A site that violates the spam policies may rank lower or not appear in results.
Separately, “scaled content abuse” names the generation of many pages from other people’s feeds and search results for rankings rather than for the reader. The policy does not say that the shop whose prices were copied loses rankings because of that. “Machine-generated traffic” on the same page means automated requests to Google itself, not bots on someone else’s shop.
robots.txt does not close the page
Google’s documentation separates crawling and indexing. Disallowing Googlebot in robots.txt does not stop a URL from appearing in search results. To keep a URL out of the index you need noindex. To stop both a robot and a person from opening the page you need a different measure, such as a password. Blocking Googlebot affects Search, Discover, Images, Video, and News.
RFC 9309 defines the Robots Exclusion Protocol: the service owner asks automatic clients, in robots.txt, to follow access rules for URIs. Section 1 states outright that these rules are not a form of access authorization. A group with no rules at the end of the file implicitly allows everything. If a robot token has neither its own group nor a * group, the rules do not apply. The URI of /robots.txt itself is implicitly allowed. The RFC describes clients that follow the protocol. It does not promise that an arbitrary program will read it.
Yandex does not pull a disallowed page out of search either. Webmaster help says the file limits crawling and can reduce load. Pages restricted in the file can still take part in Yandex search. To remove them from search you need noindex in the HTML or in an HTTP header, and those pages should not be closed in robots.txt: otherwise the robot will not see the instruction. If the file fails the requirements (no more than 500 KB, the name robots.txt at the root, a 200 response or a redirect to another robots.txt that returns 200), the site is treated as open for indexing. Cyrillic in the file and in the server’s HTTP headers is forbidden: the domain is written in Punycode, the path in the site’s encoding. Robots of other systems may read the directives differently.
| Measure | What it does | What it does not do |
|---|---|---|
Disallow in robots.txt | Asks a compliant robot not to fetch the URI | Does not authorize access or hide the URL from a person |
noindex | Asks not to index | Does not stop opening a page that is reachable |
| A password | Closes the page to both a robot and a person | That is no longer a rules file |
How much automation one vendor’s network sees
Imperva’s 12th Bad Bot Report (a 2025 document) says that in 2024 automated traffic exceeded human traffic for the first time in ten years and made up 51% of web traffic in their observation. Malicious bots were 37% of all that traffic, against 32% in 2023; ordinary bots were 14%; people were 49%. The report builds the figures from Imperva’s global network for 2024, including 13 trillion blocked malicious-bot requests across thousands of domains and industries. These are vendor data.
The denominator is traffic Imperva’s network sees; the report calls it internet traffic. It is not a census of the whole internet, not the share of spam in forms, and not a hosting bill. The 13 trillion figure is blocked requests, not the same 37% of traffic.
The same report says about 44% of advanced bot traffic was aimed at APIs and about 10% at web applications. In the breakdown of bot attacks on APIs, data collection is about 31%, payment fraud about 26%, account takeover about 12%, and scalping about 11%. There the report describes scalping as the rapid purchase or reservation of a large volume of scarce goods and services. The percentages are shares in the API-attack breakdown for Imperva’s network clients, not a share of a shop’s orders.
Catalog collection and click fraud are different automated acts. Click fraud as a term hits ad spend. Scraping hits data the page already returned. Who presses the ads is covered in who clicks ads. Invalid clicks and a refund for click fraud do not apply to a copied catalog: the ad account does not refund money because prices moved to another site.