How to protect prices and a catalog from scraping
Contents
Share
A robots.txt file does not close a catalog against scraping. RFC 9309 describes it as a request that automatic clients honor access rules for addresses. The rules are not an authorization to access. A client that does not read the file simply never sees the request.
Scraping in the OWASP catalog is collecting available content in order to use it somewhere else, including to influence prices. Below is what actually changes a search crawler's path, and which measures belong to scarce stock rather than to someone else's price list.
What robots.txt can do and what it does not promise
The service owner uses robots.txt to ask automatic clients to honor access rules for URIs. A group with no rules at the end of the file implicitly allows everything. If a robot token has no group of its own and there is no * group, the rules do not apply. The URI of /robots.txt itself is allowed implicitly. The RFC describes clients that honor the protocol. It does not promise that an arbitrary program will read it. Click fraud is not treated by this file: the file is about crawling addresses, not about a click on an ad.
In Google's reading, the supported fields are user-agent, allow, disallow, and sitemap. crawl-delay is not on the supported list. A file larger than 500 KiB: everything past the limit is ignored. A 4xx response, other than 429, is read by Google as if there were no valid robots.txt, so there are no crawl restrictions. The help says not to use 401 and 403 to lower crawl frequency: they do not affect it. A crawl disallow stops the page content from being indexed, but the address can remain in results without a snippet. These are Googlebot's rules, not a third-party price parser's.
Since 22 February 2018 Yandex has stopped honoring the Crawl-delay directive. The speed at which crawlers load pages is set in Webmaster on the "Crawl speed" page, separately for the main domain and each subdomain. The default is "Trust Yandex": speed is chosen so more pages are loaded without extra load. A value set by hand is a recommendation for the crawler: the actual speed can differ.
How Yandex's crawler lowers load
The indexing crawler plans its own visits and regulates load by how many new and already known pages it needs to crawl. A spike happens when the crawler learns of a new section or addresses change. If loaded pages greatly outnumber pages in search, the help treats lookalike pages and duplicates as likely. A Disallow in robots.txt stops requests to the named pages within a day after the file is edited. The help calls an HTTP 429 a sign that the server is under strain: the crawler will reduce the number of requests. Yandex treats subdomains as separate sites, so a crawl of each one can multiply the load on the server.
This is about Yandex's crawler, not an arbitrary parser and not a hosting bill in rubles. The Clean-param directive drops parameters such as UTM from indexing, and this help does not apply the "within a day" period to it. You cannot close prices against a third-party collector with a 429: a 429 tells your crawler to slow down, it does not forbid copying the catalog.
What OWASP calls scraping
OAT-011 Scraping is collecting application content and other data for use somewhere else. The description talks about copying available data and processed output. Among the purposes are exploiting the application's own data and influencing prices. Collection can run continuously or in periods, with a fake or captured account or with no login. If another application stands between the person and the application, the description points to another event. If the aim is to obtain money or goods, it points to another event as well. The page does not set a crawl order or a drop in the source site's rankings.
Invalid clicks do not describe price collection. Click fraud is about the click and the impression. Who clicks ads does not copy a catalog. A refund for click fraud does not depend on price-list scraping.
What to put on scarce stock
OWASP's guidance on scalping and holding stock names the first measures: a queue or a waiting room for a limited release, a purchase cap per account that the server checks, and holding an item in the cart only until payment, after which the stock returns. The cap is to be counted not only by the account but also by repeats of the same payer, address, and device. A number of seconds to hold the item is not part of this order. A queue does not promise to close price scraping: it is about goods that get bought up, not about reading the storefront. The event number in the guidance and in the project table disagree, so take the measure, not the number.
Do not treat Disallow as price protection. It is a request to clients that honor it. After a disallow, Google can leave the address in results without a snippet.
Do not set Crawl-delay. Google does not support the field. Yandex has not honored it since 22 February 2018 and takes speed from Webmaster.
Do not return 401 and 403 to "slow the crawler." They do not affect Google's crawl frequency. A 4xx other than 429 is read by Google as no valid robots.txt.
Leave 429 as a load signal for Yandex's crawler. A third-party price collector is not stopped by it.
For scarce stock, set a queue, a server-side cap on the buyer, the payer, the address, and the device, and return the stock if the cart is not paid.
Watch the file: Google ignores everything past 500 KiB, and a group with no rules at the end implicitly allows everything.