How Google and Yandex slow their own crawl
Google lowers crawl budget when a site slows down, returns 5xx, or sends 429. Yandex reduces requests on a 429. Google does not support Crawl-delay, and 401 or 403 do not lower its crawl rate.
6 min read
Website bot protection
How bots affect SEO and on-site behavior metrics
How Google and Yandex crawling relates to the index, why robots.txt does not drop a page from search, and when a foreign User-Agent is not yet a search robot.
6 min readHow to stop spam signups and fake accounts
New accounts are not stolen-password checks. Confirm email before the account is live, do not ban on the first weak signal, and do not trust a browser email check.
7 min readrobots.txt: what it can and cannot do against bots
robots.txt asks compliant robots to stay off certain URLs. It does not lock a page, does not prove a User-Agent name, and does not replace noindex.
6 min readCredential stuffing: bots replaying stolen passwords
This is a check of stolen login-password pairs, not guessing. How stuffing differs from cracking and spraying, and why OWASP puts a second factor first.
6 min readWhat website scraping is and how it hurts a business
What OWASP means by scraping, why robots.txt does not hide a page, and how Google’s spam policy hits the site that republishes someone else’s content.
6 min readProtecting a site from bots: where to start
First name the point — sign-in, registration, catalog, or checkout. Then verify a crawler by more than the User-Agent string, and do not put one switch on the whole domain.
11 min readProtecting an online shop from scraping and bots
Price copying, buying out scarce stock, a cart left unpaid, and stolen passwords are different events. What they share with routine invalid traffic, and an order for the shop.
10 min readWhat a WAF is and how it differs from bot protection
A WAF is a rule set for the HTTP conversation, usually against XSS and SQL injection. Why Cloudflare documents bots on another page, and why one edge filter is not enough.
5 min readDDoS and bot traffic: what is different
A distributed network outage and automated application traffic are different events. Where the OWASP catalog draws the line, how NIST defines DDoS, and what the Cloudflare product covers.
5 min readScalper bots: buying out stock and booking slots
Scalping takes scarce goods in a way a person cannot repeat by hand. How that differs from a cart left unpaid, and the first controls OWASP names.
6 min readHow to protect prices and a catalog from scraping
robots.txt asks compliant clients to skip some addresses and does not control access. What a 429 does for Yandex's crawler, and which OWASP measures apply to scarce stock.
6 min read