Protecting a site from bots: where to start
Contents
Share
Protecting a site from bots starts with who you do not want to hit, not with a ban list. Search crawlers, monitoring, and accessibility tools are legitimate. The job is to raise the cost of malicious automation. Click fraud is a separate loss of ad budget. The site and the ad account are closed by different actions.
Below is an order for the first day: what to check on Google’s crawler, how to lay out the layers, and how Cloudflare’s three products differ from a script that allows the visit when the check fails.
Do not stop everyone, and do not trust the string
OWASP assembles protection from three layers: the network edge, the application, and business rules. A single trick is fragile. Before you pick a trick, sort catalog events by point. Sign-in, registration, the catalog, and checkout are built differently. A lead form does not get the same rules as browsing product cards.
Google’s documentation says other crawlers often put on the Google crawler’s User-Agent. Verify the request with reverse DNS on the IP, or by matching the IP to published ranges, not by the header text. Google’s ordinary crawlers honor robots.txt on automatic fetches. Special-case crawlers may not. User-triggered fetchers ignore robots.txt because a person started the request.
The first ban of the day: do not block the User-Agent “Googlebot.” You cut both the spoof and a check you never ran. DNS or the ranges come first. Separately, do not punish a fetcher a person started: in this description it ignores robots.txt for a stated reason.
One switch on the whole domain is already a choice
Cloudflare’s documentation splits bot products in three.
Bot Fight Mode on the free plan meets detected bots with a computational challenge on the whole domain, from one switch. There are no rules of your own. There is no bypass through Skip in WAF rules.
Super Bot Fight Mode on Pro, Business, and Enterprise, without the separate Bot Management add-on, gives actions by category and exclusions through WAF rules, but not a per-request score.
Bot Management for Enterprise gives a score on the request and its own rules.
On the scenario for Free, Pro, and Business, a score of 1 means the request is almost certainly automated, scores 2–29 mean probably automated, and scores 30–99 are scores the documentation asks you not to challenge or block as a human. Score-distribution analytics start on the Business plan. The same descriptions say Bot Fight Mode and Super Bot Fight Mode are aggressive by design and false positives are expected, especially in the first days.
The scores are Cloudflare’s scale, not Binetro’s and not a Direct filter. Rule expressions and sample rate thresholds are not part of this order. The public overviews do not say the product removes invalid clicks in Direct or Google Ads.
That is the first fork. The free switch on the whole domain cannot exclude. If the domain has search, a catalog, and a form, you challenge every detected bot, including people you will catch by mistake in the first days. The paid modes add categories and WAF exclusions. A per-request score is a separate Enterprise product. The documentation asks you not to challenge or block scores 30–99 as a human: “a low score is definitely a bot” does not apply to that band.
If the check is silent, the person is not held
With Binetro, if the script got no response — an error, a timeout, or a 5xx — the visit is allowed. The check itself has no such mode: the script on the site allows the visit when the check fails. If the response could not be assembled, the visitor is allowed as well. That is an honest place to start: protection should not turn its own failure into a refusal of a live person.
The script does not promise to stop every bot and does not refund a click. The ad platform charges the click.
Order for the first day
Write down the points: sign-in, registration, catalog, checkout, lead form. Each has its own harm. Do not copy a trick from sign-in onto the catalog.
Do not close legitimate crawlers with a User-Agent string. Verify Google with reverse DNS or published ranges. Separate an ordinary automatic fetch from a fetcher a person requested.
Pick a layer. The network edge meets the request before the application. The application looks at the form and at sign-in. A business rule decides which lead goes to work. Leave a single layer as knowingly fragile.
If you take a Cloudflare product, write down which of the three. The free mode is a challenge on the whole domain, with no rules of your own and no Skip. Super Bot Fight Mode is categories and WAF exclusions, with no per-request score. Enterprise Bot Management is a score and its own rules. Do not block scores 30–99 as a human. Expect false positives in the first days on the aggressive modes.
Do not read Cloudflare scores as a Direct filter, and do not expect a click to be refunded.
Check the failure path. Binetro’s script allows the visit if there is no response or the response is 5xx, and allows it if the response could not be assembled. A mode of “silence means block” is not built into this check.
Checklist that the day is finished:
The points are written separately, and the catalog does not carry the sign-in form’s rule.
The blocks do not contain a bare Googlebot User-Agent.
You did not choose one trick “for everything.”
If Bot Fight Mode is on, you accepted a challenge on the whole domain and false positives in the first days.
The score band 30–99 is not in a “block” rule.
A failed check on the site allows the person through instead of showing an access error.
The advertising question is set aside: who clicks ads, click fraud and sklikivanie, invalid clicks, whether Yandex returns the money.
Walk a domain that has a form, a catalog, and search crawling as three points, not as one switch. On the catalog the harm is an automated client you have not named yet: a legitimate search crawler belongs in the conversation, malicious collection is a separate event, and one trick for both is fragile. On the form the harm is a submission, and a business rule decides which lead goes to work. On sign-in the harm is automated attempts. The network edge meets the request before the application. If you turned on only the edge, the application and the lead rule stay empty, and OWASP treats that single trick as fragile.
Do not ban Google’s crawler by string on this day. Reverse DNS on the IP, or a match against published ranges, is the check Google’s documentation asks for. An ordinary automatic fetch honors robots.txt. A special case may not. A fetcher a person started ignores robots.txt because a person initiated the request. Blocking the string “Googlebot” cuts both the spoof and a check you never ran.
Write the Cloudflare product down by name. Free Bot Fight Mode is a computational challenge on the whole domain, with no rules of your own and no Skip in the WAF. Super Bot Fight Mode is actions by category and exclusions through WAF rules, with no per-request score. Bot Management on Enterprise is a score and its own rules. On the Free, Pro, and Business scenario, score 1 is almost certainly automated, 2–29 is probably automated, and 30–99 are not challenged or blocked as a human. Score analytics start on Business. The aggressive modes expect false positives in the first days. The scores do not carry into Direct and do not refund a click.
Check a Binetro failure separately from Cloudflare. No response, a timeout, a 5xx, or a response that could not be assembled: the visit is allowed by the script on the site. The check itself has no mode of “stay silent and block.” That is how protection avoids turning its own error into a refusal of a person.
Write the day’s decision as three rows, with no new thresholds. The “catalog” row: a search crawler is verified by DNS or by published ranges, the User-Agent string is not banned, and a fetcher a person requested is not punished for robots.txt. The “form and sign-in” row: its own application layer and its own business rule, not a copy of the catalog rule. The “whole domain” row: if Bot Fight Mode is on, you accepted a challenge for every detected bot and false positives in the first days; if you need a score on the request, that is Bot Management for Enterprise, and the band 30–99 is not blocked as a human. A separate note sits beside the table: a failed Binetro script allows the visit. An advertising click is not in the table. The platform decides that.
Repeat the table a week later only for rows where, in seven days, a live person was refused by mistake or a catalog fetch was missed. A trick that did not move is what OWASP already called fragile: add the missing layer, not a second switch of the same kind. Library names and rate thresholds are not part of that repeat.
Say the advertising boundary out loud so it is not lost by evening. Cloudflare’s overviews do not say a bot mode removes invalid clicks in Direct or Google Ads. Score 1 and the band 2–29 describe how automatic a request is on their scale. They do not say whether a click had an interest in the ad. Binetro’s script allows the visit when the check fails, and that does not promise to return the budget: the ad platform charges the money. So at the end of the day you have two lists. The site list: the points, a crawler check that is not a string, the Cloudflare product you chose with its limits, and an allow when the script is silent. The account list is run under the platform’s rules, not under a score and not under whether the script answered.
If the domain has only a lead form and no catalog, the “catalog” row is still needed: a search crawler may fetch pages you do not think of as a shop. OWASP sets legitimate crawlers outside the job of “stop everyone.” Starting with a ban on the Googlebot string breaks that first step, even if the form is already covered by another trick.
At the end of the first day, one check is left, on yourself. The page opens and the form submits. Then break the check response — a timeout, an error, or a 5xx — and confirm the script allows the visit through instead of showing a refusal. If the response could not be assembled, the visitor is allowed as well. That is Binetro’s limit: the script on the site allows the visit when the check fails, and the check itself has no such mode. Do not close a search crawler in that same minute by User-Agent. A doubtful name is verified by reverse DNS or by published ranges.