How to clean analytics of bots so the reports stay honest

You cannot clean analytics of bots with one checkbox. Reports treat Direct’s filter differently. Reports in Metrica’s Direct group are built from clicks that passed antifraud. Other Metrica slices do not get that filter. Direct’s help text also says that every visit, fraudulent ones included, is written to the CRM and to analytics systems, so they still show activity that Direct’s reports have already filtered out.

Click fraud in this report is not “subtract a percentage”. It is a fork: you are looking at counted clicks, or at the full trace, including visits Direct no longer charges. The order below builds an honest report from that fork, from the yclid tag, from the definition of a goal, and from the “Form submission” goal.

Two layers you should not add together

Click fraud and sklikivanie split here by where the number lives.

In the Direct report group — summary, placements and costs — statistics count only clicks that click-fraud protection did not filter out. Invalid clicks are not spend inside this group. If the report for a manager is about money in the account, start here.

Other Metrica slices do not repeat Direct’s filter. The CRM and analytics systems, according to Direct’s help text, receive every visit, fraudulent ones included. A lead or a visit missing from a Direct report can still sit in the CRM. That is not a broken export and not proof that Direct “hid” a click: the help text says the filtered activity stays visible in analytics and in the CRM.

Who clicks ads, and why does not move this boundary. Why someone visited and whether the click stayed among counted clicks are different questions. Whether Yandex refunds click fraud does not clean the CRM either: a refund is about spend, and the help text leaves the visit’s trace in analytics.

Where to read yclid

Metrica’s tag help says Direct sets yclid. A sample link looks like https://example.com/?yclid=123. That illustrates the format, not your real link.

Ad visits are read in the Direct group. Until the report is built from that group, the presence or absence of yclid is easy to misread. The tag says Direct marked the link. It does not say the visit stayed among counted clicks: the Direct group keeps clicks protection did not filter, while the CRM can keep the rest.

For an honest report, keep three columns: visits in a Metrica slice outside the Direct group, visits and clicks inside the Direct group, and CRM rows for the same period. The help text explains a gap between the CRM and the Direct group in advance. A gap inside the group itself is a reason to check the goal, not the filter.

What a conversion is when you clean a report

The goals help calls a goal an action the site owner cares about: a number of views, a page, a click, a transition, a payment. An achievement is the moment the condition is met. A goal visit is a visit in which an achievement happened. In one such visit the same goal can fire more than once.

Conversion is the ratio of goal visits to all visits. Visitor conversion is the share of people who reached the goal among unique visitors. Goal revenue is either a fixed sum in the interface or a value passed as price or order_price.

Condition types include a goal event via JavaScript, the API or a file; views; pages; a click on email, phone or a button; form submission; a messenger; a social network; a file; site search; a return from a payment system; calls; chats; a multi-goal and a compound goal. Metrica creates some goals itself.

The list is a menu of conditions, not a quality score. A “button click” goal grows on real interest and on a repeated press. Cleaning a report “of bots” without naming the goal type means subtracting an unknown thing.

Repeated achievements inside one visit raise the achievement counter and do not raise the number of goal visits. If a report shows achievements next to conversion, those are different fractions. For account money, goal visits are closer. For form quality, what matters is whether the goal fires on a failed attempt.

The form goal easily counts too much

The help text records an achievement when the visitor presses the submit button. The form must be a form tag. A form built from a div is not tracked. The button is button type="submit" or input type="submit" inside that form. A type="button" button does not reach the goal.

If JavaScript handles the form, the goal works only for the standard onSubmit event. By default a failed attempt is counted too, when the data did not pass validation. To keep the attempt out of the goal, the help text suggests holding the button with disabled until validation passes. Any markup change can break the goal, so test it before you use it in ads.

A specific form is chosen by id, name or XPath. The address where you picked the form is not used when the achievement is checked.

Firing on an invalid submit is a property of the goal, not proof of a bot. A pile of “leads” can be a pile of submit presses that field validation rejected. Until the button stays disabled before validation, it is early to clean those achievements as fraud: the goal settings create them.

If the button cannot be changed, the help text points you to a “Goal event”. That replaces the condition. It is not a robot filter.

An honest report, in order

1. Name the period and decide which money you mean. Take account spend and clicks from the Direct group: summary, placements, costs. Only counted clicks are there.

2. Export the same period from the CRM or from a Metrica slice outside the Direct group. Expect a trace that includes visits Direct has already filtered. Do not call the gap an error.

3. Check that you recognise ad visits by yclid, which Direct sets. The sample yclid=123 is only a format.

4. Write down the goal the report sits on: the condition type, what an achievement is, what a goal visit is. Count conversion as the share of goal visits, not as the share of every press.

5. If the goal is “Form submission”, check the form tag, a submit button, the onSubmit event and disabled before validation succeeds. Otherwise failed attempts enter the goal.

6. After a markup change, test the goal again before a strategy starts learning from it.

7. Check goal revenue: a fixed sum in the interface, or price / order_price. Without that, a “cleaned” conversion is still without money.

8. Note separately if an external check result is sent to the counter. That mark is not Metrica’s own definition of robots and does not rewrite the ad system’s report.

What an on-site check can add

The script returns the check result to the page. If a Yandex Metrica counter is set, the result can go there too. That is a way to see the check next to the visit. It is not a “without robots” switch inside Metrica and it does not recalculate Direct spend.

While Direct-group reports and the CRM diverge by the help-text rule, an extra mark belongs in the report only with a label: the check mark does not replace counted clicks and does not delete a fraudulent trace from the CRM.

One week as an example

A teaching example, not a store’s measurement.

For the week the Direct group shows 800 counted clicks and 40 goal visits on “Form submission”. Conversion, by the help text, is 40 / 800. The CRM has 55 orders from the page on the same days. The gap of 15 does not have to be “bots Direct let through”: the help text says the CRM also stores visits that Direct’s reports have already filtered.

Inside the 40 goal visits there are 60 achievements, because some people pressed submit twice. Conversion does not become 60 / 800. If the button was not disabled before field validation, failed attempts sit among the 60. You remove them by fixing the goal, not by subtracting a “bot percentage”.

Some visit links contain yclid. They belong in the Direct group only if the click was counted. A visit with yclid that is not among counted clicks can remain in the CRM. In the manager’s report it goes in the column “visible in the CRM, absent from counted clicks”, not in the spend column.

If a check result also reached the counter, it is a third mark. It does not shrink 800 and it does not rewrite 55.

The table that keeps the report from mixing again

Do not subtract “other slices” from the CRM in the hope of getting “real people”. Those reports do not hand you a ready fraction of that kind. The rule is different: what Direct filtered stays visible in analytics and in the CRM. An honest report shows both numbers and labels which one is account money.

A markup checklist while the goal is lying

Use it when form conversion looks too good or too jagged. This is not a bot hunt. It is a check of the condition.

• The form is a form tag, not a div wrapper.

• The button inside the form is type="submit" (button or input). type="button" does not reach the goal.

• The handler is the standard onSubmit if JavaScript runs the form.

• The button is disabled until field validation succeeds, otherwise a failed attempt becomes an achievement.

• The specific form is chosen by id, name or XPath.

• After a markup edit the goal is tested again, before it is used in ads.

• Goal visits and the number of achievements sit apart in the report.

• Goal revenue is either a fixed sum or price / order_price.

• The Direct group for the same period sits beside it, not the whole counter.

• The CRM is labelled as the layer where Direct’s help text leaves fraudulent visits visible.

If the first six lines are empty, it is early to “clean bots” in this goal: the goal is not yet measuring the submission you mean. If they are filled and the CRM is still thicker than the Direct group, that is the layer gap the help text already describes, not a reason to rewrite counted clicks by hand.

Metrica creates some goals itself. An automatic goal in a manager’s report still needs a name: a view, a click, a form or a call. A nameless “conversion” does not get cleaner because the counter created it.

Put one sentence at the top of the report: “account money is the Direct group, the visit trace is the CRM, form quality is the goal after the markup check”. Without that sentence a reader will add 800 counted clicks, 55 CRM rows and 60 achievements and decide analytics is “a third off”. The three numbers answer three questions. Clean the number you named. Leave the others beside it, with the layer written down.

If the form goal was rebuilt during the week, do not average the old markup with the new. Before disabled, failed attempts entered achievements. After the change, only a validated submit does. An average conversion for the week mixes both modes and again looks like a “bot wave” on the day the button was still enabled. In the example, 60 achievements on 40 visits are a reason to ask which share fell on the days before disabled, not a reason to subtract a flat percentage from 800.