A/B testing: what your traffic lets you conclude

The article on conversion rate sets the limit and the one on landing pages points at it: comparing two versions takes a volume most Algerian sites do not have. This page starts from there.

A test is not an opinion settled by numbers. It is a procedure answering one narrow question — is this difference real or chance — and it answers nothing else. Used outside its domain it produces random decisions presented as a method.

The domain is defined by volume. Below it you do not get a weak result: you get something that is not a result, and experience shows somebody always reads it as a win.

This article says how to know which side of the limit you are on, what to do on each side, and why we will not publish the figure everybody asks for — because that figure does not exist independently of you.

The most destructive fault is not lack of volume, it is daily checking. A test looked at every day always ends up showing a gap, because chance produces gaps constantly.

The mechanism is easy to feel: if you toss a coin a hundred times and stop the moment heads is six ahead, you have manufactured proof that the coin is loaded. That is exactly what a dashboard checked every morning does.

The rule is therefore to fix the duration and the volume before launching, to decide nothing before, and not to stop because one version has pulled ahead. Discipline matters more here than anywhere else in this pillar.

The second rule of the same kind is not to relaunch a lost test hoping for better. Repeating until the desired result appears is the same error, spread over weeks and much harder to spot in a report.

One unpleasant consequence for an agency: most honest tests end with no winner. A report that always announces one describes a reading practice, not a series of discoveries.

One local particularity makes comparisons over time even less reliable than elsewhere, and it is why we insist on simultaneous groups rather than successive periods.

Traffic and buying behaviour move with Ramadan and Eid, with the pay week, with the start of the school year, with the regulated sales periods and with your trade’s seasons. Those movements are large, and they are larger than most of the effects you are trying to measure.

The consequence is direct: comparing this week to last week is not a test, it is a seasonal observation. We regularly see "+40%" results that describe the pay week and nothing else.

The only protection is simultaneity: both versions run at the same time, on the same traffic, split at random. The whole value of the method sits in that sentence, and it is what separates it from a before-and-after.

And for genuinely particular periods — the two weeks before Eid, for instance — the right decision is often not to test at all. What you would learn describes a situation that does not come back for a year.

The durable deliverable of this service is not a test, it is a log. One line per entry: the date, what was changed, on which page, why, and what was observed afterwards.

Without that document a business redoes the same modifications every eighteen months, at the rhythm of supplier changes. We have seen it three times on one site: the price published, then removed, then published, each time with conviction.

The log also serves to interpret measurements. A fall in conversion in March has a different explanation if the form changed on 3 March, and that information exists nowhere else — not in the measurement tool and not in anybody’s memory.

It has to contain failures as much as successes, and above all inconclusive tests. A test that showed nothing is real information: it says the gap, if there is one, is smaller than what you can detect.

Two lines per entry is enough. What matters is that the file exists, that it lives with you, and that it survives the departure of whoever started it.

What we will refuse: launching a test on a site whose volume does not allow a conclusion. It is a frequent refusal, it is badly received, and the alternative we propose is cheaper — which does not help it be accepted.

We will refuse to stop a test because one version has pulled ahead, and to relaunch a lost test hoping for better. Both manufacture winners, and a report that always announces one describes a reading practice.

We will refuse to present a result as a percentage without the absolute numbers, and to conclude on a subgroup that was not decided before launch.

What we do: the preliminary calculation that says which side of the limit you are on; the queue of obvious corrections when the answer is no; three-full-week tests on the offer rather than the form; the result measured on the accepted order; and the change log, kept at your premises.

And what you can do this week without us: take last month’s count of accepted orders and divide it by two. Look at that number. If you would not be willing to build a decision on it, you have just learned that the next useful expense is not a test.