Tags – A/B Testing

 

When was the last time a marketer brought up A/B testing in front of you?

For us, it happened last month. To be fair, A/B testing is such a darling of marketers that I get to hear it every month, and I have been hearing it since I came into marketing in mid 2018.

At the beginning of my marketing journey, I was doing my PhD, and I can tell you quite certainly: the way A/B testing is conducted in marketing is not really scientific; it feels like more of a bad effort to learn from academia.

Here’s how:

 

Red or Pink

This is my favourite example:

Should the buttons on my website be red or pink?

Well, why don’t you run an A/B test?

It’s a great hope that just looking at the results after a few days will conclusively tell you what colour your buttons should be. However, scientifically speaking, you are still taking a punt.

Sadly, things don’t work like that in real life.

To put it simply, and to the point, any test you are conducting to determine the colour of your website buttons is inconclusive because your testing environment is not controlled.

 

Not a Controlled Testing Environment

If you are looking to conclusively determine the behaviour of a group, you need to make sure that the following conditions are met:

  • Uniform conditions for everyone taking part in the experiment
  • All the influences must be known

This means, for an A/B test, for example, all actions must be taken at the exact same time, same kind of environmental conditions, and using the same equipment. In addition, you must also understand what else can influence the behaviour of the members taking part in the experiment; everything from their mood to what information was provided. Any change in these basic conditions can otherwise influence the final results, making the whole experiment invalid.

Now, you might be thinking: what if we accept the randomness and take some form of indication from the A/B testing. Well, the experiment will still be invalid because, in most cases, your sample size is too small to be statistically significant.

 

Statistical Significance

Before I explain ‘statistical significance’, you need to understand what a ‘null hypothesis’ is:

A hypothesis proposing that there is negligible difference between certain characteristics within a defined situation.

If that sounds complicated, this example will simplify:

Let’s say: you are looking to grow some flowers and you have decided to use a brand of bottled water. Will it make a difference if you use, let’s say: Evian or Buxton?

Statistically speaking: no. And even if there is a difference between the two kinds of water, it is simply by chance, as the two kinds of water mentioned above are priced relatively the same way, and, probably the most obvious reason: they are both brands of bottled ‘water’.

Building on this knowledge, ‘statistical significance’ testing is the process of figuring out if a null hypothesis can be rejected or retained.

For an A/B test, I will argue that 75% people need to choose one specific option, once the conditions are known, to consider that we have reached statistical significance. And I am being lenient because the full statement will still be: there is a 25% chance that you are wrong. In reality, you would want the acceptance rate to be over 90%, which is almost not achievable in marketing for any options considered, given that most people have personal preferences that can’t be mapped or their choices are extensively impacted by their surroundings.

 

Concluding Remarks

There are way too many things you cannot control in a marketing A/B test. For example, while running an A/B test on your website:

  • What if the internet connection of the user is unstable and they randomly clicked on something
  • What if you had server issues and things were not shown at all, or not shown the way you intended them to be displayed

Above all, the users were never presented with all their options to begin with, to make it a real choice. Or, simply, as I said earlier, you did not gather enough information to make the answer more conclusive.

And, I haven’t even brought up whether or not the hypothesis that you were testing was even right. You may have been asking the wrong question all along and never knew.

Saying all that, I am not discouraging the use of scientific methods; I just want people to use their head before using a method, instead of just picking something that is popular.

 

To learn more, get in touch with us today.

In the meantime, please check our digital marketing services.

 

You may also like:

  1. Is There A Science and Art to Developing Healthy Business Ideas?
  2. Philosophy of Science? – Thinking about Business, Marketing & Einstein
  3. Science of Growth, Association & Fake Business Awards – Our Experiences
  4. 5 Tips to Create the Perfect Atmosphere at Your Networking Event
  5. How to Design a Logo to Take Your Brand Up a Notch?