Make your user experience perfect with A/B testing

What is A/B testing

A/B testing is a randomized experiment in which you test two or more variants of your content against each other. The variants differ in some aspects. You’re comparing which variant gets you closer to your goal. The variants are split among your audience, either evenly, or based on weights given to each variant. Splitting is usually done evenly, but uneven distribution may come in handy if you want to limit the number of visitors who see the newly tested version. That may be the case if the new version contains something about which you don't want to go public in full yet, for instance.

Do the testing right

Here are some tips to help you get the most out of your testing. These pieces of advice are valid regardless of the platform you (plan to) test your content on.

1. Define your goal

The first thing is to define a goal you want to reach. What do you want your audience to do? Purchase your product? Sign up for a newsletter? Spend more time on your website? And what success rate do you need?

2. Identify the problem and make a hypothesis

Now that you have your goal, identify the problem – what is it that keeps you from reaching that goal? Analyze the data about your visitors and develop a hypothesis – what is it that prevents them from taking action? Narrow down the possible causes, so that you don't need to test every single element in your app.

3. Test the hypothesis

Once you have the possible causes and their possible solutions, test them using A/B tests. Make sure to always test only one small change. That way, you don't get lost in what was it exactly that lead to a change in your users' behavior.

For any testing of this kind, you need a big-enough audience. Testing a change on 50 visitors won't give you definitive results because there's a high chance of a coincidence. The optimum is 1000+ visitors per variant for the results to be statistically valid. After the test is finished, re-test the change after some time to eliminate false positives. This is important because the typical pain point of A/B testing is that the test usually lasts a fairly short time. Re-testing helps you make sure that the change in your customers' behavior wasn't a time-limited fluke caused by, say, upcoming Christmas. If the same test yields similar results repeatedly, the results are way more trust-worthy.

4. Evaluate the results

When your testing is done, look at the results and see if any of the tested variants is getting you closer to your goal. If it is, great! Implement it, and optionally start testing another hypothesis to improve your app further. Remember to always look at the results in the context of your whole business. For example, one change may skyrocket the number of individual purchases, but did your total revenue increase correspondingly? You may now have a larger number of smaller purchases which, when summed up, total in a smaller revenue overall.

If the hypothesis didn't prove right, test another one. A/B testing is an iterative process of trial and error, one false step doesn't mean you should stop trying. 🙂

How we A/B test in Kontent.ai

Now that we have the theory covered, let's see how we, at Kontent.ai, do A/B tests. We cover the approach that's proven successful for us and our website here.We have our content highly structured using components and linked items. This makes it easy to select one for testing and leave other parts of the page untouched.