The content creator’s guide to a headless CMS

A/B testing, sometimes also called split testing or bucket testing, is an experiment that compares two or more versions of something, such as a landing page, creative, or tagline, to determine which performs best based on key metrics such as clicks, signups, or engagement. 

A/B testing tools for the headless CMS

After Google sunsetted its free tool, Google Optimize, the market for A/B testing tools attracted some new players. At Kontent.ai, we’ve found VWO to be the best fit for our needs. Like many A/B testing tools, VWO offers a 30-day free trial and flexible pricing based on monthly visitor counts. 

Setting up an A/B test with VWO is straightforward. Start by creating two slightly different landing pages. You can easily duplicate an existing landing page in Kontent.ai, make the necessary changes—such as updating the website copy using our AI features—and ensure that each version has a unique URL. 

Everything else is done inside the VWO tool. You insert your two tested URLs, name your versions, define metrics you want to use to determine the winner (you can also create custom events using Google Tag Manager), and finally, choose the audience for your A/B test. You can either use all traffic, or you can be more specific and show your test only to people coming from a certain marketing channel, device type, or operating system. You can even use it only for returning visitors.

The success of an A/B test hinges on the traffic your page receives. Testing on a page with just 100 visits won’t provide reliable insights. For meaningful results, aim to gather data from at least 1,000 visits—ideally, 1,000 visits per variant.

A/B test in practice

Our website has recently undergone many significant changes, so we implemented several A/B tests to ensure we got the most out of it.

One of our recent tests was optimizing the number of forms submitted on our Pricing page.

The original (control) version used an interactive slider to give a rough estimate of the volume discount.

The tested version consisted of a set of two calls to action that gave visitors a choice. 

And the results? The A/B test showed an expected drop in the conversion rate of 32% if we were to use the tested version.  

Since our Pricing page attracted substantial traffic, each version was tested with over 1,000 visitors, making it easy to confidently identify the winner.

Pro tip: If you run out of ideas for testing, you can use the VWO browser add-on, which will scan your page and suggest ideas based on your goals.