Pros and cons to using AB testing on your website


A small business owner’s guide to split (a.k.a AB) testing

Last month our blog was all about dynamic text replacement and whether this could be something which might help you to grow your sales. We also talked about split (or A/B) testing as a way to check whether dynamic text replacement has a positive impact on your customers.

So, what is split testing?

Split testing is when customers are shown different versions of a live website to interact with. You can then measure which version of your website gives you the best results, such as more sales, a fantastic piece of information for website owners.

Sounds great, what’s the catch?

Running this testing can result in customers getting a varied experience whilst using your website, this might not be the best service they were expecting. How long can you afford to run the testing for, knowing that this may be having a negative impact on your existing customers?

Here are the pros for split testing:

You can test out your new ideas: You get solid proof as to whether a new idea works or not. Although it does mean that you have to develop this new idea in to your website before you can test whether it works.

You can make changes gradually: You can test out small changes to a page on your website, rather than changing the whole website and see if there is a measurable difference in customer behaviour.

You can answer design questions: You can test whether different colours, buttons, layouts or images make a difference to the behaviour of your customers.

And most importantly the results for all of the above tests will be definitive. E.g. Site A took more money than site B. Clear results to inform business decisions about your website.

And here are the cons for split testing:

Split testing takes time: Setting up two different versions of web pages takes a long time, as well as physically creating the two versions of the page/s in question you will need to agree variables to be tested. Testing can also need to be carried out over several weeks or months for low traffic sites to get enough data. You may then decide you need to test another set of variables which will start the process again.

Split testing doesn’t solve usability problems: Split testing does not measure or show if there are some frustrating usability issues with your website. If you suspect your website is not very easy for your customers to navigate, it is best to resolve these issues, potentially using usability testing to identify what those are.

Split testing works best with one variable: Split testing is perfect if you are trying to decide between two different options only. However, if you have multiple queries the results may become inconclusive.

So, in summary:

Split testing is an increasingly popular tool for website owners and is great for making option A or option B type design decisions. It can also be used with dynamic text replacement, does your customer respond well to customised text on your website or not? However, split testing should be used as part of a wider testing process if there are more variables which need to be considered. Usability testing is a great way to check ease of use for your website