A/B Testing – Answers to the Top 5 Questions of 2012 (with case studies, videos & links)

Why do we need to test? What should we test? How do we get started? How do we set up an A/B split test? When can we stop the test? 

Those are the top 5 questions I got in 2012 when I talked to new clients or spoke at summits on the subject of conversation optimization and A/B testing. In this article, I’ll answer each of these questions one at a time and provide you with lots of tips, case studies, examples, videos, and useful links.

1. Why do we need to test?

I love answering this question! The only problem is that there are so many great answers that I could easily go off on a tangent and talk about it all night. So for your sake, I’ll try to keep it simple and stick to the best answer I know: “You need to test because relying on guesswork and gut feeling is a dangerous business strategy.” Now let me expand on that answer and explain what I mean in more detail.

Marketing is not an exact science. All products, services, offers, and websites are different, just like the motivations of your potential customers are different. We’re dealing with real people and real decisions. And frustrating as it may be, the truth is that people don’t always act the way marketers want them to.

If you don’t test whether your optimization efforts are in fact optimizing the performance of your website – you are really relying on guesswork and gut feeling.

Even with hundreds of A/B tests under my belt, test results surprise me on a regular basis. Sometimes positively – other times not quite so positively. Here’s an example from a case study where all my experience (as well as all logic) told me that my bad ass landing page treatment would surely knock the socks off the control page. However, the test results showed that my variant actually performed significantly worse than the control page.

Had I not insisted on putting my work to the test, and instead blindly trusted in my experience and intuition, the client would have spent money on getting a new a landing page that actually hurt conversions. Moreover, I would have been in trouble, when the client at some point would come to realize that conversions went down after they hired me.

The interesting thing about the above mentioned case is that, even though the first test had a negative test result, it wasn’t a bad test. It revealed a number of findings that lead to a final treatment that outperformed the original control page by 48.69%. The beauty of testing is that, as long as you conduct experiments that make you wiser and give you insight, even negative test results can lead to positive lifts. Check out the full case study here >>

Useful Links:

This article by Peep from ConversionXL.com is a great read for beginners as it highlight some of the hard truths about A/B testing. Read the article here >>

Neil Patel wrote this awesome article summing up what he learned from spending $252,000 On Conversion Rate Optimization. Check out the article here >>

2. How do we get started?

One of the main things keeping businesses from getting into A/B testing is the fear of having to invest in a huge elaborate technical platform, involving tons of coding and IT assistance. Add to this the common misconception of A/B testing being extremely costly and complex, and it’s no wonder that most businesses never get started.

I can’t count the times I’ve been approached by employees from marketing departments who are desperately interested in A/B testing and CRO but can’t get the company buy-in to actually get started on a serious program.

My best advice is “Start small and simple and get your footing.”

You don’t need to invest in a ridiculously expensive technical setup, and you don’t have to do a complete redesign of your website right off the bat.

If you use a tool like Visual Website Optimizer, you can get started in no time for as little as $50 a month. VWO is easy to use, and you can perform a large range of tests with little or no knowledge of code. Moreover, you’ll only need to involve IT to the extent that they have to implement a bit of Javascript on the website (Watch the video under question 4 for a tutorial on how to set up an A/B test in 3 minutes with VWO).

Once the technical setup is in place, you can start thinking about setting up your first split test. Here my advice is “Start simple but use your energy where it counts.”

Start with a simple test that is likely to yield results. You could pick the newsletter signup form, the call-to-action on the home page, or the headline and sales copy on the main landing page.

If use your energy where it counts and pick mission critical pages that get a lot of traffic, small changes can give big results. Here’s an example from a case study where changing one word on the call-to-action increased conversions by 38.26%Read the full case study here >>

Once you have a few tests with positive results, it will be much easier to get the buy-in from coworkers as well as management. I like to use the argument “Look at what we did with a minimum investment of time and money – imagine what we could do if we kick it up a notch!”

Once you get started, it’s worthwhile to “strike while the iron is hot” and take advantage of the momentum that the initial excitement generates. I’ve seen several great CRO initiatives drown in bureaucracy and overplanning despite the best intentions.

Useful links:   

I wrote a guest post for Unbounce.com on how to sell A/B testing to a skeptical client. The methods described in the article can easily be applied internally in an organization or company. Check out the article here >>

I highly recommend MarketingExperiments’ online certification course The Fundamentals of Online Testing. Th course pretty much covers everything you need to know in order to get off to a flying start. Check out the course here >> 

3. What should we test?

There are really two steps involved with finding out what to test. First you need to find out where on the website to start testing (what page), then you need to find out what to experiment with on the page.

As I mentioned earlier in this article, it’s a good idea to start simple but to focus your efforts where it counts. What that translates into in this context is that you should pick mission critical pages that get a lot of traffic for your initial experiements.

Your most important landing pages are a great place to start. They usually get a high traffic volume and are directed at getting potential clients to carry out one specific goal – which makes for a clean and focused test design.

Moreover, landing pages are a great place to experiment, because they are like a little secluded test environment where to can test different hypotheses without affecting the rest of the website. Later on you can implement the things that worked globally on your website.

Some of the most successful redesigns I’ve been involved with have been based on findings from an initial phase of landing page optimization and testing.

Of course your analytics data is an obvious source for locating pages and steps in your funnel that could do with a bit of work.

Once you know which pages are worth testing, you need to find out what to test on the page itself. Again I recommend keeping it simple but focusing on elements that are most likely to impact the decisions of your potential customers.

Call-to-action buttons and call-to-action copy are low-hanging fruits that have high impact on conversions. Also form copy on lead generation pages, headers, and copy in general represent critical elements that are easy to tweak and that impact conversions directly.

Global mission critical elements like a site-wide contact form or call-to-action button are also a great place to focus your initial testing efforts. Such elements get a lot of traffic because they are featured across the website – at the same time they represent crucial steps in the conversion funnel.

Begin with the end in mind

A lot of beginners make the mistake of just testing random elements and variations without having a clear idea about why they’re doing it, and what they want to achieve. If you want to learn from your tests, you need a clear test hypothesis that defines why you are making the change what you hope to achieve by making that change.

If you don’t have a clear hypothesis, it’s likely that you’ll end being more confused than you were before you began testing – and that pretty much defeats the purpose of testing in the first place.

Useful links:

I wrote an extensive guest post for Unbounce.com on how to find out what to test on your landing pages and how to establish a real test hypothesis. Check out the article here >>

I also wrote a guest post for KISSmetrics highlighting 3 obvious but overlooked elements to test on your landing pages. Read the article here >>

In another guest post for KISSmetrics, I presented 8 questions that lead to perfect landing page copy. Reading that article should give you a clear idea of how to approach copy optimization. Check out the article here >>

You can learn how to write call-to-action copy that converts by watching this short how-to video on CTA copy >>

Here are a few case studies I’ve posted on ContentVerve.com. They are all pretty simple and should give you inspiration for a number of experiments that you can conduct on your own website:

How Changing 1 Word in the Call-to-action Generated a 38.26% Lift in Conversions

99.4% Lift in Conversions by Tweaking 4 Basic Elements on a B2C Landing Page

31.03% Increase in Sales by Tweaking the Call-to-action Copy on a Payment Page

4. How do we set up an A/B split test?

As I mentioned earlier in this article, if you choose the right testing tool setting up a split test is pretty easy – even if you’re not a code nerd.

I use Visual Website Optimizer and would recommend it to anyone interested in getting into CRO and testing. VWO let’s you do anything from super simple copy and layout tests to multivariate tests and advanced site wide experiments that involve code alterations.

But for the moment, let’s focus on simple tests. The beautiful thing about tools like VWO is that you can easily setup test variants directly via the dashboard without making any permanent changes to the website itself.

Here’s an example from ContentVerve.com where I changed the title and post image, and even rearranged the position of the boxes in the right column by using the built in features of VWO.

I made this step-by-step video tutorial to show how easy it is to set up a simple A/B split test with VWO.

Another tool I’m a big fan of is Unbounce.com. Unbounce makes it incredibly easy to create landing pages from scratch, and because the system features built-in A/B testing software, setting up tests is also extremely easy. So if you decide to start by experimenting with your landing pages, Unbounce.com is an excellent choice. Try it free for 30 days >>

5. When should we stop the test?

The only thing that’s worse than not testing is to rely on bad data. The whole point of performing an A/B test is to get answers, so you can base your decisions on data instead of guesswork and gut feeling. And if you can’t rely on your data, there’s really no point in performing the split test in the first place…

So, the simple answer to this question is “You should stop the test, when your data is reliable.” That however raises another question “How do we know when our data is reliable?” Well, that’s where test validation and statistics come into the picture.

Although these subjects aren’t particularly sexy, they are ridiculously important if you want to run valid experiments that provide true and lasting value to your online business.

The 3 most important factors, when it comes to determining the validity of an A/B split test, are: Statistical Confidence, Conversion Range, and Sample Size.  In this 10-minute video, I’ll go over these 3 factors and give you a basic introduction to finding out how reliable your test data is:

How much of a risk are you willing to take?

There’s no law against running inconclusive tests or stopping a test before you reach statistical significance. But the less you focus on test validation, the greater the risk will be that your data is off, and – as a consequence – that you’re basing your decisions on random observations not solid statistics that give you a reliable picture of how your treatments will perform in the long term.

If you stop a test at e.g. 85% statistical confidence with a standard error of 8% and a sample size of 45 visitors – you must be willing to accept that there is a high risk that your numbers are off and that the test results will look completely different if you let the test run until you reached 99% statistical confidence with a standard error of 1%, and a sample size of 5.000 visitors.

So, for your own sake, I strongly recommend that you run experiments long enough that you have sufficient data to reach statistically significant conclusion.

Useful links: 

Here’s a guest post on sample size that Siddharth Deswal from Wingify wrote for me. Check out the article here >>

Back to you!

Do you have more questions about A/B testing? Leave a comment and let me know what you’re struggling with or wondering about, and I’ll in all likelihood write an article about it here in 2013.


  1. Hi Michael,

    Thanks for another good post.

    Regarding the last question on when to stop a test. When I execute splittests, I sometimes get 95% statistical significant data after only 12 hours, because of a high number of visitors and conversions. This can give me a result with i.e. 30% increase in conversions.

    I then decide to keep the test running because a test run from i.e. 8am – 8pm on a monday mid-month, might be to narrow a window to conclude on. After the tests has run for i.e. 5 days, the result can be very different, and not statistically significant.

    Therefore….If I just stop when statistical significance of 95(or even 100)% has been reached – I will look good! But because I have a thesis that the profiles of customer on a monday mid-month might not be the same as a monday after pay-day, I keep the test running – and look less good!! :)

    Therefore – to conclude when your test should end, based on statistical significance only, can be a bit narrow?

    Let me hear your thoughts on this.


    Casper Remmer

    • Michael Aagaard says:

      Hi Casper – Thanks for your comment.

      Great question!

      There’s a difference between confidence level and statistical significance (you seem to be confusing the two). A test can reach a 95% confidence level, without being statistically significant because there are several more factors that go into finding out whether you’ve reached statistical significance (test validation).

      Sample size is very important. When your test hits a 95% confidence level after 12 hours it usually has to do with the fact the sample is still relatively small. Therefore, small fluctuations can have a large impact on data. As the sample grows you’re data will stabilize and small fluctuations will not have as great an impact and your confidence level will go down. Remember also to be aware of conversion rate range and standard error. Those two factors are very important in test validation.

      If you watch the video I made on statistical significance, you’ll get an introduction to all the above mentioned factors. One of the main points I wanted to make with that video it is never ever enough to look exclusively at confidence level. You have to look at other factors also e.g. sample size, number of conversions, conversion rate range, and standard error. Also, it’s always a good idea to keep your test running for 7 days.

      - Michael

  2. Testing *should* be the basis of which we make most our decisions regaring our online activities, but sadly, it is often not. Thank you for this guide – and keep up the good work explaining both the how’s and why’s of A/B testing.

  3. I would like to hire you to re-write the areas you see as having the highest impact on my site. I already have Visual Website Optimizer and can test everything on my site. Can you help me?

    Let me know…..thanks.

  4. We are a group of volunteers and starting a new
    scheme in our community. Your web site provided
    us with valuable information to work on. You’ve done an impressive job and our whole community will be grateful to you.

Speak Your Mind