ab testing rules

7 Important Rules to Follow For Successful A/B Testing

ab testing rulesNote: The following is a guest post from Masroor Ahmed.

AB testing is here, and it is here to stay. While it is difficult to find people who will disagree with this statement, this was not the case about 10 years ago.

At that time, we were just starting our conversion optimizations practice and many business owners, marketing professionals and top management did not know what the terms conversion optimization or AB testing meant. Recent data shows that 44% of companies websites are conducting some form of AB testing.

The challenge nowadays however is in the fact that most companies that try conversion optimizations or AB testing complain that it does not produce the results it promises to deliver. This is a natural result for two reasons: 1. overselling AB testing as the sliver bullet that will save the day, and 2. the lack of understanding of how to plan, conduct and analyze AB testing results.

The following are 7 AB testing rules you must keep in mind if you are looking to increase your website conversion rate and revenue through AB testing.

1.  It’s All About the Hypothesis

Too many e-commerce websites jump into creating and conducting AB tests without knowing the exact hypothesis they are testing.

What is a “hypothesis”? It is a predictive statement about a possible problem on the page you are testing and how removing that problem will increase conversion rates. A sample hypothesis from a test we conducted recently on a product page for a furniture e-tailer was:

Providing easier access to major elements such as price, add to cart, selection options, etc. will increase conversion rate by reducing visitor anxieties and confusion, and addressing the needs of the impulsive persona.

Notice how our hypothesis indicates the problems which visitors are facing on the retailers website: 1. visitors are having anxieties, confusion, and 2. an impulsive person who is ready to buy will struggle finding necessary information.

Creating a good hypothesis forces you to think more about your visitors, what they are struggling with and how to address their concerns. Without having a hypothesis, you are merely trying out different designs, and most likely wasting valuable time, money and effort.

2.  Know Your Goals

We generally recommend a focus on macro conversions and goals for the first few tests until there is a better understanding of customer behavior on the site. Later on, micro goals (for example, to lower bounce rates for a particular page), will be more meaningful because you’ve collected data to successfully achieve the big picture goal.

3.  AB vs Multivariate Testing

Multivariate testing allows you to test multiple variations of multiple elements within a single test. Multivariate testing is AB testing on steroids. If you are just starting to workout (i.e., split test), avoid the steroids (MVT testing!).

While MVT testing is powerful, there are too many variables that are dependent on one another and that come together to produce an increase in conversion rates.

If you are like the many companies that jump into MVT testing without careful analysis, you will have a tough time explaining why one test increased conversion rates while others failed. If you are not able to explain the rationale for a test, you are gambling with your website revenue.

4.  Do You Have Enough Conversions?

Yes, AB testing is important but you need to make sure that you have what it takes to conduct a successful test. If you are conducting an AB test, you must have a minimum of 200 conversions per month. (This is a back of the napkin calculation!) If you have 200 conversions, then you will be able to test one variation against your original website design.

If you have less than 200 conversions, then focus on driving more visitors to your website through SEO, PPC, email, etc. Only when you hit that threshold of 200 conversions, then you are ready to start some light AB testing.

You have to keep in mind that 200 conversions allow for doing one variation against a control. If you run more than one variations, your test will take a long time to conclude and might not produce any meaningful data.

ab test rule #5: run for the right length of time.5.  Length of Time

Letting your AB test run for too short of a time or too long of a time will most likely produce unreliable results. How is that? If your testing software concludes a test in few hours, you need to be aware of the fact that tested data might not large enough to cover all possible visitor segments and variations that come to you website.

On the other hand, if you run a test for more than a month, then you are allowing for external factors which you do not have control over to impact the testing. The ideal rule for AB test time length: run an AB test with a minimum of 5 days but not longer than 30 days.

6.  It’s All About Marketing Insights

The real power of an AB test is in the marketing insights that you can draw from its results. How can you take what you learned in one test conducted on one page and draw lessons from it to implement site wide? AB testing is even more powerful when you are able to come up with conclusions that impact your offline marketing campaigns.

To some, this might seems a bit radical but this is where you have to further test these insights and make sure that they are consistent with different marketing segments and across different channels. A failed test is never a test you should overlook!

The marketing insights you can conclude from a test that did not perform can sometimes be more valuable than even a positive test.

7.  Find a Faithful Partner (in Sickness and in Health)

It is rare to find an online company that decides to stop its SEO effort. Most business owners, CEO and executives understand that is critical to consistently repeat carefully planned marketing activities to achieve and maintain success. For some reason this straight logic is not always applied with conversion optimization or AB testing. Companies jump into testing with a lot of excitement but within a few months start losing interest in the process.

Success in AB testing means that you will do it for a long time. It is about developing a culture of testing in your company. When your project managers plan for AB testing as an integral part of any website development, you can be assured that you will see great results.

About the Author

masroorMasroor Ahmed is the conversion optimization analyst at Invesp, a leading provider of conversion optimization software and services. Check out Invesp blog for latest articles and tips on topics related to conversion rate and landing page optimization.



Hey, thanks for the info. Now what?

If you need any help with content creation, we have tons of free resources to get you over the hump. Please subscribe to this blog below to ensure that you never miss an article.

Have questions or comments? Please contact us by email or phone.

If you really enjoyed this post, please help us to spread the word by clicking one of the social media sharing buttons.

Thanks so much!


Just enter your email below