A/B testing is a statistical way of testing two or more versions, such as version A and version B, to determine which version performs better and also to understand if the difference between two versions is statistically significant. This is how, companies could adopt a data-driven approach rather than take decisions with instinct, consciously or subconsciously. Because the customers or clients will not behave as the company thinks most often.
A/B testing can be adopted to optimize various applications such as;
- Marketing campaigns
- Web page design according to different parameters such as;
- Conversion rate
- Bounce rate
- Number of visits
- Mobile or computer application testing
How to conduct A/B testing
- Pick one variable to test.
You may need to test many aspects, in other words, variables, in your web page, marketing campaign, or application, for example, the colour scheme of your web page or the age group that you focus on in your marketing campaign.
One piece of advice – select only one variable for each test. Otherwise, it is hard to determine which variable was responsible for changes in performance.
- Identify your goal.
You may state a hypothesis for your test. Examine the results based on that, then you can determine whether the hypothesis is correct, for example, the goal of your test would be to determine the number of people who signed up for a newsletter increase, when you name the button on your web page “Sign-up” rather than “RSVP”
- Create a ‘control’ and a ‘challenger.’
Make a version with the default variables as your ‘control’ test object and the other version in which you need to change a variable to the ‘challenger’. For each test, you can change the variables in the challenger but not the control. That way it is easy to understand which variables caused the changes in the performance.
For example on a web page, you can keep the control version without changing, but in challenger first change the sign-up button and run the test, then you may change the colour scheme of the page and run the second test and then change the format of the page and run the test. This way you can decide which versions are effective based on your goal and finally make the necessary changes.
On the other hand, if you change two or more variables in the challenger version, it is called a Multivalent test which a different one from A/B testing.
- Split your sample groups equally and randomly.
You should be able to divide your group of customers, clients, or viewers randomly but in equal numbers, the characteristics of the two sampled groups should be the same so that you can get unbiased results from your test. Characteristics of the sample groups may differ on some special occasions, in cases like you have to check which age group responds to a product more.
There are different tools available for this task to sample your test groups in Google Analytics Facebook ads, etc…
- Determine your sample size (if applicable).
Obviously, to see better results, it is better to test on an adequate sample. Otherwise, the test results can be less accurate.
If your test doesn’t have a finite audience such as your web page then assign an equal time period instead.
- Decide how significant your results need to be.
This is also important because there should be a significant difference between the statistical results in two versions of the test. This is because it helps to determine whether the change of that particular variable is effective. There is no point of making less effective changes to the product after all.
For example, there is no point in changing the text of the original or control webpage to that of the challenger version, unless otherwise there is no more than 10% statistical improvement of conversions in the challenger version.
- Use an A/B testing tool.
To perform A/B testing on your web page or email marketing campaign, you might need to use some tools which facilitate sampling of groups, recording results, calculating and presenting data etc…
There are many commercial tools available for this purpose. Mainly, you could use Google Optimize, or Optimizely. Facebook Adds in the A/B testing mode other than that if you are a HubSpot Enterprise customer, the HubSpot software has features that let you A/B test emails, calls-to-action, and landing pages.
- Test both variations simultaneously.
Time plays a significant role in A/B testing. It is important to test it for an adequate time period. The test for both versions can be executed at the same time, in fact, it is necessary to do so if the test is time sensitive.
- Ask for feedback from real users.
A/B tests mostly provide statistical or quantitative results. But if possible, it is also better to gather qualitative data and use that for the final evaluation of the results. Qualitative data can be gathered in the form of surveys and feedback from real users.
- Evaluata and plan your next A/B test.
Evaluate the test. For that, some statistics knowledge is necessary. Link
Once you are done with the first test, it will provide you insights to achieve your goal as you set on the 2nd step. But do not stop there, since you can change other variables in a sensible manner and run the test again and again for each of those instances. These results will allow you to create a final model with better optimization.
Once you are done with the above steps, check whether you have reached your goal for each of the versions. Then measure the significance of your test results manually according to your own criteria or by using A/B test results calculator such as a one directed from this link.
Take actions based on your results. If one version is statistically better than the other, you have a winner. Then you can just select the winner version. But if there is no significant difference of the results, or you can’t make conclusions with the results, then it is better to stuck with the original version or in other words the control version. Later, you can conduct tests with different variables until you get satisfactory results which will optimize your goals.