Email A/B Testing: How and When To Do It

In this article, you can learn everything you need to know about email A/B testing. What it is and how and when to use it to find the best performing email marketing campaigns and improve your KPIs and overall marketing results.

What is email A/B testing?

Email A/B testing (also known as split testing) is the act of comparing two or more variables to analyse which performs better. A/B testing is one of the most widely available features offered by ESPs (email service providers). In email marketing, A/B testing is important not only for ensuring that campaigns are running at their greatest potential but also for gaining a deeper understanding of your customers.

A/B testing is a powerful tool that can generate excellent results for your business. For example, eCommerce brands saw a 20% increase in revenue when using email A/B testing.

How to use email A/B testing?

The key to carrying out successful A/B testing is planning. A/B testing in email marketing needs to be carried out with a clear objective and a robust understanding in order for it to be successful. Without these, your results can be unreliable and negatively impact not only your engagement rates but your subscribers and customers too. There’s a difference between A/B testing and Email testing. Learn more about email testing.

The main principles to follow when planning A/B testing in marketing are the following:

1. Idea

A/B testing begins with a good idea – that is a solution to a problem that needs to be solved. Your idea is what your A/B test is based on and, therefore, should be clearly defined from the outset. Although it is just an educated guess as to how to solve a problem, it should be clear and focused from the beginning, with some evidence to support its theory. Begin your idea process by asking the question “if?” followed by “then?” e.g. When testing subject lines: if we begin using emojis in our subject lines then we will see an increase in open rates’.

2. Objective

The next step of planning your A/B test is the objective – you must be clear and concise on what you are aiming to achieve from your email campaign.  e.g. ‘our objective is to increase open rates across our marketing emails to achieve higher conversion rates.’.

3. Determine the audience.

The selected audience for A/B testing is the most important factor. If your selected group for testing is too small, it may not hold any statistical significance, which is why it is important to determine the number of people you will need in your sample audience so it will have statistical value.

Circulator makes this easy for our customers removing any need for third-party apps to calculate audience sample size, as you can select the percentage of the total group you would like to sample. The application also selects these sample subscribers by random removing any bias from the data and removing any random chance impacting the overall results. Once this number is determined, the sample audience will be split 50/50 into ‘Group A’ and ‘Group B’ for testing. The variants should not be changed for ‘Group A’, this will be your control group, that is the standard experience your users have and the baseline for your engagement. ‘Group B’ will be your variant group where the variants are changed, and you can then measure the impact of these changes on your objective.

4. Measure

The third item to consider is how you will measure the success of your objective. Ask yourself the question, ‘what metrics will I need to measure the success of my campaign?’. Once this is identified, you will have the ability to monitor your campaign more accurately and attribute activities back to your cohort, as primary and/or secondary metrics will be clearly defined from the outset.

You can evaluate the versions based on open rates, click rates, conversion rates or generated revenue to build the most effective campaign.

5. Considerations

The final step in planning for your A/B test is to have realistic expectations and be prepared for a variety of outcomes – as well as having thresholds for both failure and success.

See below how can you easily set up A/B testing with Circulator:

When to use A/B testing?

Split testing can be used for a new or existing campaign before or during its launch. It can help marketers to see how certain variant changes can impact engagement and give them a deeper understanding of their subscriber’s interactions. This in turn allows them to ensure that their campaigns are running at optimal levels.

How can be A/B testing affected by Apple Mail Privacy Policy?

When evaluating the results of A/B testing, you can track open rates, click rates, forwards, conversions and other important metrics. However, assessing A/B testing might become more difficult with the new Apple Mail Privacy policy.

The Apple Mail Privacy Protection intends to protect users and prevent senders from knowing important information about subscribers and their activity by using invisible pixels that collect data about the user. Apple Mail Privacy will cause a discrepancy in open rates on Apple devices. All emails received by Apple Mail users will be automatically tagged as opened. Therefore, you will not be able to tell which Apple Mail users really opened your email, and your A/B testing results will not be accurate. We would suggest evaluating your A/B testing based on other metrics, such as click rates and interaction.

Summed Up

Email A/B testing is a useful tool when it comes to email marketing and can provide a number of insights to the marketer as well as help to ensure campaigns are running at their highest standard. A better understanding of the planning process before A/B testing will only allow for better learning and rewards for future campaigns.

Please contact us directly for guidance and assistance with A/B testing and email marketing.

Visit our blog for more email marketing tips.