Whether you’re considering or already using email for your brand, acquisition or retention efforts, email testing is vital to any marketing or communications strategy and will pay dividends in additional customer engagement and performance – no matter what your objectives are.
Email testing can be executed in a few different ways. We’ll look at A/B and Multivariate testing in this article plus provide you with some juicy ideas to get you started – it’s never too late to start testing, no matter how small or large the idea!
What is A/B testing?
A/B testing (also called split testing) is where you choose two elements that you test head to head, with half your email volume going into each. When you’re analyzing the results it’s important to ensure that you have enough data to make a robust conclusion; the number of recipients you’re emailing will determine what results you can consider robust – the more the merrier!
You will also want to make sure that the metrics you are comparing are significantly different, for example if version A had an Open Rate of 25% and version B an Open Rate of 26%, it probably wouldn’t be wise to consider a 1% difference significant enough to rule one version the winner, so you might want to test these again (unless of course you decide just to choose the version you like best).
The Survey Star website has a nifty calculator to measure significance; simply enter the number of customers emailed test A and test B, and the performance results that you’re comparing (open rate, click rate, click to open rate, conversion rate etc.) and it’ll confirm whether the differences in performance are statistically significant.
What is Multivariate testing?
Multivariate testing is where you have multiple variations being tested in different combinations at the same time, this can be a simple email test with as few as 8 variations (plus your control, if you have one), or as complex as 128+ variations. Be aware though that the more variations you have, the bigger your email list will need to be to accurately compare performance since you’ll be splitting your list into much smaller audiences.
Now, imagining you managed marketing at a pet supplies company and you wanted to test different email subject lines: one with a teaser (cryptic message to create intrigue) and one with the customer’s first name; and you also wanted to test which animal picture (in the email content) customers responded best to. Using multivariate testing you could mix and match these variations across 8 email combinations, as illustrated in the following table.
To calculate which of the two subject lines performed best, you can sum up all the variations which had the teaser, and separately all the variations which had the first name, and then compare the open rates. In this example, the teaser performed 6% better than using the customers first name.
When you measure the results of any email campaign, your open rates will indicate how your subject lines are performing, and your click rates will show you how the content in your email (headline, body copy, CTAs, images etc.) are performing.
Now, let’s talk about the fun stuff – this is where you can be super creative and implement endless testing of your emails to keep your communications fresh and driving ever improving results.
The following email testing ideas should help to get your creative juices flowing as you plan for future testing:
When a customer receives your email, the subject line is one of the first things they see in their Inbox; in those first crucial fractions of a second, they use it to decide whether or not to open your email. There are a multitude of subject line tests such as short and long subject lines, tone (humorous, informal, formal etc.), you can pose a question, you can tease to create mystery or suspense, first letter or full capitalization, and personalization (such as including their first name or favorite product/movie/food etc.) depending on data you have available (without being too “creepy”!).
The pre-header is the short text snippet that displays on some email clients, such as Gmail and Yahoo, immediately following the subject line in the inbox. It’s important not to forget this as it will default to the first line of text in your email, which might not be ideal.
The below example illustrates how this displays in Gmail and how it can go horribly wrong if overlooked. This is vital inbox real estate and can play a large role in whether a customer decides to open your email – so don’t let it default to “View in browser” or “Is this email not displaying correctly?”.
How you decide to format your emails may depend on whether you have design resources readily available to you. Take a look in your inbox at emails you get from other businesses for inspiration or do a simple Google Images search. Depending on the characteristics of your customer base, you might like to test a plain text email that looks just like a personal one-to-one email without any imagery or fancy typography; or you might want to build something visually appealing with imagery, sections, animations and video.
Hierarchy testing can tell you a lot about your customers and what they find important. For example, if you have multiple benefits for the product/service you’re promoting, you might choose to change/rotate the order that you communicate these benefits to determine whether customers respond better to some benefits more than others.
Whatever formatting you decide, don’t forget to add that very important unsubscribe link!
If you have varied data points on your customers, it can be beneficial to test varying copy to different customer types based on their demographics, past purchase or browsing behavior, location etc. You’ll likely find different ways each customer segment prefers to be spoken to by comparing the opens, clicks and conversions you receive from these copy methods.
There are endless ways to test email copy including, but not limited to:
- copy length (short, medium, long)
- message type (soft sell, hard sell, product based, benefits based etc.)
- salutation (Hi, Hi Tom, Hi there, Hello, Dear Mr. Jones etc.)
- sign-off (the friendly team at SMTP2GO, Until next time, Speak to you soon, etc.)
- tone (formal, conversational, humorous etc.)
Emojis are a fun way of injecting some personality into your emails to better engage with your audience. That said, whether or not you use Emojis and which Emojis you decide to use, will entirely depend on your audience and the overall tone of your emails 🤔. You can test Emojis vs. No Emojis, different types of Emojis, placement of the Emojis 🕹 (in the Subject Line, body copy or Call-to-Action for example) – the possibilities are almost endless! 🐙
Just like a plain text email can perform better than a well-designed HTML email, a logo can change how customers respond to your email. Therefore, it’s a good idea to test logo vs no logo, size of logo and placement of logo to determine the best application for your customers and objectives.
Supplementary content can be, well, just about anything you think is relevant or which your customers find interesting. This can be social buttons or excerpts, customer reviews, videos, blogs or support articles. A word of caution though, use of such content should be determined by the primary objective of your email. For example, if the primary objective is to get your customers to click the CTA button and purchase a product, then supplementary content may distract and decrease overall conversions. That said, content can be a powerful way to drive awareness, inform and engage your audience; by improving how your audience perceives you, you can increase the likelihood of future purchase and referral.
Speaking of your primary objective, a well-placed CTA can do wonders for the performance of your emails. The key here is testing where these buttons/links are placed; how they’re worded and formatted can make a big difference in performance. For example, I once tested adding a greater-than sign “>” at the end of the CTA copy “Take a Look >” and the conversion rate increased 11% when everything else in the email remained unchanged.
Your CTA can be a button or a hyperlink; it can be placed once, twice or three times throughout an email; it can be red, green or blue (or whatever color works for you!). The key is to test CTA variations regularly to ensure that your emails are fresh, relevant, eye-catching and delivers response.
They say a picture is worth a thousand words. In marketing and communications, an image can help illustrate, explain and convey an emotion or an action, among other benefits. When chosen carefully, in consideration of the audience and message that’s trying to be communicated, images can enhance how recipients engage with your emails. Knowing which images your audience will respond to is tough, and that’s why testing is a powerful tool to better understand your audience and their preferences.
Colors are powerful conjurors of emotion. They can subconsciously deliver a sense of calm, urgency, happiness, concern, optimism and delight to your audience. Couple color with relevant messaging to create impact in your customer’s inbox.
Time of day, day of week & follow-up ⏰
Testing when you deliver your emails will undoubtably depend on the timeliness of your email; a supermarket wanting customers to visit on their way home from work might find better results if they send the email at 4pm in the afternoon, whereas a bank wanting customers to provide feedback might find customers more responsive and willing to volunteer 10 mins of their time at 10am on a Saturday morning. Considering your audience and testing the time and day of the week that you send your emails is paramount to capturing customers attention – or risk being forgotten, unopened or deleted.
Didn’t get noticed the first time? How about sending a second follow-up email? Do pay close attention to your unsubscribe rate as this’ll tell you whether your customers find your emails useful or just plain annoying.
The destination where your CTA lands customers is a crucial stage in the conversion process. Depending on your message and objective, a highly relevant landing page may translate to better performance.
Imagine you receive an email from a department store promoting a sale on t-shirts. You click on the email and land on their homepage – now you have to spend crucial seconds navigating to the t-shirts section. If you weren’t in dire need of a new t-shirt in the first place, it’s likely that you’ll lose interest after landing on that homepage since you can’t really be bothered hunting around – and you really should stop procrastinating and write that report… Now imagine you clicked through that email and landed on a page displaying the retailers great collection of t-shirts, one catches your eye – now you’re interested, right?
With endless ways that you can experiment with your emails and the customer journey beyond the email, even the sky is no limit for how you can start implementing small email tests each and every time you communicate with your customers.
Good luck out there! 🍀