Types of tests

The main types of tests in marketing are A/B tests and multivariate tests. A/B tests compare the effects of changing one element. In an email, this element could be a subject line, piece of content, creative, messaging, call-to-action, etc. Since you’re only comparing one variable in A/B tests, you don’t need as many data points for it to be statistically significant. Thus, the sample size doesn’t necessarily need to be very large. Multivariate tests compare multiple elements at once, which can become complicated quickly.

Since you’re changing more than one variable, it’s hard to understand what to attribute your test results to. According to Candice Lunn, our Client Strategy Manager, “Multivariate tests can be useful, but they can also be more extensive from a timing and budget perspective. It’s important to weigh your options.” A useful solution is to A/B test incrementally and optimize the tests as you go.

Strategic testing considerations

In addition to how many variables you should test, there are other important aspects to consider. To understand incremental lift, think about having a hold-out group, which is when a group of people are held out of marketing communications for a period. It doesn’t need to be a large group, but it can help determine overall lift from the campaign.

Hold-out groups can also create what we refer to as the halo effect. The halo effect exposes additional lift outside of the testing parameters that could be attributed to additional marketing exposure. Jen Streck, our VP of Client Strategy, explains her experience with the halo effect:

“The net incremental lift of the entire marketing program wasn’t just the individual campaign incrementals; it was individual campaign incrementals plus this little bonus halo effect of marketing exposure.”

It’s also important to consider the timing of testing. Best practice is to test during a slow period instead of a peak season, because it’s difficult to attribute user behavior to the changes you made when opens and clicks are already high. You could also risk making a change the consumer doesn’t like, and it could have large business implications. In some instances, it’s okay to test during peak season—just be cautious with the number of changes you make.

You don’t need to test your entire audience to have actionable results. We recommend holding 5-10% of your traffic for testing and keeping 90–95% business as usual, so it doesn’t impact overall business results. As Dr. Alvin Glay, our VP of Insights and Growth, states, “If you have at least 5-10% of your budget or audience dedicated to testing, then that 5% could lead you to exponential growth.” Since your audience is continuously evolving and their tastes and preferences change, we recommend testing more than once over time.

Conclusion

Although testing is important, many find it difficult to set up the appropriate testing strategy and structure and cumbersome to interpret the results. According to Dr. Glay, “Basically, if you’re not constantly iterating on user experience, consumer journey, and audience strategy, then you’re not growing.” Don’t become stagnant—if you need help with your testing strategy, reach out to us today.