A guide to smarter creative testing: from A/B to dynamic creative optimization

Over decades, my colleagues and I have learned the best practices of testing. Here's how to improve your testing processes — and hopefully, your bottom line.

1. A/B is the OG

Also known as split testing, classic A/B testing is a scientific method that can provide clear, reliable results. It’s when you test one thing (A) against just one other thing (B), while keeping everything else the same. Why? It removes guesswork and lets you modify your creative with relatively low risk.

Follow these steps for a smooth A/B test:

1. Set your goal: start with a distinct objective (grounded in research) of what you want to learn or accomplish.

2. Design the test: pick one element to focus on, such as a headline or hero image, creating two versions where a variation to that single element is the only difference between them.

3. Split the audience: randomly divide into two equivalent groups, large enough to trust that results will be statistically significant.

4. Run the test: put the creative into the market and allow the campaign to run an appropriate length of time.

5. Analyze results: collect and compare responses from the two groups to see which test performed better.

Don’t look at too many metrics at one time, which could lead to deceptive correlations. And ensure that your test results are statistically significant before drawing conclusions; resist the temptation to generalize across disparate channels.

It’s smart to review failed test variations for insights to drive future efforts. You can conduct retests to rule out false positive results and possibly confounding factors like seasonality. Perhaps most importantly, you should build sequential tests with different test elements to accumulate more knowledge.

2. Multivariate testing

When your audience is large enough, and your timeframe long enough, but A/B testing is not enough to meet your goals, you can branch out into multivariate testing. This tests more than one element at a time. Couldn’t that get chaotic? Yup, which is why you need to approach it in a very disciplined way.

The sequence of steps is the same as A/B testing, with two important distinctions.

The first is around test design: instead of limiting your scope, create variations for a bunch of elements to test and run simultaneously. For example, on a single landing page you might test two different headlines, three images, and four calls to action (CTAs).

Secondly, focus on audience splitting. Your universe needs to be large enough to divide into multiple groups — as many as your total variations. Using the example above, you’d need sufficient traffic to get served a total of 24 unique pages (2 x 3 x 4 variations).

Multivariate testing works best when you craft meaningful variations. Ensure that you have a strong rationale behind each element and variation you test — giving good reason to believe its difference can impact engagement, clicks, sales, or whatever you’re measuring.

3. Level up with DCO

Dynamic creative optimization (DCO) is an exciting, efficient way to test lots of assumptions about what works and what doesn’t in digital media. It leverages a flexible template, uses variable creative elements to personalize your message and achieve scale, and optimizes content throughout the testing period until the most successful iterations are the ones left standing.

Make a crystal-clear plan

It helps to begin with a small-to-medium campaign targeted at a specific audience before going too grand too soon. Meanwhile, anticipate what success looks like and envision how you’ll apply learnings to future phases.

Consider your media budget and timeframe, as well as which channel(s) to launch first. Identify the data triggers that will inform your test, such as location, moment, seasonality, and customer experience (CX).

Align on which creative elements to test before designing your template (more on that in a moment) and make sure all your various elements can play nicely with each other as they populate that template.

Balance tried-and-true creative variables with ones that break new ground

With its ability to test multiple things within a single template, DCO makes rapid creative iteration possible. Elements you can vary and test in the same campaign include:

• Which product or service to feature

• Images and/or illustrations, such as lifestyle or product-focused

• Animation to add visual interest and increase engagement

• Messaging in the form of headlines, subheads, and body copy

• CTAs: what they say and how they appear

• Background colors, images, and/or graphics

• Brand expressions, including logos and taglines

The beauty of DCO is how it optimizes the best-performing elements within each campaign: under-performers drop from the rotation and the winners rise to the top. Ultimately, the most effective combinations are seen by the audience majority, giving you rapid learnings as well as cost-efficient results.

Enjoy the thrill

For most of us creatives, there’s nothing better than finding fresh ways to connect with an audience and motivate people to respond. Granted, it’s hard not to fall in love with some of your favorite ideas before they’re tested — only to find they fell flat when the results get tallied. But there’s also the satisfaction of discovering some of your hypotheses were sound, and some aspect of creative that you advocated for really works well.

And if what you test and learn saves you or your client time and money, as well as meeting other testing goals (score!).