The Swrve guide to running A/B Tests that matter
There are few marketing research techniques as flexible as the classic A/B test. Using this method, giving participants two alternatives to select a favourite from, you can test just about anything if you really want to. However, the truth is that – perhaps because of the flexibility inherent in the technique - an awful lot of A/B tests deliver inconclusive results or, worse still, results that have little bearing on what really matters: making improvements in customer interaction that boost the bottom line of your business.
Are you getting the most from your A/B testing?
Let’s consider one common example. Lots of people think that testing the subject line of a marketing email is a valid use of A/B testing and it is, but it’s not something that we’ll learn very much by doing. Obviously, we might improve the performance of a specific campaign by sending the winning variant to the remainder of our audience (although in most cases the numbers used to make these decisions are nowhere close to large enough), but beyond that we aren’t much wiser about WHY one alternative works better than the other.
Enlightened businesses (and I am going to include Swrve customers in that group!) realise that the A/B test can be employed far more widely than simply improving the prospects of a single campaign. On the contrary, it can enable organisations to implement informed changes to the mobile app experience on an ongoing basis. Changes that will engage users, retain users, and, ultimately, drive revenues - permanently.
So, what elements of your mobile app should you test in order to deliver maximum ROI to your business? Our customer success team has worked with many of the world’s leading app businesses, so we know what works. I’ve included a few examples below, but anyone interested in learning more on the subject can download Swrve’s free whitepaper on the subject.
Your onboarding process
How many initial screens are most effective?
Within these screens - what content are users most responsive to?
Getting ‘free trial’ users to subscribe
Which user groups should be targeted?
What is the optimal time to deliver campaigns?
Which campaign creative is most effective?
Push notification opt-in
Should you request this access during onboarding, or wait until the user is familiar with the app?
Should you provide the user with details of what kind of push they might receive?
Measurement is the key to providing your customer with a first class mobile experience. Only your customers can tell you what they want - so listen, learn and keep improving. That, after all, is what A/B testing is all about.
Alex Gray is director of sales engineering at Swrve.