Facing facts: How advertisers are reading our emotions

Marketers are increasingly tapping into automated methods, such as facial coding, in an bid to measure emotional reaction to content. Katie McQuater speaks to some of the people at the forefront of this technological charge to find out more.

Reading people’s emotions was once a matter of intuition; some elusive quality that can’t be programmed or processed by advertisers. To gain insights into people’s reactions, you had to ask them how they felt, often via clunky market research methods with questionable results. Today, however, brands are increasingly taking control over the previously intangible, and understanding what was once unfathomable.

Facial coding, the ability to track and measure emotional reaction based on facial expressions, is becoming an intrinsic element of big brand marketing strategy. With a need to analyse every penny of marketing investment, brands are deploying automated methods of tracking engagement to predict the effectiveness of a piece of content, often before it’s even launched.

“One of the issues that research can often struggle with is getting to the heart of what people truly think or feel about a topic,” says Julia Ayling, head of intelligence at Mindshare UK.

“So much of the time, if we ask a respondent a question, the answer we get back will be biased by many different factors – from the language we have used within the question itself, to the approach we have used to ask it, to how the respondent has chosen to remember their behaviour, and recall it for us.

“Newer research approaches such as facial coding, eye tracking or emotion tracking can help to overcome these limitations, as they can reduce that ‘lost in translation’ dimension that claimed research techniques can sometimes suffer from.”

Realeyes, a technology company operating in emotion tracking, develops algorithms to understand people’s reactions via their webcam. Requiring a one-click opt-in from the consumer, the technology gathers feedback on content via their webcam, measuring six ‘basic emotions’ as well as attention and heart rate.

Working with media agencies and brands including P&G and Phillips, the common theme across all the company’s clients, according to chief executive Mihkel Jäätma, is that following decades of building understanding of what works on TV, they are now trying to establish how best to approach online video content.

“Big brands are not necessarily geared up to take creative risks or do bold things, so we can show them that people have responded well, or that according to our algorithms this will do well and raise your brand metrics,” says Jäätma.

According to Omar Bakhshi, head of user experience at OgilvyOne UK, which uses facial coding in its Customer Lab to better understand how consumers react to a brand’s digital presence, the main benefits for clients is understanding the “actual underlying visceral, irrational and emotional responses their customers are experiencing”.

Far superior to traditional market research methods when it comes to measuring non-conscious response, facial coding is a step ahead because people aren’t necessarily conscious of their emotional response to a piece of content.

The well-versed ‘Mad Men vs Math Men’ rhetoric in the industry has given rise to much contemplation on the ideal formula for advertising creativity, with the most recent example being M&C Saatchi’s proclamation that it had discovered the ‘magic formula’ for advertising effectiveness. So has facial coding unearthed a common pattern for creativity that wins with consumers and at Cannes?

In Jäätma’s view, successful creativity uses no such hard and fast rule. “As soon as you have a formula or a favoured approach it becomes mainstream and loses its appeal and the special power that it has. So in terms of how or what makes people surprised, laugh or cry, that’s an ever moving target. That’s why the creative function is so important – I think it’s very undervalued right now.”

He does concede, however, that consistent success patterns share certain characteristics – high happiness peaks and ending on a good note, for example. So far, so obvious – but what advertisers may not realise is that eliciting emotion of any kind is better than none at all in terms of moving campaign metrics, with the ‘Devil Baby’ stunt video one such example.

Graham Page, executive vice-president and head of global research at Millward Brown, is also quick to maintain that there is no such magic, simple formula for generating the best emotional response from an ad. However, ads that tell a story are naturally more effective when it comes to evoking feelings, evidenced perhaps by the continual success of John Lewis’ annual Christmas tearjerker.

“Ads with a narrative arc, rather than a series of vignettes or product information, are much more effective at evoking feelings. This shouldn’t be surprising, as we naturally engage with human stories,” says Page.

While facial expressions are “excellent ciphers” for immediate responses, they are really only the foundation of building an understanding of an ad’s emotional resonance. They don’t reveal how people construct an advertising story in their head, or how their thought processes and reactions to the ad might influence outcomes, says Page. Millward Brown uses the technology with its partner Affectiva on all its quantitative ad research for a client base of blue chip companies, but Page says it complements, rather than replaces, existing market research methodologies. “Viewers both react and think when they engage with advertising, so we need to ensure we measure both of these processes. Using both sorts of approach together in ad research yields a more balanced and realistic view of campaign effectiveness.”

Though brands and agencies are becoming increasingly savvy in their methods of establishing what content resonates with consumers, accurately measuring emotional response is not yet fully understood within all corners of the industry, particularly when it comes to integrating several methods of analysis.

“There is a general lack of expertise in the industry for integrating the various types of measurements to provide a holistic analysis,” says Elissa Moses, executive vice president of neuro and behavioural science at Ipsos, adding that capturing the universal emotions as mastered by facial coding is “like identifying individual musical notes” – just the foundation of the process. The approach at Ipsos, she explains, is to weave together findings from neuro or emotional metrics with cognitive and survey levels to build layers of insight on how an ad performs. But brands often fail to conduct research in the right way, she says.

“I see a lot of mistakes on sample sizes, leaps of faith on interpretation and a lack of understanding of the need for measures covering cognitive response with the non-conscious reactions for emotional response, engagement patterns and brand impact. Only in this way can we be comprehensive in our understanding of how an ad works.”

Disconnect between creative and media agencies is another barrier to success, according to Jäätma. Gaining a comprehensive understanding of how an ad works is only possible if all stakeholders – client, agency and research partners – have a clear, shared end goal, though this is improving as technology develops, he adds.

“The disconnect between the creative and the media people is the single biggest source of inefficiency in the whole industry. The fragmentation of the industry is why we have bad ads. But I think the technology is changing that – what we think and do are largely driven by emotions and now that data can be available on a large scale, without friction, and shared across different functions.”

Reading people’s emotions may be changing the way ads are made, but at the end of the day, it’s the people making them, and the magic they can bring to the process, that will determine an ad’s effectiveness. Sentimental? Maybe. But nobody said emotions were rational.

This feature was first published in The Drum's 8 July issue.

Katie McQuater

As magazine editor at The Drum, I edit the monthly print edition of the magazine as well as commissioning and writing features for the publication.

Send feature pitches to katie.mcquater@thedrum.com

All by Katie