Neuroscience Tech Artificial Intelligence

Could emotion-sensing technology signal the death knell for tracking as we know it?

By Meg Carter, Features Writer

May 16, 2016 | 7 min read

How does this picture make you feel? Sad, angry, happy, confused? Technology is enabling marketers to paint a better picture of our emotions than we could paint ourselves, Meg Carter writes.

Emotion technology – a combination of psychology and data mining used to unmask often hidden emotions as people watch ads – is entering the mainstream. But the extent to which emotion-sensing technologies such as facial coding are set to rewrite the future of marketing is not clear-cut.

Proof of the rapid ascent of this fast-emerging research discipline lies in the string of high-profile announcements made in recent months. The year began with Apple’s acquisition of artificial intelligence (AI) startup Emotient which uses AI-based technology to assess emotions by reading facial expressions – a proposition primarily marketed to help brand owners assess viewer reactions to ads.

Soon after, digital marketing agency Jaywing announced a landmark strategic collaboration with Imperial College London’s Data Science Institute to run joint emotion research and analysis to understand and predict audience response.

Meanwhile, MediaCom announced plans to integrate the Realeyes emotion analytics platform into its central content hub. Realeyes uses second by second emotional analytics gathered from detailed analysis of consumers’ faces via webcam to give content an ‘EmotionAll’ score based on attraction, retention, engagement and impact.

“Over the next five to 10 years, better understanding of consumers’ emotional response will be as significant as the recent emergence of online analytics and programmatic,” Jaywing chief executive Martin Boddy predicts.

Opinion is divided however on the nature and the potential impact of this anticipated step change. For Eleanor Thornton-Firkin, head of content and creative development at Ipsos Mori, emotion technology may transform tracking.

“Ipsos Mori currently uses facial coding and EEG [electroencephalography] to understand emotional response to advertising – although facial coding is used more often as it is more scalable, and it can be easily incorporated into an online surveying allowing people to act naturally in their own homes without having to come into a lab-style environment,” she explains.

“We also use implicit reaction time (a measure of unconscious ad or brand impact by quantifying shifts in brand perception on an unconscious level which can be imbedded into online surveys) to understand the underlying impact of emotion.”

Currently, brand owners most interested in this approach are those eager to better understand video content, Thornton-Firkin says. The appeal is obvious as this is where most money is spent and where subtle changes can have significant impact. But as more and more of us wear biometric technology, this could become the new dominant technology if closely linked to what people are watching and when.

“At some stage someone will probably and robustly predict campaign success through emotional response that point may signal the death knell of tracking as we know it,” she adds.

Palle Finderup Diederichsen, head of MediaCom Beyond Advertising EMEA, goes further. Look beyond current uses – which are mainly focused on understanding emotional response to optimise content for different media or markets, and the potential to apply emotion technology to the whole campaign development process, including channel planning, makes it “an extremely powerful proposition”, he believes.

“The future potential to track a whole range of different responses beyond facial tracking – body language or unconscious reactions, for example – is very interesting,” Diederichsen says.

And the step beyond that? Linking emotional response with sales. “It is already proven that emotions correlate with earned media,” he adds. “That, as an indication of intention to buy, is strong, so emotion has a natural sales effect. It’s just a question of having the right technology to prove it.”

Others are less bullish. Following M&C Saatchi’s AI billboard experiment last year, the agency’s chief innovation officer Dave Cox is cautious.

The campaign, for an invented brand of coffee called Bahio, used facial coding and an algorithm to analyse the emotional responses of people passing a digital billboard in order to test and refine different creative components into a finished ad.

Though no-one’s privacy was compromised, feedback suggested that some found it intrusive.

Emotion technology: A quick guide

An array of technologies can now be used to generate emotional insight in a variety of different ways.

Techniques fall into one of two camps: biometrics (the automatic checking of measurable physical characteristics) and neuroscience (the study of brain function).


Galvanic skin response: Monitoring emotional response as demonstrated by changes in conductivity of the skin that can be brought about by emotional arousal

Heart rate monitoring: tracking changes in heart rate that can be caused by emotional excitement

Facial coding: Analysis of changes in facial muscles triggered by different emotional responses to a stimulus. A number of these analytical tools are based on a catalogue of 5,000 muscle movements compiled in the 1970s known as the Facial Action Coding System


FMRI: Functional neuroimaging procedure using MRI technology that measures brain activity by detecting changes in associated blood flow

EEG: Electroencephalography, measuring the electrical activity of the brain to gauge a positive or a negative response to a stimulus

SST: steady state topography, a refinement of EEG

Opinion is divided when it come to which, if any, single technique will dominate in years to come. Yet current consensus is that due its immediacy, cost efficiency and scale-ability, facial recognition tools will occupy the centre stage for now.

Looking ahead, however, tools and approaches to support the effective integration of a variety of different emotion-gauging techniques and metrics are expected to provide a focus for next stage development and investment.

“I like the idea that by researching emotional response using an algorithm to test an ad, no one can insert their own bias,” Cox observes.

“But I fear there might be a tendency to over-simplify. The idea that there are six universal facial expressions underpins many of the facial coding systems now being used, but is this a little blunt?”

Without doubt, more science is needed, says Heather Andrews, UK chief executive of neuroscience-based market researcher Neuro-Insight. “Emotional response is useful, but only insofar as it persuades a person to lodge something in their memory. Lodging or encoding a memory is what makes the difference between transient entertainment and something more meaningful. Only measure emotion, and all you are measuring is enjoyment.”

Emotion and memory – encoding, as that’s what prompts future action, rather than recall – are equally important and moving forward, what will become increasingly important is understanding the relationship between the two.

“As tools and applications evolve we will see brand owners drawing on a broader range of research approaches and metrics,” Andrews predicts. “Already we are seeing greater interest in media context – the differences in emotional response experienced when consuming content on a tablet, TV, laptop or mobile – and this will continue to grow.”

Jaywing’s Boddy agrees. More science is essential to distance the emerging research discipline from gimmicky and PR-generating initiatives, he says.

“Emotion tracking to gauge the intensity of response can be a powerful tool. But work in this field must go deeper before we can emotionally target – targeting creative content to elicit different, distinct reactions according to a particular brand’s specific objectives.”

Asking how important measuring emotion will be in the future is like asking how important research is, adds Boddy: “It won’t change how marketing is done. But it will help us all make better decisions to do what we do more effectively.”

This feature was first published in The Drum's 4 May issue.

Neuroscience Tech Artificial Intelligence

Content created with:


Find out more


Find out more


At Jaywing, we’ve made it our mission to help clients establish concrete foundations in a world of shifting sands. As a data-powered integrated agency, we bring...

Find out more

More from Neuroscience

View all


Industry insights

View all
Add your own content +