2013 Christmas adverts analysis: a 10 step guide to help marketers understand social media sentiment
Jaywing conducted analysis of over 100,000 Tweets to find out who was shedding tears over this year’s John Lewis Christmas ad. The results will surprise many as it turns out that Northerners are 20 per cent more likely to cry at 'the Bear and the Hare' than those in the South, with Londoners crying the least.
Further investigation by Jaywing’s data scientists into people’s reactions to both the John Lewis and M&S ads, showed that Londoners and the South East were the most cynical about both ads, displaying significantly more negativity than northerners. And in the race between the two Christmas advertising giants, John Lewis has not only seen far more tweets about its ad but far more internet searches than M&S too.
At the recent Big Data Analytics conference in London, one of the key recurring themes was sentiment analysis. For the uninitiated, sentiment analysis tools mine text captured in reviews, blogging and micro-blogging streams to extract subjective information, which can then be used to manage service, increase sales or develop propositions.
The appeal to determine and drive value from such consumer voices and the associated attitudes is obvious, as is the link to Big Data but the opportunity to exploit this value is littered with difficulties and perils; it’s not as easy as it might appear.
For example, when analysing the data, while it would have been desirable to draw more granular regional differences for the sake of a good story, there simply wasn’t enough data to be confident at this level in some regions, hence the aggregation of Scotland and Northern Ireland. The differences in emotion between the north and south, however, were statistically significant and therefore remain useful measures.
Social media data is rich and interesting, full of fascinating observations that, if we can harness them, will deliver insights the like of which has never been seen before. But as with any large sets of data or analysis, it’s easy to draw all kinds of incorrect conclusions; context and objectives are everything.
Clearly warm and engaging communication isn’t about cold hard numbers, but using the numbers to make sure you’re investing precious resources in the right places, at the right time is all-important. And that requires analytical rigour.
So we’ve created a 10-stage guide to help marketers understand how good analysis of social media data works and how to make sure they get an understanding that has genuine, actionable meaning.
1. Context is King
In the case of our recent Christmas campaign analysis we saw high occurrences of words such as ‘cry’, ‘cried’, ‘tears’, ‘sad’ from Tweeters. On face value, without knowledge of context or intent, these words appear to express negative sentiment and are often tagged as such in lexicons. But eliciting this response was the intent of the advert. A simple sentiment analysis tool, without looking at the underlying data, could consequently have the John Lewis Marketing Director with his head in his hands unnecessarily. Ensuring the word lexicon correctly scores words as positive or negative according to the context is crucial.
2. You can’t use standard analytical packages
Having accepted off-the-shelf tools may mess with your context, if you choose to avoid them, standard analytical packages won’t cope with the unstructured data or required analysis. Don’t expect Excel to be able to handle the volumes, or anyone other than an analyst with a proper mathematical background, to be able to deliver robust and valid analysis. Even heavyweight packages like SAS and SPSS won’t take you far down the sentiment analysis route.
3. Set clear objectives
You’ve got the context, tools and analysts. Now you need to be clear about what you want to learn from the analysis. Setting clear objectives means that when you set out in search of data, you will gather the right data to measure against your objectives. This is even more critical when considering Big Data sources such as Twitter; despite its 140-character limit, Twitter still generates over 8TB of data each day.
4. Be prepared for some serious data processing
Even with careful data specification, pulling data through the Twitter API is tricky. Using one of the few Firehose providers, that allows full access to all Twitter data, is preferable as it removes the barriers set by Twitter’s API restrictions; but you will need expert application programmers to interface with these organisations. It hasn’t gone unnoticed that Apple has recently acquired such a Firehose provider in Topsy.
5. Prepare clean data
The data you’ve acquired will be full of rogue characters that you need to convert and interpret, such as emoticons and you’ll need to de-dupe and prepare the data for your analytical package. Using Mapreduce and Hadoop can speed this process up enormously but requires specialist skills in this area.
6. Consider the breadth of data
We live in a world of content sharing so anything that perpetuates the intent of the original sentiment, such as a retweet, should be considered as part of the sentiment analysis.
7. Number of followers = reach, not necessarily impact
Sentiment tools often include measures of reach, assuming that those with large numbers of followers have greater influence over those followers. However, it’s not necessarily the case that a negative tweet that is seen by a large number of people means attitudes for those viewing the tweet will also be negative. Consider weighting sentiment by the number of people that have seen the tweet and how this might impact overall sentiment.
8. Verified accounts can skew results
Verified accounts on Twitter can have a huge impact on the analysis for a couple of reasons. Firstly, they are often associated with organisations or celebrities and, as such may have a disproportionate impact on the analysis you are conducting. Secondly, if the tone of voice of one of these organisations or celebrities is always positive, or negative, or cynical, or sarcastic, then once re-tweeted this is likely to impact the overall campaign sentiment.
9. So can brand tweets
Brand messages are more likely to be positive, which may similarly skew any analysis. Consider the impact of removing or keeping brand Tweets within the analysis.
10. Beware of comparisons producing neutral outcomes
When we compared the John Lewis Christmas advert with M&S’s equivalent, a tweet along the lines of “I loved John Lewis Xmas ad but hated M&S’s” could produce a neutral result as the negative sentiment cancels out the positive, when in fact there was a clear expression of positive sentiment about one (John Lewis) and negative sentiment about the other. Being able to spot and disentangle these kinds of issues at the data processing stage is important to establish a robust foundation for the sentiment analysis.
Social media and sentiment analysis is a fantastic opportunity to tap into the rich seams of information and data produced every minute of every day. But there are significant potential pitfalls if relying on the conclusions of such analysis to make brand and marketing decisions, unless you are absolutely confident in how this rather loose data are being treated. Is it in the right context? Did the analysis include anything that is likely to give a skewed outcome? Did it just look at a bunch of words or were phrases and parts of sentences considered?
Sentiment analysis, like any analysis for that matter, is not straightforward and without proper analytical minds, is open to substantial mis-interpretations and subsequent poor decision-making.
Analysis has always been about being clear on your objectives, creating robust and reliable data platforms where the analyst has confidence in the data, using the right analytical tool for the right job and being expert in the use of these, knowing the strengths and weaknesses of approaches and being able to generate robust conclusions once all of these things come together.
So whilst things like sentiment analysis are new, the disciplines that will ensure it delivers the value it promises are well established. It’s more a matter of making them all work together.
Adrian Lingard, is consulting managing director, Jaywing