Friends or foes: Are home smart speakers overstepping their boundaries?

Is there more to these devices?

I have recently experienced a couple of instances in which Facebook seemingly used background conversation to power ads, picking out keywords that are then used to influence what ads the user sees

The first case is a friend of mine who, as he is in his 40s, had a conversation over dinner with his partner about writing a will. That evening, he was served with ads for will writing services on Facebook, despite having had no device interaction on the subject, no Google searches or liking a relevant Facebook page.

The second example is when my wife and I were reminiscing one evening about a recent trip to Italy, laughing about an overly-enthusiastic barman in our hotel, who said ‘buona sera’ (good evening) in a particularly amusing way. Lo and behold, that evening my wife was served with Facebook ads promoting travel packages to Buenos Aires! (OK, so the technology is not foolproof.)

These are just examples from my life, but surely these can’t be coincidences…

Official responses

Last summer, Facebook rubbished rumours that they were using device microphones to eavesdrop on conversations, but there is evidence in action (as my examples demonstrate) that this may not be the case.

Google also claimed it "categorically" does not use what it calls "utterances" – the background sounds before a person says, "OK Google" to activate the voice recognition – for advertising or any other purpose. It also said it does not share audio acquired in that way with third parties. “Its listening abilities only extend to activating its voice services”, a spokesperson said.

It also states in its content policy for app developers that apps must not collect information without the user's knowledge. Apps found to be breaking this are removed from the Google Play store.

So, despite public denials, is there enough evidence out there to suggest that, in some way, eavesdrop advertising is happening?

Could these devices be linked to programmatic advertising?

Programmatic advertising is one of the biggest growth sectors in marketing; it powers display, outdoor, TV and social advertising. In simple terms, behavioural information is used to affect the ads users see and when. This could be anything from the weather influencing the display creative that’s served, to the adverts you see on TV in commercial breaks being based on the shows you’ve recently watched.

Are these devises another way forward for ad platforms to eavesdrop to serve ultra-targeted programmatic ads to users?

How could this be leveraged in the future?

Amazon Echo and Google Home also facilitate connections to smart home devices like electricity meters, thermostats and home appliances. Bearing in mind what we know, could this imply that ad platforms could, now or in the future, gain information about things like energy usage or food consumption, and advertise products and services to us accordingly?

The recent influx of smart kitchen appliances, especially Samsung’s Family Hub fridge, has opened a new way of managing chilled food in the home. With cameras in the fridge and freezer, wifi connectivity, a calendar and a media centre, devices like this when connected to Google (for example) could let the advertiser know the food we use the most and what we have run out of, what music we are listening to and even which appointments we have. This sounds like a gold mine for the platform and advertisers alike.

Artificial Intelligence

Ultimately, this is Artificial Intelligence at work. Sourcing information, analyzing it, and acting based upon the results, all within an algorithm. It is not just smart speakers and programmatic advertising where this is at work.

My wife posted a picture of the two of us on Facebook last year and I received a notification to that effect. When I clicked on the notification, the image was broken, but the meta information Facebook applied to it was very interesting. This meta information was shown because the image was broken, as any broken image would show alt text (usually the name of the file/image).

I tweeted about it at the time. For those that don’t know me, I have a beard and wear glasses…

check out the alt text on this broken #facebook image... clever stuff (can you tell it's a pic of me!?!) pic.twitter.com/EBr7qFoMVL

— Mark Leech (@Markba55) October 20, 2016

Is it possible that the AI applied to this image by Facebook (which identified the features of the photo), was used by Facebook to affect the ads my wife and/or myself were served? Beard oil products, perhaps, or an optician.

Disclaimer

With the aforementioned Facebook examples in mind, I took a look at Facebook’s privacy policy to look for any specific mentions of eavesdropping, mic usage, speech, etc. and there is nothing mentioned. The most pertinent part of the policy is the below:

Device information. We collect information from or about the computers, phones, or other devices where you install or access our Services, depending on the permissions you’ve granted. We may associate the information we collect from your different devices, which helps us provide consistent Services across your devices. Here are some examples of the information we collect:

-Attributes such as the operating system, hardware version, device settings, file and software names and types, battery and signal strength, and device identifiers.

-Device locations, including specific geographic locations, such as through GPS, Bluetooth, or WiFi signals.

-Connection information such as the name of your mobile operator or ISP, browser type, language and time zone, mobile phone number and IP address.

Unsurprisingly, Google’s privacy policy contains nothing specific either.

In lieu of my cynicism, the public denials from various platforms/services regarding involvement in any eavesdropping brand my thoughts pure conjecture.

What does this mean for the everyday user?

As technology in this area advances, the connectivity of voice-activated devices and other devices in the home, car and office will become more complex and reliant on the understanding of speech. Surely it is only a matter of time before platforms have to open up the prescript usage of this to allow them to not only maximize their services, but also potentially monetise it through advertising.

With all this in mind, how long will it be before we see Facebook-branded/powered assistants take over the house?

Mark Leech is operations director at Zazzle Media.