Initial attempts to identify a standard procedure for interpreting facial expressions were made as early as the 1970s. What did people expect from it? The truth, the whole truth and nothing but the truth. If people’s faces could be read like open books, it would be possible to find out, for instance, if a defendant was actually guilty or how a patient was really feeling. The Facial Action Coding System (FACS) devised by Paul Ekman and Wallace Friesen – the forerunner of today’s software – appeared to make this possible for the very first time. People’s feelings are analysed on the basis of tiny facial movements called microexpressions. A real smile or just a front? Psychologists in particular began using the technique during therapy. Changes in expression can be recorded immediately and attributed to one of the seven basic emotions: happiness, surprise, fear, anger, disgust, contempt and sadness. However, to this day, one emotion remains a mystery: that thing called love.

If we watch a new M&M’s advertisement, it may be a product of emotional decoding. Various spots are tested on potential customers beforehand – and the “most emotional” one appears on TV. It is increasingly the case that big companies like Mars Inc. – of which M&M’s is a part – as well as Coca Cola and Unilever do not conduct laborious surveys after new advertisements have been broadcast. Why makes things more complicated? Customers sit in front of a tablet PC with facial recognition software and the job’s done. The software that observes our expressions usually comes from Affectiva, one of the world’s most successful businesses in the industry. Affectiva claims that its program can even perceive subconscious reactions, thanks to an extensive database containing analyses of 5.3 million faces from 75 countries. Before we even realise that we have pulled a face, the program has already recorded our disapproval.

In this way, advertisements can be fine-tuned down to the last second and perfectly adapted to the audience. Attention levels fall from the first moment. Or nine out of ten people found the talking dog funny. Oh dear, this actor isn’t going down well. The BBC uses Affectiva to trial new shows, and Disney has adopted it for trailers for new films. Whether flops will become a thing of the past remains to be seen. MarketsandMarkets currently estimates that the market for emotional decoding will be worth USD 42 billion by 2020. That is almost the same as the total revenue from online selling in Germany!

Let’s assume you’ve had a rotten day and you just want to get home, but before that you have to do the shopping. When you enter the supermarket, a digital display shows you a winking smiley. Or a special offer: a family pack of chocolate ice cream now at half price. Just what you need right now. You’re in luck! But it’s no coincidence. The facial recognition software behind the screen has immediately recognised an unhappy customer. The remedy: a sweet smiley or something sweet to warm your heart.

Emotional decoding is also opening up new opportunities in retailing: the chance to address individual customers in real time. Until now, this was only possible online. On the web it is comparatively easy to understand customers because you can track their movements. But the ability to respond straight away to customer needs before they even exist is completely new in bricks-and-mortar retailing. In Germany, 60 screens developed by Berlin start-up Pyramics are currently showing personalised advertisements. A few supermarkets and electronics retailers also record information such as customers’ facial expression, age and gender. The question is whether we even want that. And whether we even have a choice. What if the software is used without our knowledge? Is that even allowed?.

Es braucht keine künstliche Gefühlsanalyse, um mögliche Bedenken wahrzunehmen. Emotional Decoding hört sich ein bisschen Identifying possible concerns does not require automated emotional analysis. Emotional decoding sounds a bit like surveillance. Nowadays, many of us don’t even want to have our photos taken at family parties – so how are we supposed to agree to be filmed while shopping? No thanks! In theory, the software can currently be deployed – and is deployed – without our permission. As one opinion (clearly from operators) goes, as long as no “personal data are collected and stored”, so people remain anonymous, customers do not need to be informed and give their permission.

In contrast, German data protection officer Johannes Caspar says that a connection to an individual is possible if the person recorded is identifiable. This means that simple observation represents intrusion, even if the data are not stored. And we must not forget that this idea goes even further: all the way into our private lives. It’s not just a question of retailing and advertising. When using emotional decoding, it is also a matter of whether our friends should know that we have just told them a little white lie. This would be possible with ooVoo, a video telephony program similar to Skype. The technology assesses callers’ emotions in real time (using Affective, as it happens). The end of the poker face – and maybe of a friendship.

The thing about recognising emotions is this: we can already see how a person feels and analyse their faces down to the last wrinkle. To come back to the example of the comforting chocolate ice cream: companies like Affectiva (still) do not really know everything about customers. The unhappy customer might not even like chocolate. Or they find ice cream too cold. Maybe the customer is on a diet, so the personalised offer is even more annoying. What’s more, no matter how good it is, software will never be able to take one parameter into account: people don’t always act rationally. Purchasing decisions often depend on gut feeling. Even if emotional decoding recognises at precisely the right time – when we have a product in front of us – what that feeling is telling us, we change our minds at the last minute.
Actually, I think I’ll have gummy bears today.