The Drum Awards Festival - Official Deadline

-d -h -min -sec

Mixed Reality SK-II Skincare

How SK-II disrupted the beauty industry in Japan with emerging technology


By Shawn Lim, Reporter, Asia Pacific

July 2, 2018 | 4 min read

SK-II launched its new smart store in Tokyo, Japan in May, with the aim of disrupting the beauty industry through the use of emerging technology.

The two-level mixed reality learning lab uses technologies like facial recognition, smart sensors and computer vision technology to allow customers to be a part of the experience using a digital wall called ‘The Art of You’, which reads a user’s emotions to generate personal digital art pieces by analysing facial expressions, head, eye and mouth movements.

Customers can also use the SK-II facial scan which does not require physical contact and instead, scans customer skin in a photobooth-like setting and then analyses skin across five KPIs. Once the facial scan is complete, the store uses facial recognition to give each customer a personalised journey as they move through the space, using the 43 cameras and three dozen computers powering the store’s intelligent interfaces.

The Japanese beauty giant is also previewing a limited quantity beta test of their ‘Smart Bottle’, an at-home enchanted object that features sensors to know when the bottle has been opened or closed and tracks daily usage with a companion app to maximise the product experience.

The lab was designed by experience design and digital marketing agency Huge. Speaking to The Drum about how the idea for the lab was conceived, Emily Wengert, group vice president of experience design at Huge says the agency began with a simple question: Could there be an augmented physical retail experience completely tailored to consumers?

This is because, as Wengert points out, customers want to be recognised as individuals, not just foot traffic. However, physical retail has been slow to deliver on the personalisation that has been common in e-commerce for years.

“Thanks to burgeoning capabilities in computer vision, machine learning, facial recognition and sensor technology, SK-II wanted to test these technologies to learn how they can change how customers experience their brand and products in an actual store,” Wengert adds.

Delving deeper into the tech features that were chosen for the store, Wengert explains a big focus for SK-II is skin, so naturally facial recognition was an obvious match. That is why Huge wanted to pair the depth of the SK-II proprietary skin scanning technology they have used for several years with a holistic personalised experience.

“People can move through the store with the least friction by letting their face unlock the experiences. We also added a few other touches, like sensors on the products to react to what people were picking up, and an art piece that built a custom visualization in real-time based on the emotions on the faces of people passing by,” she says.

However, as the technologies Huge used were still nascent and rarely tested in large physical spaces, it quickly learned that while tracking one person and one face was quite easy, multiplying that with hundreds of people every day, the system had to work a lot harder, according to Wengert. “We spent quite a lot of time optimising the camera placements and the underlying logic to achieve success for the majority of people visiting the popup,” she adds.

Both the agency and SK-II are pleased with how the smart store turned out, says Wengert, as one of its biggest goals was to hide the technology so that the space would feel magical and not ‘tech-y’ when it unlocked for customers.

“Oddly enough, we hid some of it so well that sometimes visitors didn't always realize just how much tech was working to create their experience. The good news is that the 42 cameras (39 of which were hidden) and 48 computers that powered the space aren't the focus. But many people still can't quite believe that it's working just from face alone!” she quips.

Mixed Reality SK-II Skincare

Content created with:


Huge is a digital agency that provides business strategy, user experience, visual design, marketing and technology services to some of the world's largest businesses...

Find out more

More from Mixed Reality

View all


Industry insights

View all
Add your own content +