Can robots read your mind? Emo can predict your facial expressions

Robots might soon be able to smile back at the right moment. Researchers at Columbia Engineering’s Creative Machines Lab have built a robot named Emo that may change how we interact with machines.

Emo can not only make its own facial expressions but can also pick up on subtle cues in your facial expressions. The robot can use this information to predict what your face is about to do and even mirror your smile in real time.

Why smiling robots matter

We humans are pretty good at reading nonverbal cues, like facial expressions. They help us communicate and form social bonds. Unfortunately, robots have (up to now) been a little clueless in this area.

That’s a problem because we’re about to see a lot more robots in our daily lives. An expressive robot could make these interactions feel way more natural.

If a robot coworker can understand that you’re frustrated, it might know to offer help. A robot tutor might be better at figuring out if you’re confused, allowing for better teaching.

How robot Emo got its smile

Emo may look like a disembodied head, but there’s a lot going on under the surface:

The hardware

Beneath Emo’s seemingly simple exterior lies a sophisticated assembly of components designed to bring it to life.

The soft, silicone skin not only gives Emo a more human-like appearance but also houses an intricate network of actuators.

These actuators, akin to human muscles, enable Emo to move its face in subtle, complex ways that go far beyond basic mechanical movements. They allow for a range of expressions that can mirror the nuanced emotional states of humans.

Cameras

Adding another layer to Emo’s human-like capabilities are the cameras discreetly placed within its eyes.

These are not mere recording devices but windows through which Emo sees and interprets the world. They enable the robot to make eye contact, an essential aspect of non-verbal communication and emotional connection.

This visual input is critical for the AI to analyze and predict human expressions, forming the foundation for Emo’s interactive abilities.

The brains

The essence of Emo’s ability to anticipate and mimic human expressions lies in its “brains” – two advanced AI models working in tandem.

The first AI model plays the role of an attentive observer, scrutinizing the human face for early signs of an impending smile or other expressions.

It looks for the subtle, almost imperceptible, changes in the face that precede a smile, such as a slight twitch or the beginnings of a grin.

Second AI

Once these precursors are identified, the second AI model springs into action. It interprets these cues and translates them into a series of commands for Emo’s actuators.

This translation process is where the magic happens, enabling Emo to replicate the observed expression almost simultaneously with the human.

The coordination between these two models allows for a seamless interaction, making Emo’s responses feel genuine and timely.

Self-modeling

Perhaps the most fascinating aspect of Emo’s development is its process of self-modeling. Emo learned to control its facial movements through a process analogous to human learning.

By observing and experimenting with its own expressions in front of a camera, the robot discovered the relationship between facial movements like smiling and the actuator commands required to produce them.

This self-guided learning process is reminiscent of how humans learn to control their facial expressions by observing themselves in a mirror.

Human-robot interactions

“I think predicting human facial expressions accurately is a revolution in HRI [Human-Robot Interaction],” said Yuhang Hu, PhD student and lead author of the study describing Emo.

“Traditionally, robots have not been designed to consider humans’ expressions during interactions. Now, the robot can integrate human facial expressions as feedback.”

In essence, if robots can better understand what we’re feeling, they could respond in ways that make more sense and build more genuine trust.

Robots with a smile and personality

The researchers behind Emo say the next step is to combine its nonverbal abilities with the power of large language models like ChatGPT, so it could hold conversations, too. That means robots with true personality might not be so far away.

Hod Lipson, James and Sally Scapa Professor of Innovation and head of the Creative Machines Lab, explained why this work is significant:

“By advancing robots that can interpret and mimic human expressions accurately, we’re moving closer to a future where robots can seamlessly integrate into our daily lives, offering companionship, assistance, and even empathy.”

Of course, with this power comes responsibility. If a robot truly understands our emotions, that opens a whole can of ethical questions around privacy, manipulation, and influence.

For now, just be prepared – the next time you encounter a robot, it might be reading your face more carefully than you think, and possibly smiling back before you even know you’re about to.

The study is published in the journal Datadryad.org.

Video Credit: Creative Machines Lab/Columbia Engineering

—–

Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates. 

Check us out on EarthSnap, a free app brought to you by Eric Ralls and Earth.com.

—–