The advancement in humanoid robots over the past few years has been incredible to watch. While robots were once very clunky and couldn’t do much, now they are able to jump, run, dance, and actually perform very useful functions.

Even the most advanced robots, however, were very easy to identify as robots and not actual humans.

While that is certainly still the case for now, researchers at Columbia Engineering made a robot that is learning how to lip sync to speech using nuanced movements, just like humans do.

This robot, dubbed EMO, is being trained so that its facial expressions will more perfectly match up with a real person’s while speaking or singing.

How is this training done?

Using artificial intelligence (AI), the robot looks at itself in the mirror and makes faces at itself. This allows it to learn how it is able to control its incredibly advanced mouth, lips, and cheeks. The lips themselves are made out of soft silicone, and they have 10 degrees of freedom thanks to 26 separately controlled motors.

Then, the robot is directed to watch YouTube. Lots of YouTube. By analyzing the mouth movements of real people talking, singing, emoting, and doing everyday things in their videos, the robot learns to mimic that authentic appearance.

After the training, EMO is able to not just talk in a fairly convincing way, but do it with accurate lip and mouth movements that look far more natural than any robots in the past. On top of that, it is able to perform this activity in multiple languages.

Hop Lipson is the director of Columbia’s Creative Machines Lab and the author of a study on EMO, published in the journal Science Robotics. In a statement about this robot, he said:

“Something magical happens when a robot learns to smile or speak just by watching and listening to humans. I’m a jaded roboticist, but I can’t help but smile back at a robot that spontaneously smiles at me.”

If you’re interested in seeing the robot in action, check out this impressive video:

EMO is clearly not 100% perfect at this point, but it is far better than other robots out there, which essentially just move their mouth up and down while speaking. And, the more it is trained, the better it will get. Yuhang Hu is another study author, and in the statement, he said:

“When the lip sync ability is combined with conversational AI such as ChatGPT or Gemini, the effect adds a whole new depth to the connection the robot forms with the human. The more the robot watches humans conversing, the better it will get at imitating the nuanced facial gestures we can emotionally connect with.”

The progress EMO has made so far is already quite impressive, and it seems clear at this point that in the coming years, robots will be able to imitate humans quite well, which, in some ways, is a scary thought.

If you enjoyed that story, check out what happened when a guy gave ChatGPT $100 to make as money as possible, and it turned out exactly how you would expect.