Learning Realistic Expressions for Humanoid Face Robots

For a humanoid face robot to appear approachable and believable in education, companionship, and performance settings, it needs to generate full-face dynamic expressions that engage the brows, eyes, lips, and cheeks. Existing systems typically treat the robot face as a downstream execution target of human facial animation, relying on low-dimensional expression parameters or pixel-video intermediates — which splits expression generation from physical execution: the former struggles to carry the diverse full-face motion that accompanies a single utterance, while the latter places the real robot under an out-of-distribution visual target and still requires extraction, smoothing, and mapping before it reaches the actuators. This work proposes a text- and audio-driven, high-DoF-density biomimetic face-robot system. On the hardware side, a flexible tendon-sheath transmission network decouples the servos from the facial actuation points, packing 34 servo actuators into a compact, approachable head only 19.5 cm tall. On the algorithm side, we learn a robot-aware motion latent interface and generate full-face motion directly in this space, so that text controls emotion and action semantics, audio constrains lip-sync, and robot image observations anchor the interface to the real hardware. The system avoids first generating pixel video or reconstructing the robot’s face at inference time, reducing intermediate artifacts and extra latency. Experiments show the system generates natural, controllable speaking expressions and provides a unified interface for robot expression driving that stays stable across performers.

Motivation

Humanoid face platforms fail in two directions: either the motion looks mechanical (physically valid but visually dead) or the learned controller drives the hardware outside its safe operating envelope chasing visual realism. We want expressions that look human and run on real motors.

Hardware

Contributions

High-DoF-density biomimetic face hardware. We build a biomimetic head with a flexible tendon-sheath transmission network, packing 34 servo actuators into a 19.5 cm-tall head, so an appearance closer to a compact, feminine proportion can still carry fine-grained, full-face expressions.
A robot-aware motion latent interface. We fold robot image observations into motion-representation learning, so generation, visual performance read-out, and actuator mapping share the same motion space, instead of treating the robot merely as a downstream retargeting target of a pixel-video or blendshape pipeline.
Conditional generation inside the motion latent space. A single network supports three input settings — T2V, A2V, and TA2V — inside the motion latent, with both an offline bidirectional-Transformer version and a streaming autoregressive version; inference never passes through a pixel-video intermediate, avoiding video artifacts and extra reconstruction latency.
System validation. We close the full loop from text/audio to motion latent to virtual rendering / actuators, and provide experimental validation of generation quality, controllability, cross-performer consistency, and robot deployment. Robot-side quantitative evaluation (expression recognition, lip/emotion-vertex error, user study) is currently reported with placeholder results, to be replaced once the physical closed-loop experiments are complete.