ROSE: A Shared Cross-Embodiment Facial-Motion Latent Space for Humanoid Robot Expression
ICRA 2027, Submitted, 2026
abstract
Natural, coherent full-face dynamic expressions are central to making humanoid robots socially approachable, yet existing systems commonly realize only limited lip motion or scripted expressions. The limitation is both physical and algorithmic: a compact head leaves little room for densely packed, independently controllable actuation, while learned systems often generate an avatar video, landmarks, or facial parameters and then infer actuator commands through visual retargeting — a render-and-invert path that adds latency and artifacts and does not reliably transfer generated behavior to a physical robot. We present ROSE (RObot facial expression through Shared motion Embeddings), a framework that integrates biomimetic hardware, cross-embodiment motion representation learning, and multimodal motion generation. On the hardware side, a flexible tendon-sheath transmission integrates 34 actuators into a 19.5 cm biomimetic head; by decoupling actuator placement from facial action points, it enables coordinated full-face expression within a compact volume. On the algorithmic side, ROSE-VAE learns an identity-independent shared facial-motion latent space from human, digital-human, and robot observations; ROSE-Flow maps text, audio, or both directly to trajectories in this space without using video as a generator input; and ROSE-Act converts these trajectories into constrained actuator commands.
Overview
Robot platform
The platform accommodates 34 actuators for facial expression and neck motion, combining tendon-sheath transmission with local rigid mechanisms. Actuators are concentrated near the top and rear of the head, with tendons routed to the brows, eyelids, cheeks, and lips; local linkages, pushrods, and shafts additionally drive the jaw, mouth corners, philtrum, and eyeballs, while the neck supports pitch, roll, and yaw.
Method
- ROSE-VAE: identity–motion factorization. Identity and motion encoders extract an identity latent and per-frame motion coefficients; the coefficients are projected through a shared, orthogonality-regularized learnable motion basis and added to the identity latent before decoding. Human, digital-human, and robot observations share the same encoder, motion basis, and decoder, and a masked component perceptual loss keeps fine lip, eye, and brow motion from being washed out.
- ROSE-Flow: text–audio conditional motion generation. A dual-stream MMDiT performs flow matching in the motion latent space. Text specifies global intent and expression style, while Wav2Vec 2.0 audio features drive articulation through frame-local audio window attention. Progressive modality unification — audio-to-motion (A2M) pretraining, text-to-motion (T2M) adaptation initialized from it, weight merging, then joint training — yields one model for A2M, T2M, and TA2M.
- ROSE-Act: latent-to-actuator mapping. Synchronized robot image / actuator-state pairs are collected through safe motor babbling; ROSE-VAE extracts their motion coefficients to train the latent-to-actuator mapping. At inference, calibration, joint limits, and rate limits constrain the commands, with no intermediate image reconstruction.
Key results
- Cross-identity motion representation: on 24 cross-identity reenactment tasks, ROSE-VAE reaches a target-identity similarity (CSIM-T) of 0.741 vs. 0.730 for the FLOAT VAE, and a head-pose error (APD) of 1.26° vs. 1.35°.
- Talking-face generation: lowest FID and FVD on both HDTF and RAVDESS (FVD 54.6 / 56.8 vs. 70.2 / 81.8 for the strongest baseline, FLOAT), and the lowest LSE-D on both (9.785 / 7.298).
- Real-robot lip articulation: lowest mouth-latent MSE on all three RAVDESS test utterances, e.g. 0.0158 vs. 0.0202 for RIT on Speech 1.
- Real-robot deployment: skipping image generation, landmark extraction, smoothing, and retargeting raises the execution frame rate from 15 FPS to 60 FPS; in a user study ROSE scores 3.41 / 3.36 / 3.78 for realism, emotional resonance, and lip sync (RIT: 2.98 / 2.73 / 3.40).
Video
Demo clips
Eight short expressions generated by the same network from text alone (T2V), showing fine-grained coordination of brows, eyes, and lips.
Face-to-robot retargeting
The same human performance, read out through the motion latent interface and retargeted onto the face robot’s actuators.