← Back

ROSE: A Shared Cross-Embodiment Facial-Motion Latent Space for Humanoid Robot Expression

Yongji Fu, Man Cao, Yi Zhou, Fei Lin, Mark F. Hansen, Guanqun Cao

ICRA 2027, Submitted, 2026

abstract

Natural, coherent full-face dynamic expressions are central to making humanoid robots socially approachable, yet existing systems commonly realize only limited lip motion or scripted expressions. The limitation is both physical and algorithmic: a compact head leaves little room for densely packed, independently controllable actuation, while learned systems often generate an avatar video, landmarks, or facial parameters and then infer actuator commands through visual retargeting — a render-and-invert path that adds latency and artifacts and does not reliably transfer generated behavior to a physical robot. We present ROSE (RObot facial expression through Shared motion Embeddings), a framework that integrates biomimetic hardware, cross-embodiment motion representation learning, and multimodal motion generation. On the hardware side, a flexible tendon-sheath transmission integrates 34 actuators into a 19.5 cm biomimetic head; by decoupling actuator placement from facial action points, it enables coordinated full-face expression within a compact volume. On the algorithmic side, ROSE-VAE learns an identity-independent shared facial-motion latent space from human, digital-human, and robot observations; ROSE-Flow maps text, audio, or both directly to trajectories in this space without using video as a generator input; and ROSE-Act converts these trajectories into constrained actuator commands.

Overview

ROSE overview
Fig. 1: ROSE overview. Human, digital-human, and face-robot observations are mapped by ROSE-VAE into a shared facial-motion latent space. ROSE-Flow generates motion-latent trajectories from text and/or audio, and ROSE-Act maps them to the actuator actions of a real robot. The right side illustrates real-time interaction, human-to-robot retargeting, and text/audio to expression applications.

Robot platform

The platform accommodates 34 actuators for facial expression and neck motion, combining tendon-sheath transmission with local rigid mechanisms. Actuators are concentrated near the top and rear of the head, with tendons routed to the brows, eyelids, cheeks, and lips; local linkages, pushrods, and shafts additionally drive the jaw, mouth corners, philtrum, and eyeballs, while the neck supports pitch, roll, and yaw.

ROSE robot platform: tendon-sheath transmission and actuator layout
Robot platform. (a) Full assembly; (b) eyebrow and cheek drive wires; (c) eye module with eye-pitch actuator and eyelid pushrod; (d) mouth module with jaw drive rod and upper/lower lip linkages; (e) neck module with yaw actuator and paired tilt actuators.

Method

  • ROSE-VAE: identity–motion factorization. Identity and motion encoders extract an identity latent and per-frame motion coefficients; the coefficients are projected through a shared, orthogonality-regularized learnable motion basis and added to the identity latent before decoding. Human, digital-human, and robot observations share the same encoder, motion basis, and decoder, and a masked component perceptual loss keeps fine lip, eye, and brow motion from being washed out.
  • ROSE-Flow: text–audio conditional motion generation. A dual-stream MMDiT performs flow matching in the motion latent space. Text specifies global intent and expression style, while Wav2Vec 2.0 audio features drive articulation through frame-local audio window attention. Progressive modality unification — audio-to-motion (A2M) pretraining, text-to-motion (T2M) adaptation initialized from it, weight merging, then joint training — yields one model for A2M, T2M, and TA2M.
  • ROSE-Act: latent-to-actuator mapping. Synchronized robot image / actuator-state pairs are collected through safe motor babbling; ROSE-VAE extracts their motion coefficients to train the latent-to-actuator mapping. At inference, calibration, joint limits, and rate limits constrain the commands, with no intermediate image reconstruction.
ROSE-Flow architecture
ROSE-Flow architecture. (a) Text–motion dual-stream MMDiT trained with the frozen ROSE-VAE encoder; generated motion latents are mapped to real-robot servo commands, and decoded faces are for visualization only. (b) Audio window attention: each motion-frame query attends only to audio tokens in its local window. (c) Progressive modality unification: A2M pretraining → T2M adaptation → weight merging → joint training.

Key results

  • Cross-identity motion representation: on 24 cross-identity reenactment tasks, ROSE-VAE reaches a target-identity similarity (CSIM-T) of 0.741 vs. 0.730 for the FLOAT VAE, and a head-pose error (APD) of 1.26° vs. 1.35°.
  • Talking-face generation: lowest FID and FVD on both HDTF and RAVDESS (FVD 54.6 / 56.8 vs. 70.2 / 81.8 for the strongest baseline, FLOAT), and the lowest LSE-D on both (9.785 / 7.298).
  • Real-robot lip articulation: lowest mouth-latent MSE on all three RAVDESS test utterances, e.g. 0.0158 vs. 0.0202 for RIT on Speech 1.
  • Real-robot deployment: skipping image generation, landmark extraction, smoothing, and retargeting raises the execution frame rate from 15 FPS to 60 FPS; in a user study ROSE scores 3.41 / 3.36 / 3.78 for realism, emotional resonance, and lip sync (RIT: 2.98 / 2.73 / 3.40).

Video

Demo clips

Eight short expressions generated by the same network from text alone (T2V), showing fine-grained coordination of brows, eyes, and lips.

Blinks quickly, looking slightly nervous
A faint smile that quickly fades into a more serious look
Narrows the eyes and looks away, seeming a little impatient
Raises one eyebrow and parts the lips slightly
A soft smile with a playful eyebrow raise
Tilts the head, with a hint of doubt
Tilts the head slightly while raising the eyebrows in curiosity
Opens the mouth slightly, with a soft look of wonder

Face-to-robot retargeting

The same human performance, read out through the motion latent interface and retargeted onto the face robot’s actuators.

Motion read out from a human facial performance, retargeted to face-robot actuator commands and driving the real hardware frame by frame.