Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

▶Overview Video

§Abstract

Learning generalizable robot manipulation policies requires large-scale and diverse demonstration data. Egocentric human manipulation videos offer rich scene and task diversity, and prior work has shown that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale. However, whether this approach can provide pretraining benefits for vision-language-action models at scale remains unexplored.

We present Ego2Robot, a scalable pipeline that converts egocentric human manipulation videos into robot training data through action retargeting, robot-arm visual synthesis, and multi-level quality curation. Ego2Robot supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies, making it the largest ego-to-robot dataset to date. To evaluate generalization, we extend RoboTwin 2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics. Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment.

1The Ego2Robot Pipeline

Given egocentric video of human hand manipulation, the pipeline produces embodiment-specific robot training data in three stages. It supports two input paths: Path A uses ego datasets with existing hand-pose annotations; Path B estimates hand poses from raw video (WiLoR per-frame reconstruction + DynHaMR temporal optimization), then both paths share a unified pipeline that renders any of 15 morphologies in parallel.

2Data at Scale

Applied to four egocentric sources — ANT (7h, in-house), EgoDex (732h), ViTRA (249h), and EgoVerse (954h), ~1,940h total — and rendered across 15 morphologies to yield 18,561h of synthetic robot data, co-trained with ~6,565h of real robot data (DROID, AgibotWorld, InternData).

3Disentangled Evaluation Framework

Existing protocols bundle many distribution shifts into a single OOD score. We extend RoboTwin 2.0 with 11 independent perturbation settings plus the external EBench (higher-mounted camera, closer to the egocentric view), decoupling four generalization axes.

4Main Results

A VLA (Qwen3.5-4B backbone + Diffusion Transformer action head, 32-step camera-frame relative EEF chunks) is pretrained under four configurations (robot-only, plus Ego2R:robot at 1:3, 3:1, 1:1), finetuned on RoboTwin's 50 clean-setting tasks (Aloha-Agilex), and evaluated per perturbation. Every pretraining run uses identical hyperparameters (200K steps × batch 12 × 8 GPUs) and thus processes the same ~19.2M frames, so comparisons are fair regardless of dataset size.

Success rate (%) on RoboTwin 2.0 (per-dimension) and EBench. Bold = best in column; green = >5% gain vs. Robot-only.

Per-perturbation breakdown

Per-perturbation breakdown (%). Columns: Visual (BG, Light, Color), Scene (Height, Clutter, Camera), Embodiment (ARX, UR5, Franka), Task (Obj, Lang). Bold = best in column; green = >5% gain vs. Robot-only.

• Co-training improves OOD generalization. Ego2R+Robot (1:1) leads in five of seven columns — 53.5% on RoboTwin Randomized (+2.6) while keeping 68.1% Clean (+5.9).
• Visual appearance benefits most. At 1:1: background +4, lighting +8 (scene diversity in ego video), robot color +6 (multi-morphology rendering).
• Scene layout shows moderate gains. Camera-offset robustness improves +6 at 1:1, as ego videos naturally exhibit diverse head poses and viewpoints.
• Embodiment transfer benefits from multi-morphology data. ARX 44→51 (1:1); UR5 peaks at 31 (3:1). Franka stays below 7% — large kinematic gap.
• Task semantics improve consistently. Unseen-object 29%→40% (+11 at 3:1); paraphrased-instruction robustness reaches 69% at 1:1.
• EBench confirms gains under higher viewpoint. 3:1 reaches 51.7 (+12.1) — benefits amplified when the camera bias partially matches the egocentric view.

5Ablations: Pipeline Value & Morphology Scaling

Ego-only pretraining (no robot data) isolates the pipeline's contribution. Training on raw ego alone reaches only 28.1% on RoboTwin Randomized; processing through the pipeline (single morphology) lifts it to 31.7% (+3.6). Scaling 1→15 morphologies improves to 33.5%, and adding raw ego alongside 15-morphology data jumps to 37.3% — raw ego acts as a 16th "morphology".

6Real-Robot Experiments

On an ARX ACone platform across five long-horizon tasks, in a few-shot regime of only 20 teleoperated demonstrations per task, we compare Robot-only, Mix (Ego2R+Robot 1:1 pretraining), and Mix + Ego2R Play (same pretraining, but with pipeline-converted ego-play demonstrations mixed into the finetuning data — synthesized from casual first-person recordings of hand manipulation in the scene, ~7 min each). Mix + Ego2R Play is best on all five tasks.