Skip to main content
Visit DexHubOrder Now

CAIP: Contrastive Action-Image Pre-training for Visuomotor Control

UC Berkeley and NVIDIA pre-train a vision encoder on 3D hand keypoints from 32,041 hours of human video. On a Dexmate Vega with Sharpa Wave hands it beats the strongest baseline by more than 30 points on real-world dexterous tasks.

Paper

15 Jun 2026

CAIP pairs 3D hand keypoints from egocentric human video with images in a contrastive objective, then drives dexterous manipulation on a Dexmate Vega with Sharpa Wave hands.

Contrastive Action-Image Pre-training for Visuomotor Control

Yuvan Sharma1,*, Dantong Niu1,2,*,‡, Anirudh Pai1,*, Zekai Wang1, Zhuoyang Liu1, Baifeng Shi1, Stefano Saravalle3, Boning Shao1, Ruijie Zheng2, Jing Wang2, Konstantinos Kallidromitis4, Yusuke Kato4, Fabio Galasso3,5, Yuke Zhu2, Danfei Xu2, Linxi "Jim" Fan2, Jitendra Malik1,†, Trevor Darrell1,†, Roei Herzig1,†

  • 1UC Berkeley
  • 2NVIDIA
  • 3Sapienza University of Rome
  • 4Panasonic
  • 5ItalAI

* Equal contribution · ‡ Project lead · † Equal advising

arXiv preprint arXiv:2606.17256

PDFarXivProject PageCode

Abstract

Existing vision encoders for robotics face a fundamental bottleneck: robotic datasets lack the scale necessary for large-scale pre-training. Prior work circumvents this data scarcity by turning to internet-scale image and language data or egocentric human video. While these models show promise, neither paradigm learns from paired vision and action data, which downstream visuomotor control policies require. However, robot trajectories, the most direct source of this paired signal, are not available at pre-training scale, motivating us to extract action signals from abundant human video instead. To this end, we introduce CAIP (Contrastive Action-Image Pre-training), a vision encoder that treats human hand poses from large-scale egocentric video as a proxy for end-effector actions. By extracting 3D hand keypoints, a representation that aligns naturally with downstream robot action spaces, CAIP learns a unified action-image representation through a contrastive objective. Leveraging 32,041 hours of egocentric human video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art vision encoders including DINOv2, SigLIP, MVP, and R3M. Evaluated on a challenging real-world dexterous manipulation setup using Dexmate Vega and Sharpa Wave hands, CAIP yields performance gains of more than 30% on tasks involving folding, pouring, and fine-grained manipulation. Our results show that our method of contrastive action-centric pre-training yields a scalable path to achieving robust visual representations better suited for physical interaction.

Citation

@misc{sharma2026contrastiveactionimagepretrainingvisuomotor,  title = {Contrastive Action-Image Pre-training for Visuomotor Control},   author = {Yuvan Sharma and Dantong Niu and Anirudh Pai and Zekai Wang and Zhuoyang Liu and Baifeng Shi and Stefano Saravalle and Boning Shao and Ruijie Zheng and Jing Wang and Konstantinos Kallidromitis and Yusuke Kato and Fabio Galasso and Yuke Zhu and Danfei Xu and Linxi "Jim" Fan and Jitendra Malik and Trevor Darrell and Roei Herzig},  year = {2026},  eprint = {2606.17256},  archivePrefix = {arXiv},  primaryClass = {cs.RO},  url = {https://arxiv.org/abs/2606.17256}, }

Start Your Physical AI Journey with Us

Talk to Us
Vega 1 Pro reaching forward with both arms