arXiv 2409.08273
Hand-Object Interaction Pretraining from Videos
By Himanshu Gaurav Singh, Antonio Loquercio, et al.
Published 2024-09-12
Mindmap
Browse the paper's core ideas, clusters, and relationships in a structured outline.
We present an approach to learn general robot manipulation priors from 3D hand-object interaction trajectories. We build a framework to use in-the-wild videos to generate sensorimotor robot trajectories. We do so by lifting both the human hand and the manipulated object in a shared 3D space and retargeting human motions to robot actions. Generative modeling on this data gives us a task-agnostic base policy. This pol…