ROBOTOK: AN INTERNET-SCALE DATA ENGINE FOR HUMAN DEMONSTRATION RETRIEVAL AND DEXTEROUS MANIPULATION LEARNING

Technical Analysis is an internet-scale data engine that, given a query human manipulation video, retrieves manipulation-relevant human demonstrations from web videos for training dexterous robot policies. High-level Analogy: Imagine you're trying to teach a robot to perform complex hand movements, like expertly peeling an apple or turning a tricky knob. Usually, you'd look for videos of people doing 'apple peeling' or 'turning knobs.' But the internet has millions of videos, all shot from different angles, with different lighting, different hands, and often the person's body is obscured. How do you find the exact way a hand moves, regardless of all these visual distractions? RoboTok is like a super-smart 'hand movement pattern matcher' for the entire internet. Instead of caring what the video looks like (e.g., 'Is this a kitchen?') or what it's about (e.g., 'cooking video'), RoboTok focuses purely on the 3D path of the hands. It first figures out where the hands are in 3D space, and then, crucially, translates these movements to be relative to the person's own body. This is like having a tiny GPS on each hand that reports its position not to a world map, but to a 'personal body map.' So, whether the camera is far away, close up, or from the side, the 'personal body map' always shows the hand moving in the same way relative to the person. Then, if you show RoboTok a video of a specific hand movement (say, a precise twisting motion), it quickly scans its enormous database of…