MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision

Technical Analysis (Minting IN-the-Wild Trajectories) is a foundation model for world-space hand motion reconstruction from ego-centric RGB video. From a single shared spatiotemporal video representation, MINT jointly predicts the camera trajectory, field of view (FoV), camera-frame hand states, and per-frame hand observability, and then produces world-space hand motion via explicit coordinate transformations. High-level Analogy: Imagine you're wearing smart glasses that record everything you see. To understand what you're doing, like picking up a cup, you need to know exactly how your head is moving in the room (the camera's motion) and precisely where your hands are in relation to the objects (the hand motion). The old way (state-of-the-art) is like having several different specialists analyze your video separately: one for head movement, one for finding your hands, one for estimating depth, and another for combining it all. This process is slow, expensive, and if one specialist makes a mistake, it can throw off all the others, making it hard to get a smooth, accurate, combined picture. MINT is like having a single, super-smart detective built right into your glasses. This detective watches your video and, in one go, instantly understands both your head's exact movement and the precise 3D position and shape of your hands in the real world. It doesn't need separate specialists or complex back-and-forth calculations. It sees the whole picture at once, making the process much…