RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation

Technical Analysis is a unified resource including data, benchmarks, and models of intermediate representations for manipulation. It comprises RoboInter-Tool, RoboInter-Data, and RoboInter-VLA. RoboInter establishes a practical foundation for advancing robust and generalizable robotic learning via fine-grained and diverse intermediate representations. High-level Analogy: Imagine you want a chef robot to cook a complex meal. The traditional way is like giving the robot a high-level command ('Cook pasta primavera') and a video of a human cooking it. The robot often struggles because it doesn't understand the tiny, intricate steps, like 'how to chop an onion' or 'where to place the spatula'. It's like having a recipe that only says 'chop vegetables' but doesn't explain how to chop, what a chopped vegetable looks like, or which tool to use. RoboInter is like creating the most detailed, step-by-step interactive cookbook imaginable for robots. First, human experts use a special RoboInter-Tool (like an interactive recipe editor) to go through cooking videos frame-by-frame. For every micro-action – say, picking up a tomato – they pinpoint exactly where the robot's gripper is, draw a box around the tomato, mark the contact point, and even trace the gripper's path. This generates RoboInter-Data, a colossal collection of these super-detailed, 'intermediate' instructions and visual cues for millions of small cooking actions. Then, this data is used to train a Planner (the…