Data Pyramid for Embodied Manipulation: A Survey

Technical Analysis What data is being used in robotics and Embodied AI, and which one might be the most useful? "Data Pyramid" provides a thorough survey. High-level Analogy: Imagine you want to teach a robot how to be a master chef. The 'Data Pyramid' is like a comprehensive education system for this robot, combining different types of learning experiences: Bottom Layer (General Data): This is like the robot reading every cookbook on the internet, watching countless cooking shows, and learning about all sorts of ingredients, techniques, and culinary concepts. It gains a vast amount of theoretical knowledge and common sense about the world, but no hands-on experience. Second Layer (Simulation Data): Now, the robot steps into a super realistic cooking video game. Here, it can practice chopping, stirring, and baking endlessly, making mistakes without real-world consequences. The game gives it precise feedback on every virtual movement, but the virtual food might not feel exactly like real food. Third Layer (Egocentric/Exocentric Data): Next, the robot watches real human chefs at work, either through their eyes (egocentric, like a GoPro on their head) or from cameras around the kitchen (exocentric). It sees how actual human hands interact with real ingredients and tools, experiencing the true physics of cooking. It learns natural human behaviors, but it still needs to translate those human movements into commands for its own robotic arm and hand. Fourth Layer (UMI-style Data):…