Show-Harness: Just a VLM Agent Can Play Robots

Technical Analysis High-level Analogy: Imagine you have a super-smart 'brain' (a VLM) that knows a lot about the world but doesn't have a body. You want this brain to control various robots, like different types of industrial arms or grippers. Instead of trying to teach the brain the intricate, highly technical details of every single motor and joint for each specific robot (which would be like learning a completely new language for every machine), you give it a simplified, universal 'instruction manual'. This manual has clear, high-level commands like 'Move Forward', 'Turn Left', 'Grasp', or 'Release'. When the smart brain says 'Move Forward', a specialized 'translator' built into the robot knows exactly how to convert that universal command into the precise, low-level movements specific to that particular robot's motors and parts. The brain doesn't need to worry about the robot's physical anatomy; it just gives the high-level intent. 'Show-Harness' acts like this universal instruction manual and its intelligent translator. It allows a VLM to 'play' (control) any robot using a common, understandable set of actions, while the harness handles the complex, robot-specific physical execution. It's like a universal remote control for robots, where the remote interprets your simple button presses into the right signals for each different device. Motivation of the Work Despite their vast intelligence about the world, foundation VLMs struggle to translate this knowledge into…