Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting

Technical Analysis High-level Analogy: Imagine you have a super-smart student who knows a lot about the world – they’re great at languages, history, science, and can answer all sorts of questions (this is like a powerful Vision-Language Model, or VLM, trained on vast internet data). Now, you want this student to become a robot technician, learning to control a robot arm precisely. The old way was like teaching them a completely new, very rigid 'robot code' language. They would spend so much time learning this code that they'd start forgetting all their general knowledge. They could operate the robot, but if you asked them a complex question about the world or gave them an instruction in a slightly different way, they'd be lost. They'd forget their 'human' language for the 'robot' language. This new approach (VLM2VLA) is like teaching the student to describe the robot's actions using their existing human language. Instead of just a sequence of numbers, the robot understands 'move forward by 5 centimeters' or 'grasp the object.' Because the actions are described in a language the student already understands, they can learn to control the robot without forgetting all their other knowledge. They become a brilliant robot technician who can still understand complex, varied instructions, adapt to new situations, and reason about the world, because their core intelligence and language skills are fully preserved. Motivation of the Work The core motivation is to overcome the problem…