Open-source Robot VLAs and VLMs

Technical Analysis Open-source codebases have become very popular and useful in the robotics community, as they simplify, unify, and speed up model building and trainig. Here come two poluar frameworks for VLMs and VLAs "RobotVLA" and "Prismatic VLM" Robot VLAs The central motivation for the RoboVLMs codebase is to provide a standardized, flexible framework to understand "What Matters in Building Vision-Language-Action Models" (VLAs). This work addresses the rapid development of multi-modal foundation models and aims to determine how these models will help robots. RoboVLMs provides a flexible codebase designed to help the community investigate VLA models. It is explicitly engineered to allow researchers to integrate most existing VLMs into the VLA framework with minimal effort (approximately 30 lines of code). Additionally, the codebase released the strongest VLA model to date, which uses the KosMos VLM backbone. What is provided: VLM Integration: The framework requires setting up necessary attributes of the VLM backbone to prepare it for input token forwarding, including the , , , and the central (backbone). Configurations are needed to define how the model processes images into vision tokens for multi-modal feature fusion. Supported Backbones and Architectures: The codebase supports major backbones such as KosMos2, Flamingo, LLaVA, Uform-Gen, MoonDream, and PaliGemma. It supports four main VLA architectures: One-Step Continuous, One-Step Discrete, Interleaved Continuous,…