FLORA: An Open Model Suite for Scaling Robotics Data Engines

Technical Analysis is a set of models and pipeline that allow generating outcome reward labels (needed for monitoring real-world deployments), process reward labels (needed during RL post-training), timestamped captions (needed for filtering and balancing pretraining data). High-level Analogy: Imagine you're watching a long, complex cooking video. Instead of you having to carefully re-watch, pause, and write down every tiny step ('chop onion,' 'sauté garlic,' 'add spices') and then rate how well each step was done, Flora is like a super-smart assistant that does it all for you automatically. It breaks the video into specific actions, describes each one clearly, and then gives a score for how well the chef performed, even spotting tiny mistakes or if the final dish was a success or a partial success. It's much faster and cheaper than hiring many human assistants or using a very expensive, slow expert, allowing you to quickly get detailed feedback for thousands of cooking videos. Motivation of the Work Robotics foundation models, which are advanced AI systems designed to control robots, critically rely on vast amounts of high-quality labeled data. This data includes different types of annotations: Outcome Reward Labels: To monitor how robots perform in real-world environments and evaluate their overall success or failure. Process Reward Labels: To guide robots during complex learning processes, providing feedback at each step. Timestamped Captions: Detailed descriptions of…