SELF-IMPROVING VISION-LANGUAGE-ACTION MODELS WITH DATA GENERATION VIA RESIDUAL RL

Technical Analysis Supervised fine-tuning (SFT) has become the de facto post-training strategy for large vision-language-action (VLA) models, but its reliance on costly human demonstrations limits scalability and generalization. ["Probe, Learn, Distill (PLD)"] (https://wenlixiao.com/self-improve-VLA-PLD) is a three-stage plug-and-play framework that improves VLAs through residual reinforcement learning (RL) and distribution-aware data collection. High-level Analogy: Imagine you have a main chef (the VLA generalist model) who knows how to cook many dishes but sometimes struggles with tricky parts of a new recipe, like perfectly flipping a delicate crepe or getting a soufflé to rise just right. Instead of hiring an expensive master chef to show them everything from scratch (costly human demonstrations), our system uses a clever learning approach. First, for specific tricky steps, the main chef gets a small, focused 'mini-chef assistant' (the residual specialist policy). This assistant doesn't replace the main chef but learns only to make small, precise corrections when the main chef is about to mess up. It learns by trying slightly different ways to fix near-failures, essentially 'probing' where the main chef struggles. Next, the main chef starts cooking a dish. If they get to a tricky part and are about to make a mistake, the 'mini-chef assistant' quickly steps in, makes the necessary correction, and finishes that tricky step successfully. All these cooking sessions,…