RL for VLA: Recent trends

Evolving Robot Capabilities: Reinforcement Learning in VLA Post-Training Large Vision-Language-Action (VLA) models, typically trained through Behavioral Cloning (BC) on vast datasets of human demonstrations, have revolutionized robot manipulation by enabling impressive generalization across various tasks. However, these policies often hit a "demo ceiling" – they cannot learn from their own deployment failures without new human data, and fine-tuning billions of parameters with sparse, online experience is computationally expensive and risks degrading existing skills. Recent research has focused on integrating Reinforcement Learning (RL) into VLA post-training to enable self-improvement, boost precision, and enhance robustness. This report examines prominent trends in this area: advantage conditioning, modifying noise distributions, and residual policy learning. Advantage Conditioning: Decoupling Actor and Critic One approach to injecting RL capabilities while preserving a large, pre-trained BC policy is to decouple the actor (the BC policy) from a new, lightweight critic (Q-function). This allows the Q-function to learn from a broader range of experiences, including failures, without altering the foundational BC model. Q-Planning: Value-Guided Self-Improvement Q-Planning introduces a small, off-policy Q-function alongside a large visuomotor BC policy. The core insight is an asymmetry: while a BC policy can only be trained on successful demonstrations, an off-policy Q-function…