Steering Your Diffusion Policy with Latent Space Reinforcement Learning

Technical Analysis "Diffusion steering via reinforcement learning (DSRL)" adapts the BC policy by running RL over its latent-noise space. High-level Analogy: Imagine you have a highly skilled robot artist (the pre-trained diffusion policy) that can generate many different actions, like painting various scenes. This artist is excellent, but sometimes its default 'inspiration' leads to actions that aren't quite perfect for a new, specific task, like precisely stacking blocks. Traditionally, to make the artist better at this new task, you'd have to try and re-train their fundamental artistic techniques or tools (finetune the policy's internal weights with complex RL), which is very difficult, slow, and might even break their existing skills. DSRL, our new approach, is like giving the robot artist a special 'mood dial' for its initial creative thought. This dial doesn't change how the artist paints, but it subtly adjusts their initial artistic spark or 'inspiration' (the latent noise, 'w'). By learning to turn this dial just right (using a small, separate RL policy), you can gently 'steer' the artist to produce exactly the actions needed for the new task, without ever touching the artist's core painting skills or messy tools. You're influencing the beginning of its creative process, rather than trying to fix the artwork mid-way or re-teach the artist from scratch. Even cooler, sometimes different initial 'moods' (different settings of 'w') can lead to very similar paintings. Our…