Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Technical Analysis "Q-Planning" equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. High-level Analogy: Imagine a talented chef who learns to cook by precisely following amazing recipes from a master chef (this is like a robot's 'Behavior Cloning' policy, or BC). The chef is great at following instructions, but if a dish goes wrong, they don't know why or how to fix it without getting a new recipe from the master. Now, imagine this chef also has a super-sensitive 'taste tester' who can evaluate any dish the chef makes, even if it's a disaster. This taste tester (our 'Q-function') doesn't tell the chef how to cook, but rather how good a dish (or a sequence of cooking steps) is. With Q-Planning, the chef keeps their original, excellent recipes (the BC policy) untouched. They try different variations of their recipes, and the taste tester quickly gives each variation a score. The chef then combines these variations, leaning towards the higher-scoring ones. Crucially, the taste tester learns from every…