RL Token: Bootstrapping Online RL with Vision-Language-Action Models

Technical Analysis "RLToken" is a lightweight method that enables sample efficient online RL fine-tuning of pretrained VLAs using just a few hours of real-world practice. High-level Analogy: Imagine you have a highly skilled, experienced master chef (that's the VLA model). This chef knows how to cook thousands of dishes (perform diverse robot tasks) pretty well, based on years of watching others (huge datasets). However, when it comes to one extremely delicate and precise part of a recipe, like perfectly decorating a tiny cake with intricate details, the master chef might be a bit slow or need a few tries to get it absolutely flawless. You want to train a new, super-fast apprentice (that's our lightweight RL policy) to master just this one tricky decoration part. You don't want to re-teach the apprentice everything the master chef knows from scratch. That would take forever! Instead, the master chef quickly creates a special "cheat sheet" (the RL Token). This cheat sheet isn't the whole cookbook, but a super-condensed summary of only the most critical insights the master chef uses for that specific decoration. The apprentice uses this cheat sheet and also watches the master chef's suggested sequence of moves for the decoration (the VLA's action chunk). The apprentice then practices, getting feedback (rewards). They don't try to invent completely new decoration techniques; instead, they tweak and refine the master's suggestions, focusing on making them faster and more…