Modality-Autoregressive World-Action Models

Technical Analysis is the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. High-level Analogy: Imagine you're an artist trying to paint a complex scene. Instead of trying to paint everything at once, you approach it step-by-step. First, you sketch the basic outlines and movements (like 'point tracks'). Then, you add the main objects and their meaning ('DINO features'). Next, you give everything a sense of three-dimensional space and structure ('depth maps'). Finally, you might consider adding fine-grained colors and textures ('RGB images'), and then, based on this complete mental picture, you decide where to put your brush next ('actions'). Each step builds on the clearer picture formed by the previous ones, making the final decision more informed and accurate. Motivation of the Work The core motivation for this research is to improve how robots learn to interact with the world by making their 'world models' more effective. While WAMs are powerful, their reliance on predicting only RGB images limits their efficiency and effectiveness in capturing critical manipulation-relevant features like object geometry, movement, and semantic meaning. Different visual 'modalities' (like depth or DINO features) offer unique advantages. The key question is: how can WAMs intelligently combine these multiple modalities to leverage their strengths? The authors also wanted to…