Latent Action Model
A ViT encoder reads consecutive frames (oi, oi+1) and outputs a continuous latent action zi, L2-normalized on the unit sphere—an inverse-dynamics representation of the transition.
: Learning Continuous Latent Action World Models via Adversarial Latent Regularization
1University of Chicago ·
2Toyota Technological Institute at Chicago ·
3Argonne National Laboratory
*Equal contribution
TL;DR
CLAW jointly learns continuous latent actions and a diffusion world model from action-free video using adversarial latent regularization.
Method
A Latent Action Model infers actions from frame pairs; a diffusion world model predicts futures from the initial frame and those latents. We introduce a gradient-reversal branch to adversarially regularize latent actions, resulting in strong disentangling of actions from observations.
Diffusion world model
A UViT diffusion model denoises future frames conditioned on the initial frame and inferred latents. A causal attention mask keeps noised targets from seeing future actions or observations.
Gradient-reversal regularization
The world model runs two shared-weight paths—(oi, zi) and zi alone. A gradient-reversal layer on the z-only path blocks latents from encoding future pixels while the standard path keeps transition structure.
What you can do with CLAW
We evaluate the same pretrained model on the following settings without retraining the world model.
Imitation learning from observation
Behavior cloning in latent space from expert videos, with nearest-neighbor grounding to labeled transitions at execution time.
42% close drawer on OGBench (vs. 29% AdaWorld)
Goal-directed visual planning
MPC over planned latent-action sequences in the learned dynamics model, with nearest-neighbor execution on tabletop manipulation benchmarks.
43% RoboDesk upright block (vs. 5% AdaWorld)
Latent Action Policy Learning
Policy pretraining on inferred latent actions from large-scale human and robot video, with finetuning on a small number of real-robot demonstrations.
1.7/3.0 long horizon (vs. 1.5 AdaWorld)
Visual Planning
On VP2 tabletop manipulation, CLAW plans latent-action sequences with MPC and executes them via nearest-neighbor retrieval from a small labeled reference set.
Latent Action Transfer
Latent actions from a source video are rolled out on a new scene. Gripper motion transfers faithfully; object motion adapts to the target—showing disentangled, reusable actions.
Latent Action Retrieval
On Robosuite, nearest neighbors in CLAW's latent space match end-effector motion, not just scene appearance.
CLAW
AdaWorld
LAPO
Real-World Latent Action Policy
On a tabletop UR5 arm, we pretrain an ACT policy on latent actions inferred from action-free human and robot play video, then finetune on a small set of teleoperated demonstrations—without retraining the world model.
Evaluation
Robotics-focused evaluation on VP2 (Robosuite, RoboDesk), OGBench UR5 manipulation, and a real-world UR5 arm—trained only on task-agnostic play video. Additional game benchmarks (Crafter, Procgen) are in the paper.
BibTeX
Citation
@article{ayalew2026clawlearningcontinuouslatent,
title={{CLAW}: Learning Continuous Latent Action World Models via Adversarial Latent Regularization},
author={Tewodros Ayalew and Matthew Jeung and Samuel Wheeler and Xiao Zhang and Andre de la Cruz Arce and Kaylene Stocking and Michael Maire and Matthew R. Walter},
year={2026},
journal={arXiv preprint arXiv:2606.04130},
}