Latent Action Model
A ViT encoder reads consecutive frames (oi, oi+1) and outputs a continuous latent action zi, L2-normalized on the unit sphere: an inverse-dynamics representation of the transition.
: Learning Continuous Latent Action World Models via Adversarial Latent Regularization
1University of Chicago ·
2Toyota Technological Institute at Chicago ·
3Argonne National Laboratory
*Equal contribution
TL;DR
CLAW jointly learns continuous latent actions and a diffusion world model from action-free video using adversarial latent regularization.
Method
A Latent Action Model infers actions from frame pairs; a diffusion world model predicts futures from the initial frame and those latents. We introduce a gradient-reversal branch to adversarially regularize latent actions, resulting in strong disentangling of actions from observations.
Diffusion world model
A UViT diffusion model denoises future frames conditioned on the initial frame and inferred latents. A causal attention mask keeps noised targets from seeing future actions or observations.
Gradient-reversal regularization
The world model runs two shared-weight paths: (oi, zi) and zi alone. A gradient-reversal layer on the z-only path blocks latents from encoding future pixels while the standard path keeps transition structure.
What you can do with CLAW
We evaluate the same pretrained model on the following settings without retraining the world model.
Imitation learning from observation
Behavior cloning in latent space from expert videos, with nearest-neighbor grounding to labeled transitions at execution time.
42% close drawer on OGBench (vs. 29% AdaWorld)
Goal-directed visual planning
MPC over planned latent-action sequences in the learned dynamics model, with nearest-neighbor execution on tabletop manipulation benchmarks.
43% RoboDesk upright block (vs. 5% AdaWorld)
Latent Action Policy Learning
Policy pretraining on inferred latent actions from human video (Something-Something V2), then finetuning on a small set of real-robot demos.
Best progress on both real-world tasks
Visual Planning
On VP2 tabletop manipulation, CLAW plans latent-action sequences with MPC and executes them via nearest-neighbor retrieval from a small labeled reference set.
Latent Action Transfer
Latent actions from a source video are rolled out on a new scene. Gripper motion transfers faithfully; object motion adapts to the target, showing disentangled, reusable actions.
Latent Action Retrieval
On Robosuite, nearest neighbors in CLAW's latent space match end-effector motion, not just scene appearance.
CLAW
AdaWorld
LAPO
Latent Action Structure
On Crafter, UMAP embeddings of latent actions (colored by ground-truth labels for visualization only) show that CLAW organizes transitions into locally action-consistent regions, without action supervision or a discrete codebook.
Real-World Latent Action Policy
World model pretrained on Something-Something V2 (human video only). An ACT policy is pretrained on inferred latent actions, then finetuned on 50 teleoperated demos per task on a tabletop UR5.
Same architecture and finetuning budget; only the latent-action pretraining signal differs.
Limitations
Like other video world models, CLAW learns what transitions look like, not which ones the environment permits, so it can generate futures that would never occur.
Evaluation
Robotics-focused evaluation on VP2 (Robosuite, RoboDesk), OGBench UR5 manipulation, and a real-world UR5 arm, trained only on task-agnostic play / human video. Additional game benchmarks (Crafter, Procgen) are in the paper.
BibTeX
Citation
@article{ayalew2026clawlearningcontinuouslatent,
title={{CLAW}: Learning Continuous Latent Action World Models via Adversarial Latent Regularization},
author={Tewodros Ayalew and Matthew Jeung and Samuel Wheeler and Xiao Zhang and Andre de la Cruz Arce and Kaylene Stocking and Michael Maire and Matthew R. Walter},
year={2026},
journal={arXiv preprint arXiv:2606.04130},
}