UniWALM uses a mixture-of-transformers backbone with separate diffusion heads for video, action and language.
Generated language gives robot policies a working memory
UniWALM jointly generates video, actions and task-state text, then reuses its own text to guide later control steps.
Big Tech
Rahul Sajnani · Diego Romeres · Mingyuan Zhang · Kai Gao · Carl Qi · Baichuan Huang · +6 more
Brown University · Amazon · Northeastern University · UT Austin · University of Washington
Research Digest··2 min read
Sajnani et al.
§