Fri 13 Nov
LiveGet it free
ZOTPAPER

News without the noise

← All concepts

unified multimodal understanding and generation

Imagine a single AI model that can both understand and create different types of information, like pictures and words, all at once. Traditionally, one model might be great at captioning an image while another excels at drawing from a description. Unified multimodal understanding and generation combines both abilities into one system: the same model can look at a photo and describe it, or take your sentence and generate a matching image, often without needing separate components for each task.

The papers show that by carefully designing training and shared representations, these unified models can match or exceed systems built from separate parts. A key finding is that treating modalities symmetrically, for example, adding noise to both images and text during generation, improves joint tasks like solving a maze from a description. Several works also highlight that using language as an in-loop working memory, rather than a separate reasoning module, boosts performance on tasks requiring memory and planning, such as long image sequences or bimanual robot control. This approach reduces complexity while enabling new capabilities like coupled generation and editing.

Papers
28
Released code
2
First seen
Mar 2026
Latest
Nov 2026

25 papers in the last two months, against 1 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1AI models can stop drawing something while still recognizing it

    UniForget helps unified multimodal models forget target concepts in both image generation and visual recognition better than single-pathway methods.

    Nov 2026
  2. 2Motion model understands and generates in one shared space

    Möbius unifies motion understanding and generation by diffusing noisy motion directly in the same multimodal space as text.

    Nov 2026
  3. 3AI models can improve image understanding and generation from their outputs

    Deciding per sample whether text or image is the more reliable half of a self-generated multimodal trajectory lets a unified model improve both understanding and generation without outside supervision.

    Nov 2026

How fast it is moving

JunJulAugSeptOctNov so far

Often studied with

flow matchingA generative training method that learns an ordinary differential equation mapping noise to data by matching velocity fields along probability paths.7 togethervision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.6 togetherworld model for agentsA predictive environment model that agents use to simulate outcomes, enabling planning, reasoning, and long-horizon task execution.4 together

Who is working on it

The Hong Kong University of Science and Technology 3Tsinghua University 2Tencent Hy 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.

unified multimodal understanding and generation · Research concepts | Zotpaper