Hybrid backbones can efficiently adapt into diffusion language models

Qwen3.5 models combining attention and recurrent layers reached matched training loss with roughly half the tokens required by a full-attention control.

Academic

Anton Xue · Litu Rout · Aditya Akella · Adam Klivans · Sujay Sanghavi · Sanjay Shakkottai

University of Texas at Austin

Research Digest·19 September 2026·2 min read

Thread:Diffusion Language Models

In one line

Hybrid-attention diffusion models from Qwen3.5 adapt faster and decode in any order and in parallel.

Before you read this

Three ideas this paper builds on

diffusion language modelA language model that generates text by iteratively denoising a latent or discrete representation.28 papers →hybrid attentionCombining different attention types (e.g., linear and full) or positional/semantic strategies in one model to balance efficiency, accuracy, and memory.19 papers →parallel decodingA method that generates multiple output tokens simultaneously in a single forward pass to accelerate inference.3 papers →

Read it as

The authors adapt the hybrid-attention Qwen3.5 model (which interleaves attention and RNN layers) into a (DLM) at four scales (0.8B, 2B, 4B, 9B), creating the dQwen3.5 family. They bidirectionalize only the attention layers, leaving the causal RNN layers unchanged, and train with a diffusion objective.

They find that this hybrid backbone reaches a given training loss in roughly half the tokens compared to a full-attention control of similar trunk size. The resulting models exhibit any-order decoding behavior similar to full-attention DLMs and perform well under , with the 9B model outperforming comparable DLMs on several benchmarks despite training on far fewer tokens.

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

diffusion language model hybrid attention AR-to-DLM adaptation any-order decoding parallel decoding large language model