Generative media

Generating images, video, 3D and audio: diffusion, flow matching, editing, generative pipelines

273 articles · page 5 of 6

Generative media

Variable-size tokens cut diffusion costs where less detail is needed

Nakayama et al. introduce Level-of-Token Diffusion, which assigns smaller tokens to detail-sensitive regions and larger tokens to simpler areas such as skies, flat surfaces or blurred backgrounds. By shortening the sequence processed by a diffusion transformer while retaining full-resolution denoising updates, the method achieved reported average speedups of up to 2.04 times for images and 3.53 times for videos.

Generative media

Steepest Guidance aligns generative models by maximizing local reward improvement

Takakura et al. treat inference-time alignment of flow and diffusion models as sequential optimization in probability measure space. They introduce Steepest Guidance, which steers the generative process along the steepest ascent direction of the reward functional. The method achieves unbiased estimation for linear rewards, handles non-linear rewards via particle approximation, and consistently outperforms existing training-free approaches.

Generative media

Spatial resolution hierarchy enables parallel mesh generation with explicit topology

The authors propose MeshOctave, a discrete diffusion model that generates artist-quality meshes by defining scale through spatial grid resolution. Coarsening merges vertices in the same voxel (resolution collapse), and refinement is factorized into per-face decisions about octant children and connectivity (split-and-rewire). This yields parallel intra-scale generation and dynamic-length meshes, achieving better geometric fidelity and topological validity than strong baselines.

Generative media

Splitting experts by role improves video diffusion model scaling

The authors propose SplitMoE, a video diffusion architecture that bifurcates the expert pool into semantic experts (for high-level abstraction) and generic experts (for residual visual details). Using prototype-guided routing and pull-push regularization, they avoid the uniformity trap of standard MoE—where load-balancing scatters coherent patches across experts—achieving faster convergence, lower fragmentation, and better generation fidelity on video benchmarks.

Generative media

Cached token selection narrows diffusion transformers’ sparse attention quality gap

Chen et al. analyze why sparse attention loses fidelity in diffusion transformers, separating errors caused by coarse token grouping, approximate selection and discarded attention mass. Their training-free MC-Sparse method addresses all three and reports denoising speedups of 1.80 times for video and 2.32 times for 3D generation, with negligible measured quality loss relative to dense attention.

Generative media

Explicit weather transitions improve control in generated future-scene videos

Wu and colleagues present a video world model that generates future views from a single sunny or adverse-weather image while separately controlling camera motion and weather. Their central result is that explicitly representing the change from observed to requested weather produces more accurate weather control than asking a generation backbone to infer that change implicitly, while maintaining competitive scene consistency and camera adherence.

Generative media

GeoVerse blends geometric latent diffusion with video generative priors for consistent novel views

The authors propose GeoVerse, a framework that synthesizes world-consistent novel views by combining a geometric latent diffusion model with video generative priors from Wan2.2 VACE. Using a ControlNet-style adapter and a global point-cloud memory for cross-view coherence, the method achieves a 2.23 dB higher PSNR on DL3DV and 32.4% lower Absolute Trajectory Error on Mip-NeRF360 compared to the prior state-of-the-art GLD, while reducing inference time to under 10 seconds via 4-step reflow distillation.

Generative media

Verifiable visual rewards let image generators train on synthetic scenes and transfer to natural prompts

Li et al. introduce Verifiable Visual Rewards (VVR), a framework where image generation tasks are scored by deterministic program verifiers rather than learned detectors or vision-language models. Training with these rewards raises Stable Diffusion 3.5 Medium's accuracy on the VVRBench benchmark from 2.8% to 28.3%, and the gains transfer to out-of-domain natural prompts.

Generative media

Weighting schedules in score-based models determine which data features are learned and at what rate

The authors analyze the training dynamics of score-based generative models on multimodal Gaussian mixtures. They decompose the time-integrated loss and find that the signal-to-noise ratio at a given training time dictates which features are learned: at high SNR only mode directions are acquired, while near the speciation time (SNR ~ 1) both directions and relative weights become learnable. The weighting schedule determines how much of the training loss is concentrated near that critical region, thereby governing the learning timescales of different features.

Generative media

A Stackelberg game framework lets discrete diffusion learn which token corruptions most improve the denoiser

Discrete diffusion models can revise earlier predictions during generation, but their performance depends on the fixed forward noise process. Moor et al. introduce VSDD, a variational Stackelberg game that learns a semantically aware corruption process by optimizing corruptions based on how much the denoiser improves after training on them. VSDD improves molecular validity over uniform and masked diffusion, reduces text perplexity relative to uniform diffusion while remaining competitive with masked diffusion, and achieves sizable gains in offline playlist metrics.

Generative media

Framework guides discrete diffusion with sequence-level objectives without exponential enumeration

The authors introduce COFFEE, a plug-and-play framework that avoids exponentially enumerating all possible completions when guiding discrete diffusion models with sequence-level objectives. A target-free carrier captures joint token dependencies from the denoiser, while a compiled finite-state model records how token combinations affect preferences. Pairing their states allows polynomial-time sampling of guided reconstructions.

Generative media

Concept entanglement forces a fundamental trade-off in diffusion model unlearning

The paper shows that the failure of concept unlearning methods—either leaking the target under indirect prompts or degrading related concepts—is not a bug of individual algorithms but a structural consequence of overlapping concept representations in activation space. The authors derive a theorem (Theorem 4.6) that for any robust erasure method, the preservation error on neighboring concepts is at least linearly proportional to the overlap between target and neighbor concepts, and confirm this trade-off empirically across thirteen methods on Stable Diffusion v1.4 and SDXL.

Generative media

Structured residual connections improve diffusion transformers by enabling adaptive cross-layer information retrieval

Liu et al. systematically analyze representation dynamics in Diffusion Transformers (DiTs) and find a latent preference for early-layer feature reuse and symmetric layer guidance. They introduce structured connectivity that allows each transformer block to dynamically retrieve spatial and semantic cues from earlier layers via differentiable cross-depth paths. This design yields up to 1.73x fewer training iterations to reach baseline FID and improves a strong REPA-XL/2 model from 5.9 to 4.34 FID without classifier-free guidance, and to 1.39 FID with guidance, all with less than 0.1% parameter overhead.

Generative media

Spectral hybrid parameterizations robustly accelerate flow matching optimization

The paper studies why different prediction targets (x1, v, x0) in flow matching yield different performances, and identifies two drivers: signal-to-noise ratio per data covariance direction and neural architecture bottleneck. The authors propose spectral hybrid parameterizations that adapt across time and directions, showing they are optimal for Gaussian data and robust across architectures and source scales, accelerating optimization at no extra training cost beyond a one-time PCA.

Generative media

Semantic token prefixes let smaller autoregressive video models match larger ones

Dereviannykh et al. introduce SemanTok, a flexible-length video tokenizer that aligns coarse token prefixes with frozen DINOv2 features through encoder input injection and auxiliary reconstruction heads. Across seven autoregressive model sizes and two video datasets, SemanTok improves semantic alignment and fidelity at every budget, with a 201M model matching a VideoFlexTok model 3.4 times its size.

Generative media

Large-to-small diffusion model distillation fails due to CFG error amplification, new method resolves it

Zhang et al. investigate why standard on-policy distillation (OPD) fails when transferring a large diffusion model to a smaller student. They introduce Fixed-State KL to measure the distribution gap and find that classifier-free guidance (CFG) accumulates and amplifies discrepancies between the student's conditional and unconditional branches. Their proposed method, GFD-OPD, folds the teacher's guided policy into the student, eliminating artifacts and achieving state-of-the-art results on all benchmarks.

Generative media

Text queries retrieve SAE features for steering diffusion transformers

The authors introduce D-Scope, a framework that retrieves sparse autoencoder (SAE) decoder directions from text queries by aggregating SigLIP 2 embeddings of highly activating image patches into visual centroids. Across 150 SAEs and a 1,000-case benchmark, contrastive retrieval yields larger mean regional target-alignment gains than direct retrieval, reaching up to 86% of an oracle that uses target-generation activations, while showing that reconstruction fidelity does not guarantee visual-evidence coverage.

Generative media

A zero-shot pipeline that turns monocular 360° video generators into stereoscopic ones

The authors present EPIC, a zero-shot method to generate stereoscopic 360° videos from existing monocular panoramic video diffusion models. They create the right-eye view via spherical depth warping and use the model to inpaint missing regions. To enforce geometric consistency, they introduce PEGS, an epipolar residual metric on the viewing sphere, and use it as a preference signal for Direct Preference Optimization, reducing stereo inconsistencies.

Generative media

Adversarial post-training fixes high-frequency deficit in pixel diffusion models

Lin et al. show that adding a hinge adversarial loss to the predicted clean image of pretrained pixel diffusion models corrects a systematic underproduction of natural-image high-frequency content. Across two backbones, this post-training step jointly improves FID, recall, prompt alignment, and perceptual quality, while matched latent diffusion models show no comparable gain, pointing to direct output access as the key enabler.

Generative media

Interactive world model adds synchronized spatial audio to visual simulation

The authors introduce HelixWorld, the first real-time interactive audio-visual world model that co-generates visual scenes and camera-grounded spatial stereo audio under user control. Trained on a new high-fidelity dataset, it sustains drift-free joint rollouts at 24 FPS on a single GPU, matching state-of-the-art silent models in visual fidelity while significantly improving spatial-acoustic alignment.

Generative media

One-step video refiner achieves 8.91x speedup for 4K upscaling

SoL-Refiner is a one-step video refiner that takes low-resolution outputs from various generators and produces high-resolution videos (up to 4K) with a single denoising step. It uses a three-stage training pipeline: high-resolution continual training, reinforcement learning post-training, and one-step distillation. At 2K resolution, it outperforms all external refiners on average VBench and UniPercept scores, and achieves an 8.91x speedup over the three-step LTX-2.3 baseline.

Generative media

Periodic stripe structures in video DiT attention enable efficient sparse computation

The paper reveals that attention layers in video Diffusion Transformers exhibit periodic diagonal stripe structures along temporal and spatial dimensions. The authors propose PSA, a parameterized stripe attention that encodes these patterns into a unified mask, enabling a single hardware-efficient CUDA kernel to handle diverse sparse configurations. This approach yields up to 1.57× speedup on HunyuanVideo with minimal visual quality loss.

Generative media

Denoiser Jacobian spectrum links to generative model quality and enables targeted improvements

The authors analyze the Jacobian of denoising networks in diffusion and flow-matching models, finding that models with larger principal eigenvalues perform better at generating images. They introduce a regularization scheme that perturbs inputs during training to either amplify data-relevant eigen-directions or suppress noisy ones, and show that applying it on ImageNet improves generative metrics, with the two strategies working cumulatively.

Generative media

Evolving physics captions improve video models’ physical plausibility

Lu and colleagues test whether explicit natural-language descriptions can carry enough physical knowledge to improve video world models without requiring a separate numerical or latent representation. Their framework, Physis-Lang, uses assertion-level caption evaluation and language-guided data retrieval, producing consistent benchmark gains and enabling an open Cosmos3-Nano-based model to outperform Veo 3.1 on the reported physical-video evaluations.

Generative media

Reusing Transformer blocks improves image generation without enlarging the model

Chng et al. test whether text-to-image diffusion models can gain computational depth by repeatedly running the same Transformer blocks rather than adding parameters or denoising steps. Their Looped-DiT architecture consistently beats non-looped baselines under matched parameter and compute budgets, and a 260 million-parameter version reportedly outperforms a model 6.5 times larger while using 4.9 times less inference compute.

Generative media

Small feature adapters remove visual domains while preserving scene content

Deixelberger and Steinberger argue that conventional image translators preserve unwanted fog, rain or rendering style because their generators receive representations from which the source appearance can be recovered. Their Representation Feature Adapter instead changes DINO features before a frozen decoder, matching or exceeding much larger translation systems on several benchmarks while requiring substantially less training.

Generative media

A method that generates consistent video observations across multiple vehicles in a shared driving scene

Meng et al. propose CoDrive, a cross-vehicle video generation framework that produces world-consistent observations for multiple vehicles in the same scene. By interleaving local and global self-attention and injecting camera trajectories in a shared coordinate system, the model significantly improves trajectory controllability and cross-agent geometric and instance consistency while maintaining visual quality.

Generative media

Gaussian mixtures improve distributional training for one-step image generators

Zhang, Shi, Liu and colleagues present a common mathematical account of distributional training, where generators learn by matching collections of real and generated features from frozen image encoders. Their resulting method, MGFlow, represents those distributions as Gaussian mixtures and reports stronger one-step generation than prior feature-distribution objectives on ImageNet and text-to-image benchmarks.

Generative media

Keeping token uncertainty improves discrete diffusion generation and fast sampling

Deschenaux et al. introduce Simplex Diffusion Models, which keep intermediate predictions as probability distributions rather than repeatedly collapsing them into categorical token choices. The authors derive closed-form reverse transitions, use ordinary cross-entropy training, and report gains over masked and uniform diffusion on code generation, including after aggressive sampling distillation.