Generative media

Generating images, video, 3D and audio: diffusion, flow matching, editing, generative pipelines

273 articles · page 4 of 6

Generative media

Diagnose-to-Generate aligns flow models with localized structural corrections

Li et al. introduce Diagnose-to-Generate (D2G), a post-training alignment framework that separates structural diagnosis from generative optimization. A dedicated diagnosis model, ARGUS, outputs anomaly categories and bounding boxes, which become confidence-weighted spatial rewards for patch-level optimization. On FLUX.1-dev, structural fidelity rises from 62.16 to 85.60 while FID-30K stays nearly unchanged, and independent evaluators confirm the gains.

·2 min
Generative media

Adaptive feedback targets each image model’s changing weaknesses

Zhang et al. present APT, an automated post-training loop in which a text-to-image model’s evaluation results determine what it trains on next. Across multiple models, architectures and benchmarks, the authors report that this adaptive approach improved performance more than reinforcement learning with a handcrafted training set, while data generated for one model transferred less effectively to another.

·3 min
Generative media

Distilling geometry attention into video diffusion models improves long-range consistency

The authors propose Geometry Attention Distillation (GAD), a training-time method that distills cross-frame correspondence targets from a frozen geometry model into video diffusion attention. Across three video backbones, GAD improves long-range scene consistency and dynamic temporal consistency, including at least 30% relative gains in cross-frame matching accuracy on Wan2.2 5B, without adding inference-time modules.

·3 min
Generative media

Sparse aerial photos become persistent, agent-editable 3D city models

Sun et al. introduce CityCode, an agent-operable 3D city representation, and Air2Code, a pipeline that constructs it from a single low-altitude aerial RGB image per region plus known camera parameters. The work demonstrates that recovered metric entities can serve as a stable interface connecting visual evidence, 3D assets, and long-horizon spatial operations, with all 2,500 steps across 50 task chains passing their documented task checks.

·3 min
Generative media

Multi-agent pipeline with simulator feedback produces physically usable 3D assets

Artisan generates articulated 3D objects from text or reference images by delegating design, coding, and validation to independent agents, with feedback from a physics simulator. Across a benchmark covering appearance, physical plausibility, and downstream robotic tasks, Artisan achieves the lowest collision rates and highest stability rates among evaluated methods while keeping spatial fidelity competitive.

·3 min
Generative media

Single-pass GAN matches diffusion quality for real-time talking heads at 43 ms latency

Authors from Meta Reality Labs present FaceGAN, a single-pass GAN for audio-driven facial animation that runs at 43 ms latency and 12x real-time on one GPU. They identify that a causal generator driven by i.i.d. noise cannot produce smooth motion without collapsing innovation variance, and introduce acausal noise shaping to dissolve this limitation without adding audio latency. FaceGAN matches or surpasses diffusion models in fidelity while avoiding their iterative sampling cost.

·2 min
Generative media

Distribution matching distillation and drifting models are equivalent under finite-sample analysis

The authors prove an exact finite-sample correspondence between Distribution Matching Distillation (DMD) and Drifting Models, two approaches for one-step generative modeling. By converting the velocity-field from pretrained diffusion models into attraction force fields and estimating repulsion from the generative distribution, they show that DMD's optimization is equivalent to drifting model training. They then introduce Multi-Bandwidth DMD (MBDMD), which improves convergence and generation quality on image datasets.

·2 min
Generative media

Decomposing video editing into keyframe editing and motion-guided interpolation

The authors propose AVE, which first edits a sparse set of keyframes using a strong image editor, then generates the full video via motion-guided image-to-video diffusion, treating edited keyframes as anchors. On IVEBench and VIE-Bench, AVE achieves state-of-the-art performance in instruction following, temporal consistency, and content fidelity, with the quality strongly tied to the image editor used.

·2 min
Generative media

Causal video diffusion matches bidirectional quality without a teacher model

The authors propose Conditional Residual Prediction (CRP), a training method that forces a causal video diffusion model to first predict each frame without using past frames, then let history add only a residual. In controlled experiments, this nearly closes the 6.14-point VBench quality gap between causal and bidirectional models of the same size. Scaling this recipe, they train Optica, a 2B-parameter causal model that generates 5-second 480p videos and scores 82.78 on VBench using only about 15 million training videos.

·3 min
Generative media

Interaction-aware sparse attention and timestep-adaptive sparsity accelerate video generation

Han et al. propose iCATS, a training-free sparse attention framework for Diffusion Transformers that improves importance estimation by clustering tokens based on query-key dot-product interactions (reformulated as a quadratic form) and adapts sparsity using a signal-to-noise ratio (SNR) guided schedule. On HunyuanVideo-T2V-13B, iCATS achieves 2.03× acceleration at 31.017 dB PSNR, and on Wan2.1-T2V-14B, 1.55× at 29.301 dB PSNR, outperforming prior training-free methods.

·2 min
Generative media

Polynomial student paths improve four-step video distillation quality

Feng and colleagues introduce Parametric Trajectory Distillation (PTD), a method for compressing iterative video generation into four model evaluations. Rather than forcing the student to reproduce difficult teacher transitions directly, PTD represents each trajectory segment as a polynomial and obtains teacher guidance along the student’s predicted path. Across two large video backbones, the authors report better naturalness and diversity than strong few-step baselines.

·3 min
Generative media

Canvas instructions enable precise image editing without separate text prompts.

The authors introduce VibeEdit, an image editor that follows canvas instructions—circles, scribbles, arrows, and optional short notes placed directly on an image—to perform object addition, removal, replacement, attribute change, and movement. They train the model on 1.55 million edit pairs using layer-decoupled conditioning and rubric-guided reinforcement learning. On a 419-case benchmark emphasizing target selection among similar objects, VibeEdit achieves a VLM rubric score of 79.9 and an outside-region PSNR of 32.8 dB, substantially exceeding the best text-instructed baseline (FireRed: 67.4 and 24.0 dB).

·3 min
Generative media

Adaptive tokenization cuts compute for pixel-space diffusion models by up to 75% with no fidelity loss.

Kara et al. introduce BudgetPix, a compute-adaptive tokenization framework for pixel-space diffusion models that allocates tokens based on visual complexity rather than a uniform grid. On text-to-image generation, BudgetPix matches the fidelity of baselines MiniT2I-L at 512² and PixelDiT at 1024² using only 25% of the original token budget. For class-conditional generation with a MeanFlow backbone, the method produces near-zero quality degradation at 60% of the full compute budget.

·3 min
Generative media

Image-based visual examples enable powerful video editing without large paired video datasets

The authors propose VINCIE-NExT, a video editing framework that leverages in-context visual demonstrations from image editing pairs to guide video edits. By decomposing video editing into a chain of sub-tasks that route through the image domain, the model can be trained jointly on abundant image editing data and video data. On the OpenVE-Bench benchmark, VINCIE-NExT achieves state-of-the-art performance across various editing types without requiring large-scale paired video editing data.

·3 min
Generative media

Parallel adapter composition yields real-time joint audio-video generation without chained training

Li et al. demonstrate that block-causal attention and few-step sampling can be acquired independently and combined by simple weight addition at inference, avoiding the sequential distillation typically required. The resulting model generates joint audio-video at about 26 fps at 480x832 on six GPUs, matching the bidirectional teacher on appearance and outperforming one chained baseline on most metrics.

·2 min
Generative media

Recursive latent reasoner boosts diffusion models for visual puzzles

The authors propose Painter-Thinker (PaTh), which couples a small recursive reasoner (the Thinker) to a frozen diffusion model (the Painter) via ControlNet adapters. The Thinker refines a latent state within each denoising step, enabling iterative constraint satisfaction from pixels without symbolic supervision. On visual reasoning tasks, PaTh dramatically outperforms standard diffusion models and prior specialized methods, with margins that widen as problems become harder.

·3 min
Generative media

Generating camera trajectories from a single panorama and text using geometry-grounded pose tokens

The authors introduce OmniCam, an autoregressive model that generates camera pose sequences from a single panoramic image and a textual trajectory description. The model uses a panoramic point-cloud encoder, a hybrid tokenization scheme (absolute rotations with sign-consistent quaternions and relative translations), and separate geometric and semantic conditioning streams with a 3D target anchor. On the new OmniCaT dataset of 267,700 trajectories across four camera behaviors, OmniCam reduces average trajectory error by 43.0% and collision rate by 62.3% compared to the best baseline.

·3 min
Generative media

Explicit context constraints improve interactive video story continuations

Li and colleagues study how a video generator should respond when viewers redirect a story after part of it has already been shown. Their VIS-Ground framework explicitly models dependencies among the grounding source, the rendered video and the new request, improving a composite measure of grounded continuation by an average of 10.3 points over the strongest baseline for each of three generation backbones.

·2 min
Generative media

Data-free distillation of diffusion models with a single consistent objective

The authors propose Consistent Distribution Matching Distillation (CDMD), a method that uses only a frozen teacher and a trainable student with one objective, avoiding distillation datasets, teacher rollouts, and auxiliary networks. On ImageNet 256x256, CDMD achieves an FID of 2.04 with a single function evaluation and 1.37 with four evaluations after 40 epochs of training, surpassing prior data-free distillation baselines.

·2 min
Generative media

Learned control coordinates frozen diffusion models into multi-agent generators

The authors propose Coordinated Multi-Agent Diffusion Steering (CMDS), which treats independently trained diffusion models as fixed generative primitives and learns a control that steers their joint reverse processes toward high-reward assembled outputs. They show that this amortised control recovers known targets, satisfies spatial constraints, separates degraded mixtures, and coordinates agents in maze navigation, robot planning, and human motion synthesis.

·3 min
Generative media

Splitting diffusion distillation into two phase-specific half-sized experts beats a full-size student.

The authors propose Phase-wise Velocity Distillation (PVD), which splits the diffusion generation process into a structural phase and a refinement phase, each handled by a half-sized expert. Using two half-sized phase-specific experts outperforms a single full-size monolithic student under the same one-forward-pass compute budget. PVD achieves FID 1.48 on ImageNet 256x256 and competitive results on T2I benchmarks while reducing active parameters by about 50%.

·2 min
Generative media

New estimator achieves optimal sample complexity for FID evaluation

The authors provide a comprehensive theoretical analysis of the sample complexity of estimating the Fréchet Inception Distance (FID), a standard metric for evaluating generative models. They introduce a new estimator, Relative Taylor Debiasing (RTD), which achieves the minimax optimal sample complexity of O(d/ε²) for d-dimensional image embeddings, proving it is impossible to improve upon. Their experiments on synthetic Gaussians and ImageNet confirm that RTD reduces estimation error, and a second-order variance-aware estimator (VALE₂) matches the accuracy of the prior state-of-the-art with only 10,000 samples instead of 50,000.

·3 min
Generative media

Loop-specific targets let diffusion transformer improve with extra inference loops

Derakhshani et al. introduce LiFT, a looped generative model where each recurrent step is trained toward a distinct point on a straight path from the initial estimate to the flow-matching target. This allows the model to loop far beyond its training depth at inference, improving generation quality without retraining or parameter increase. On ImageNet 256x256, LiFT-L/2 achieves an FID 3.34 points lower than a dense DiT-XL/2 baseline while using about 60% fewer parameters and 52% fewer inference FLOPs.

·3 min
Generative media

RL-trained cache scheduler adapts video diffusion to user-specified speedups

Authors propose MORCA, a cache scheduling framework that treats cache decisions as a Markov decision process and optimizes them via reinforcement learning. By focusing on terminal error — the quality loss in the final generated video — rather than per-step error, and by allowing users to specify an exact acceleration target, MORCA achieves better speed–quality trade-offs than existing dynamic caching methods on multiple video diffusion models.

·3 min
Generative media

Position-based self-conditioning improves single-stage 3D generation without extra stages

The authors propose Position Forcing, a self-conditioning framework that recovers token positions from predicted clean latents, quantizes them at progressively finer resolutions according to the noise level, and feeds the positional encodings back into a diffusion Transformer. This provides coarse-to-fine spatial guidance, improving generation quality in single-stage 3D generative models and outperforming several multi-stage approaches.

·2 min
Generative media

Test-time method prevents drift in long video generation by keeping caching in-distribution

The paper identifies the KV-provenance problem, where cached key-value entries become out-of-distribution beyond the training horizon, causing drift in autoregressive video generation. The authors propose In-Distribution Forcing (ID-Forcing), a test-time framework that uses self-caching to ensure each chunk is cached without attending to prior KV entries, keeping the rolling window in-distribution. This extends short-horizon models to minute-scale generation while maintaining visual quality and motion dynamics.

·2 min
Generative media

Symmetry-regularized flow matching enables video generation from any camera pose without ground truth novel views

The authors introduce SymRegFlow, a flow-matching framework that generates multi-view-consistent video from continuous camera poses without requiring paired novel-view RGB supervision. By geometrically warping source views to target poses and applying masked dual-anchor supervision with cross-anchor consistency, they mitigate errors from imperfect 3D geometry. On the nuScenes benchmark, SymRegFlow achieves the lowest FVD and FVMD among evaluated methods, with over 31% reduction in FVD relative to the best baseline.

·2 min
Generative media

SatisDive method navigates the satisfaction-diversity frontier for text-to-image generation

The authors formulate text-to-image batch generation as satisficing: each image must meet a reward floor and the batch must meet a diversity cutoff. Varying the reward floor traces a Pareto frontier between worst-image reward and batch diversity. SatisDive, a training-free method, traverses this frontier and improves worst-candidate reward over FK steering by up to 0.43 and 0.70 on FLUX and SANA at matched DreamSim diversity.

·2 min
Generative media

Video RL improves by assigning credit to individual visual tokens

The authors introduce TVRL, a framework that derives token-level credit for video reinforcement learning from the same reward signal. By using gradient magnitudes from a frozen vision-language model's question-answer likelihood, they reweight denoising log-probabilities, leading to a 3.60 point improvement on VBench-2.0 Overall score over the base model and consistent gains across multiple samplers and reward models.

·3 min
Generative media

Generating B-roll footage grounded in a user's captured video collection.

The authors introduce MemComposer for collection-grounded B-roll generation. Given a user's video collection, a natural-language directive, and a target duration, the system builds a structured memory of characters, settings, objects, and style, then retrieves conditioning frames and generates a multi-shot sequence. In a user preference study, MemComposer significantly outperforms ungrounded text-to-video planners on both prompt adherence and visual alignment to the collection.

·3 min
Generative media

Distributional distillation cuts diffusion language model sampling steps by half or more

The authors propose two distributional distillation methods for continuous diffusion language models that significantly reduce the number of network evaluations (NFEs) needed for high-quality generation. Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. On OpenWebText with 1024-token sequences, Simplex-DMD achieves a generative perplexity of 45.6 in just 4 NFEs, a 49% reduction relative to the strongest diffusion baseline at matched entropy and budget; Reinforce-DMD reaches 14.9 at 256 NFEs, a 20% reduction.

·3 min
Generative media

ALIVE inserts objects that respond coherently to video actions

The authors introduce ALIVE, a framework for inserting objects into videos so that they participate in ongoing interactions, such as being lifted or cut. They build a large curated dataset and a diffusion editor that uses a vision-language model to predict temporal interaction prompts. On the ALIVE-interaction benchmark, ALIVE improves overall scores by 43.9% over the strongest baseline, and VLM-predicted guidance adds further gains.

·1 min
Generative media

Geometry-aware alignment keeps diffusion models near natural image structure

Wang et al. argue that standard preference optimization can gradually push diffusion models away from the low-dimensional structure associated with natural images, reducing quality and diversity. Their proposed method, Anisotropic Geometry-Aware Preference Optimization (APO), adjusts regularization by direction and reportedly beats several alignment methods with average win rates above 60 percent across multiple benchmarks.

·3 min