Generative media

Generating images, video, 3D and audio: diffusion, flow matching, editing, generative pipelines

273 articles · page 3 of 6

Generative media

Video sampling needs both model likelihood and external physics guidance

Zheng et al. distinguish between unlikely errors produced during sampling and physically implausible behavior that a video model nevertheless considers likely. They use this decomposition to build HA-Sampler, which adds external physical guidance during latent refinement, and HA-Scorer, which estimates model likelihood to rank generated videos. Tests on Physics-IQ and VideoPhy show improvements over the compared sampling and scoring baselines, although the supplied text does not report effect sizes.

Generative media

Combining compression and adaptive inference for diffusion transformers without quality loss

The authors identify a Combination Gap: directly attaching existing adaptive reuse methods to a compressed diffusion transformer degrades sample quality (FID from 5.56 to 10.40–13.18 on ImageNet 256×256). They propose TamedFlow, which trains a compact DiT-B/2 student via cross-architecture feature alignment, residual velocity correction, and randomized rollout supervision, then deploys a 4M-parameter Learnable Trajectory Estimator to accept reliable student drafts or trigger teacher rescue. On ImageNet 256×256, TamedFlow achieves FID 2.58 at 2.25× measured speedup over the DDIM-50 teacher, and outperforms training-free adaptive baselines at comparable latency.

Generative media

World coordinates replace learned features for game model control

The authors propose ActionRoPE, a parameter-free interface that converts action-driven camera displacement into world-coordinate rotary positions for planar-camera game world models. This approach reduces revisit drift by 66% over the strongest baseline and generalizes across model sizes and out-of-distribution scenes, solving a core mismatch between learned feature conditioning and the geometric structure of top-down and isometric games.

Generative media

Predicting velocity fields helps video models generate more realistic fluids

Zheng, Li, Deng and colleagues target a persistent weakness of video generators: smoke, fire and explosion debris often look detailed but move without coherent turbulence. Their system combines supervised training on Houdini simulations and real-world footage with preference optimization guided by predicted fluid velocity, producing more structured vortices, advection and fine-scale motion than the tested alternatives.

Generative media

Revising skipped diffusion steps with later feedback improves caching efficiency and quality.

The authors propose PoSTCache, a training-free framework that makes diffusion caching postdictive: predictions for skipped steps are first forecast, then revised using discrepancies observed at later full-computation steps. They also introduce Posterior Variance Reduction (PVR), an anchor selection strategy based on offline-profiled cross-timestep dependencies. On representative image and video generation models, PoSTCache improves reconstruction fidelity over existing caching methods at comparable or higher acceleration.

Generative media

Small frozen-model corrector curbs drift in long autoregressive video generation

Wang et al. study exposure bias in autoregressive video diffusion, where each generated chunk becomes imperfect context for the next and quality deteriorates over time. They find that about 95 percent of the resulting denoising error is systematic across random noise seeds, then train a compact correction module that substantially reduces measured drift while leaving the 1.3-billion-parameter base model frozen.

Generative media

Strong text-to-image generation is possible without any language pretraining

Chandrasegaran et al. ask whether the internet-scale language pretraining that dominates text-to-image (T2I) budgets is actually necessary. They introduce Halcyon, a family of early-fusion autoregressive models trained entirely from scratch on GPIC, a dataset of 100M image-text pairs. Halcyon beats stronger, larger baselines that depend on pretrained text encoders, with a fraction of the total pretraining compute.

Generative media

One model combines text, image and video references for editing

The authors present a unified video-editing system that accepts variable combinations of text instructions, images and videos, with each reference controlling attributes such as camera movement, motion, content, style or visual effects. On a new 125-case benchmark, they report better reference adherence and source preservation than task-specific baselines and selected commercial systems, although the provided material does not state performance margins.

Generative media

Pixel diffusion benefits from adapting its prediction target by data direction

Mukhopadhyay et al. examine two limitations of vanilla pixel diffusion transformers: the choice of training target and weak reproduction of fine image structure. Their MFlow method learns a separate interpolation between clean-image and velocity prediction along each principal data direction, adapting to the representation and model capacity without a manual subspace search. A post-hoc ConvRefiner then improves high-frequency statistics while leaving the diffusion backbone frozen.

Generative media

Grounding 3D texture generation in surface-aligned material observations improves fidelity

The authors introduce SUPRA-Tex, a framework for single-image 3D generation that separates material estimation, surface alignment, and geometry-conditioned completion. It projects reference-view PBR maps into the target latent space and uses an interleaved flow transformer to complete the full surface. Trained only on public data, it matches or exceeds commercial systems on in-the-wild images.

Generative media

Feature-trained image tokens improve generation quality at lower sampling cost

Zhang Xiaofeng and colleagues separate token learning from pixel rendering: the tokenizer is trained to reconstruct semantic features, while a detached decoder converts tokens into images without letting pixel losses reshape the representation. Across four pretrained vision encoders, this produced tokens that image generators could model more effectively, improving generation quality at substantially lower sampling compute than evaluated continuous alternatives.

Generative media

Correcting classifier-free guidance in early steps improves image editing fidelity

Floros et al. identify a flaw in how classifier-free guidance (CFG) is applied to text-and-image-to-image (TI2I) flow matching models. They find that CFG's unconditional branch approximates the average of all possible edits, which harms source preservation during the noisy early phase of generation. They propose a simple inference-time correction: substitute a no-edit velocity into the unconditional branch for early flow steps. This intervention recovers a richer alignment-preservation tradeoff without additional model evaluations and is validated on both pixel-space and latent TI2I flows.

Generative media

Two denoising experts make video diffusion distillation far more efficient

Dong and colleagues investigate whether autoregressive video diffusion models really need large collections of precomputed denoising trajectories for distillation. They report that separating high-noise and low-noise denoising into specialized parameter-efficient experts prevents the quality collapse seen with one-trajectory training, matching or slightly exceeding a fully trained baseline while reducing trajectory collection to about one A100 GPU-hour.

Generative media

StreamErase enables real-time streaming video object removal at 26 FPS

The authors present StreamErase, a framework for real-time video object removal that achieves sub-second latency and up to 26.06 FPS on a single H200 GPU. By progressively distilling a bidirectional diffusion teacher into an efficient unidirectional autoregressive student with attention sinks and rolling KV caches, the model enables indefinite-length streaming generation while maintaining visual fidelity. It is over 46× faster than prior methods and generalizes to diverse real-world video scenarios.

Generative media

PathFM generates complete forecast trajectories without sacrificing marginal accuracy

Yao et al. introduce PathFM, a 300-million-parameter time-series foundation model that generates complete multivariate futures rather than separate quantiles for each time step. The model combines conditional flow matching, sampler distillation, attention-cache reuse and direct calibration, achieving competitive marginal forecasts while better preserving dependencies within trajectories.

Generative media

Dual-frame interleaving lets holistic 3D priors produce part-decomposed assets

The authors introduce StructParts, a framework that reuses holistic 3D generation priors for part-decomposed asset generation. By interpreting each shape latent under both a part-local and an object-global coordinate frame, and processing these through a global-local interleaved transformer, the method inherits both per-part geometric detail and whole-object structural coherence from the same pretrained weights. On PartObjaverse-Tiny, StructParts matches or surpasses prior specialized part-decomposed models while requiring substantially less training.

Generative media

Steering input noise aligns few-step generators without retraining them

Shin and colleagues address reward alignment for generators that produce outputs in one or a few evaluations, where conventional reinforcement learning can damage output quality and per-sample search erodes their speed advantage. Their method, Noise Policy Optimization (NPO), shifts optimization into the generator’s initial noise space and reports better reward-quality trade-offs than direct fine-tuning across image, language and protein generation.

Generative media

Object-centric 3D memory enables consistent video generation under large viewpoint changes

The authors introduce Totem, a framework that represents each selected object as a persistent 3D memory in a shared world frame, combining object memories with scene geometry for target-view synthesis. They show that Totem achieves state-of-the-art performance on world-consistency and object-consistency benchmarks, particularly under challenging camera trajectories with occlusions, reappearance, and scale variation.

Generative media

Calibrated depth reward improves 4D consistency without harming visual quality

The authors propose GeoFix, a framework that uses a single geometric reward (FOCAL) to improve geometric consistency and visual quality in 4D generation from video diffusion models. FOCAL measures cross-view depth consistency in static regions and calibrates it against the expected depth agreement for a given camera field-of-view overlap, preventing larger overlap from being mistaken for better geometry. GeoFix achieves a 39% improvement in WorldScore 3D consistency and a 4% increase in VBench Quality Score compared to baselines.

Generative media

One LiDAR model generates, forecasts and resimulates driving scenes in real time

Che and colleagues present LidarVerse, a conditional world model that generates new LiDAR scenes, predicts future point clouds and resimulates recorded environments from new sensor trajectories. By distilling a diffusion teacher into a four-step causal model, they report persistent rollouts at 11 frames per second on one NVIDIA H100 GPU, exceeding the native 10 Hz rate of common automotive LiDAR.

Generative media

Spatial rewards and staged training improve image layout accuracy

Tan et al. address two weaknesses in reinforcement learning for spatial text-to-image generation: unreliable reward signals and training rollouts that attempt complex compositions without first establishing basic skills. Their framework uses an agentic evaluator called SpatialCritic and a curriculum that progresses from foundational spatial relations to combinations of those relations, producing gains on both SD3.5-Medium and BAGEL.

Generative media

Persistent memory keeps generated 4D worlds consistent across camera paths

Zhu and colleagues present Demiurge, a system that turns one monocular video into a navigable 4D representation, meaning a 3D scene that evolves over time. Rather than generating every novel camera path independently, the system follows a generate, recover and update cycle built around persistent geometric memory. The authors report more consistent appearance and geometry across trajectories, including views with extreme parallax.

Generative media

Diffusion transformer depth profiles reflect prediction targets, not just feature granularity

Meng and colleagues re-examine why intermediate features in diffusion transformers improve and then deteriorate on downstream vision benchmarks as network depth increases. Their experiments support a three-phase account centered on whether the model predicts noise, clean images or a mixture, and they use this account to anticipate where REPA representation alignment should be applied.

Generative media

Explicit predictive states boost physical plausibility in video world models

The authors introduce Physical Registry, dedicated slots inside a video Diffusion Transformer that encode predictive dynamics. Initialized by a JEPA predictor (a method that forecasts future representations) and supervised to predict future frame representations, these slots co-evolve with video tokens during denoising. With only 0.45% additional parameters, Physical Registry improves Physics-IQ Verified scores by up to 9.9 points, enabling a smaller model to outperform a larger one.

Generative media

Human-rated clips improve rewards for synchronizing generated sound and video

The authors assembled 80,000 professional synchronization ratings covering outputs from eight audio-visual generators, then built a separate 1,000-clip benchmark with multiple ratings per clip. Their SyncReward model agreed more closely with human judgments than the evaluated specialized metrics and multimodal models, and also improved synchronization when used to select or optimize generated videos.

Generative media

Feedback-driven scheduling beats heuristics for caching in video diffusion acceleration

Jiang et al. present AlignCache, a training-free caching framework for video diffusion models that replaces heuristic skip decisions with a closed-loop controller calibrated against the true influence of caching on the generated video. Across Wan2.1-1.3B, Open-Sora-1.2, and HunyuanVideo, AlignCache reports the best speed-quality Pareto frontier among cache-based accelerators, delivering higher speedup and better fidelity simultaneously.

Generative media

Learning powered conditional distributions directly with importance-weighted flow matching

Falkiewicz et al. introduce Importance-Weighted Conditional Flow Matching (IW-CFM), which uses density ratios from frozen conditional and unconditional flow models to reweight flow-matching examples and directly learn the ω-powered conditional distribution p_ω(x|y). On synthetic distributions, IW-CFM recovers the analytical powered target exactly, while classifier-free guidance (CFG) at the numerically matched scale γ=ω exhibits a visible mismatch. On ImageNet, a lightweight amortized corrector based on IW-CFM aligns more closely with weighted and filtered reference distributions than fixed- and matched-guidance CFG baselines.

Generative media

Visual memory keeps characters and scenes consistent across long image generation trajectories

Peng et al. train a large unified multimodal model natively on interleaved text-image data and introduce a visual-memory framework to scale it to long autoregressive trajectories. The model selectively retrieves relevant historical visual states, keeps persistent references for stable entities, and adapts via few-step self-rollout. On their new LongMIG-Bench, it achieves high-fidelity long-horizon generation with strong character and scene consistency.