Generative media

Generating images, video, 3D and audio: diffusion, flow matching, editing, generative pipelines

273 articles · page 2 of 6

Generative media

Fast generative models improve with RL when noise split is optimized geometrically

The authors introduce a principled way to choose the stochastic transition policy when applying reinforcement learning to few-step flow generative models. By minimizing the conditional KL divergence for a given velocity update, they derive a closed-form optimal noise split. This results in better policy optimization and higher rewards than existing approaches in both image and video generation.

Generative media

Diffusion models can be composed and steered at inference time without retraining

The authors introduce generalized Feynman-Kac correctors (GFKC), a method for sampling from distributions defined by combining several pretrained diffusion models, with optional reward-based steering. The algorithm operates entirely in the log domain and corrects generation on the fly, requiring no retraining. Experiments on Gaussian models and MNIST confirm accurate sampling from non-trivial combinations and effective steering.

Generative media

AI model controls characters in unfamiliar game worlds in real time

The authors present Wizard, an agentic game world model that uses structured proxies such as skeletons for characters and geometric maps for scenes to guide video generation. An agentic framework maintains explicit game states and updates these proxies in response to player inputs, enabling real-time, fine-grained control over character motions. The system generalizes zero-shot to unseen scenes, characters, and actions, resulting in consistent and visually coherent gameplay across diverse scenarios.

Generative media

Joint video and 3D motion prediction reduces trajectory errors by over half

The authors introduce WorldFlow, a generative world model that jointly predicts future video and 3D trajectories for specified query points. Trained on 1.44 million object-level samples from simulated and real-world data, it significantly outperforms existing methods on motion forecasting benchmarks, particularly on spatial reasoning tasks where it reduces average displacement error by 54.2% relative to the best external baseline.

Generative media

AI image generators and robot controllers can follow guidance more reliably

Clémente and colleagues investigate why techniques for steering diffusion models can become unstable when transferred to flow-matching models, which generate outputs by progressively transforming noise into data. They identify a mismatch with the relationships learned during training and report that correcting it improves sampling quality and guidance stability across robotic control and image restoration tasks.

Generative media

AI models steered toward formal writing can generate more formal clothing

The authors show that linear steering vectors extracted from either text or image representations causally transfer zero-shot to the other modality in unified autoregressive models. Cross-modal steering recovers 84% of intra-modality steering effect from text to images and 74% from images to text in UniAR, with similar or better transfer in Emu3.5 and Liquid. Even modality-specific concepts like formal language induce consistent visual effects (e.g. formal clothing) when transferred.

Generative media

AI models can answer image questions and refine pictures they generate

The authors propose NLD-Uni, a unified discrete diffusion language model for visual understanding and generation. They introduce cross-paradigm on-policy distillation to transfer skills from autoregressive VLMs and continuous flow models, along with a progressive self-refining diffusion process that improves image quality at inference. NLD-Uni achieves state-of-the-art results among unified DLLMs on question-answering, grounding, generation, and editing.

Generative media

Video generators follow user-defined rules better when separate software controls object behavior

The authors propose BoxWorld, a framework for real-time interactive video worlds where user-defined rules control object behaviors. They separate interaction reasoning from generative rendering: a coding agent creates a Box World Engine that applies rules and updates a minimal object-centric state interface, while a Box-State Generative Renderer autoregressively renders the state stream into video. BoxWorld outperforms end-to-end world models across multiple benchmarks and enables fine-grained control over object motion, state switches, and conditional events.

Generative media

AI video generators can follow new commands while remembering earlier actions

Chen and colleagues present Hy-Nomas, a video world model that generates successive scenes in response to free-form text instructions and continuous camera controls. Its central design keeps descriptive context available while limiting action instructions to their intended intervals. The authors report better human-evaluation results than existing world models, although the supplied material gives no scores or evaluation sample sizes.

Generative media

Tuning only the first denoising step restores lost diversity in distilled models

The authors measure how diversity is lost during distribution-matching distillation and find that the largest drop occurs in the first, high-noise denoising step. They introduce ReSeed, which trains small per-channel bias adapters for that step only, aligning the student's velocity with the teacher's average velocity. In a 4-step FLUX.1-dev student, ReSeed improves DINOv3 diversity by 48.3% with only 0.00196% additional parameters, using 8,000 prompt-noise pairs.

Generative media

Text-to-motion model animates unseen skeletons from their skeleton file alone

Sun et al. present TopX, a 73M-parameter flow-matching transformer that generates motion for any kinematic tree directly from its skeleton file, tree topology, rest pose, and joint names, with no clips of the character and no per-rig normalisation. On held-out skeletons, TopX reduces the descriptor Fréchet distance from 5.73 to 4.58 and the kinetic Fréchet distance from 1.48 to 0.55 compared to UniMate, while producing fewer frames with foot skating.

Generative media

Video generators better preserve subjects’ appearance as cameras circle them

Ahn et al. introduce OmniShowcase, a subject-to-video generation framework that trains on synthetic 360-degree renders and conditions the video model on reconstructed surface-normal maps from multi-view reference images. OmniShowcase leads prior single- and multi-view S2V methods on subject identity and cross-view 3D consistency on both a new benchmark (ShowcaseBench) and the existing NAVI set, with competitive video quality and motion.

Generative media

AI video generators reduce flickering and preserve detail with few steps

The authors identify that distribution matching distillation (DMD) for few-step video generation degrades because it only supervises output samples, lacking regularization on internal representations. They introduce Self Flow Forcing (SF²), which forces the student to predict the teacher's internal representations from a more corrupted input using an exponential moving average teacher. This consistently improves spatial fidelity and temporal coherence in both bidirectional and causal video models with minimal computational overhead.

Generative media

AI can render moving scenes from new viewpoints in real time

Hong and colleagues present FL4N, a model that turns monocular video into a representation for rendering a moving scene from different viewpoints and at different times. Rather than learning scene structure and motion from scratch, it reuses a pretrained static-scene model and learns temporal changes as corrections to its internal representation. The authors report leading or competitive rendering quality, depending on the camera information supplied, alongside real-time rendering.

Generative media

Image generation improves through verified repairs without retraining the generator

Han and colleagues introduce ImageRSI, a system that checks generated images, repairs their prompts and saves repair strategies only after comparative testing. With the image generator held fixed, the authors report Checklist Strict Accuracy of 52.6% on MindBench and 65.66% on their new ImageReq-Bench, exceeding the strongest compared baselines by 10.2 and 9.28 percentage points, respectively.

Generative media

Video generators can change characters or cameras while preserving other details

The authors propose Factor-Aware Predictive Learning (FAPL), a three-stage training procedure that makes the representations of a frozen multimodal large language model directly actionable for a Video Diffusion Transformer in cinematic generation. FAPL consistently outperforms baselines on sparse-to-video, factor-aware editing, and multi-shot tasks, with each stage contributing to the overall improvement.

Generative media

Image generators create less distorted panoramas with fewer visible seams

The authors propose Squaring-the-Sphere, a framework for 360° panorama generation using CubeAtlas, a new representation that arranges cubemap faces into a single canvas. This preserves perspective fidelity while allowing a pre-trained diffusion transformer to model the full sphere in one forward pass, without modifying the backbone. The method achieves better geometric fidelity and cross-view consistency than prior work, with reduced attention computation via a layout-aware sparse mask.

Generative media

System streams infinite 3D worlds that remember and restore user edits

The authors present WorldStream, an open-ended 3D world model that concurrently extends terrain beyond any pre-specified map, acts on open-ended language in the running world, and restores visited places with their edits. It uses a hierarchical planner (Macro/Micro) over a typed decision model and deterministic local code to generate terrain, roads, settlements, and entities, all coordinated through a single persistent world record. The key finding is that streaming keeps per-frame cost bounded as explored extent grows, while the retained record ensures consistent state across planning, generation, interaction, and re-rendering.

Generative media

A single flow matching Transformer generates images and text jointly in pretrained encoder spaces

The authors introduce JUICE, a flow matching Transformer that models the joint distribution of images and text directly in frozen pretrained encoder spaces, with each modality having its own noise level. This lets text-to-image generation, captioning, VQA, and joint generation become paths through a square of noise levels. At 2.1B parameters, JUICE reaches up to 0.88 on GenEval and 73.1 on VQAv2, and coupled sampling raises joint route-and-text success on multi-solution mazes from 54.6% to 93.7%.

Generative media

Two classifier heads align few-step diffusion with external rewards.

The authors propose DualHeads, which uses two lightweight binary classifiers on a frozen diffusion backbone to tilt a few-step generator toward higher rewards. The reward head ranks samples by their black-box scores, and the reference head separates generated samples from reference images; the generator maximizes a weighted sum of their logits. A four-step student matches or exceeds the reward of 40-step reinforcement learning methods while maintaining image quality and incurring no extra inference cost.

Generative media

Single-stage RL method unifies preference alignment and few-step image generation

The authors propose AdvFidelity, which integrates artifact-sensitive fidelity supervision into reinforcement learning (RL) through an online adversarial reward. This enables joint reward alignment and high-fidelity few-step generation within a single RL stage, without explicit distillation objectives. Across two model backbones and three task settings, AdvFidelity matches full-step LeapAlign in alignment, task performance, and overall generation quality using only 4 or 8 sampling steps without classifier-free guidance.

Generative media

Time-resolved diagnostic separates irreducible velocity ambiguity from reducible training error

Mei et al. propose the Velocity Ambiguity Profile (VAP), a time-local Bayes-risk floor for velocity prediction in flow matching, and prove that the time-resolved loss decomposes as L_N(t) = A_v(t) + R(t;N), where only R(t;N) can be reduced by more data or a better estimator. In controlled Gaussian-mixture experiments the diagnostic behaves as predicted, and on CIFAR-100 DINOv2 feature flow guided sampling cuts uniformly evaluated velocity loss by 6.78% and global feature Fréchet distance by 23.6%, though class-conditional KID shows no improvement.

Generative media

Future chunk prediction improves physical plausibility in autoregressive video generation

The authors introduce Future Chunk Prediction (FCP), an auxiliary objective for training autoregressive video diffusion models. In addition to standard teacher-forced denoising, FCP predicts a future chunk without access to intervening visual context. They report improved physical plausibility across model scales and conditioning settings, with a lower overall training compute budget, suggesting that future-chunk supervision efficiently models visual evolution.

Generative media

Guided diffusion enables trustworthy synthetic text with formal privacy guarantees

The authors propose Guided Diffusion, a framework that uses diffusion language models (DLMs) to generate synthetic text conditioned on semantic embeddings. By applying differential privacy (DP) to the embeddings and then generating text via post-processing, they achieve rigorous privacy guarantees with minimal utility loss. Across two datasets and two DLM backbones, the method outperforms Private Evolution and DP-SGD fine-tuning in distributional fidelity and downstream utility, while generating text significantly faster. It also extends to Data-Specific Indistinguishability (DSI), offering a formal certificate against data poisoning and memorization at a small utility cost.

Generative media

Self-correcting flow matching via lightweight refiner that reuses hidden states.

The authors introduce CorrFlow, a self-correcting flow-matching framework that uses a lightweight refiner to predict velocity residuals. The refiner leverages the base model's clean prediction, hidden states, and conditioning, without extra forward passes. CorrFlow improves visual quality and physical consistency across class-conditional image generation, text-to-video, and image-to-video generation.

Generative media

Feed-forward framework unifies 4D reconstruction and generation from video latents

SplatFlux is a feed-forward model that takes video latent sequences -- either from a VAE-encoded real video or from a video diffusion model -- and outputs a 4D Gaussian splat representation of the dynamic scene. The authors achieve this by introducing a static-dynamic decoupled Gaussian representation with explicit dynamic probability prediction, allowing complete 4D scene construction in one forward pass. They demonstrate state-of-the-art performance on both video-based 4D reconstruction and conditional 4D generation tasks.

Generative media

3D diffusion model generates body CT masks with controllable anatomy, tumors, demographics

The authors present NV-Generate-BodyMask, a 3D latent diffusion model that synthesizes whole-body CT segmentation masks conditioned on explicit organ size quantiles, five tumor types, and four demographic variables. Trained on 8,571 masks from 26 cohorts, the model achieves organ size correlations (Spearman ρ = 0.70-0.92), high tumor recall for most lesion classes, and demographic control validated by BMI, weight, and sex.

Generative media

Noise-rejuvenated SMC enables test-time scaling for flow matching image generation

The authors introduce noise-rejuvenated sequential Monte Carlo (SMC), an inference-time scaling method for text-to-image flow matching models. By perturbing resampled particles via source noise replacement, the method enables exploration without converting the deterministic ODE sampler to an SDE. Experiments on three state-of-the-art models show consistent improvements in reward-guided generation across multiple benchmarks.

Generative media

Geometry-aware test-time scaling improves camera control in video diffusion without retraining

The authors present GeoART, a training-free self-refinement method for camera-conditioned video diffusion. By generating multiple candidates at selected denoising steps and evaluating them with a geometry prior, GeoART improves camera controllability and multi-view consistency without modifying the pretrained model. Rank-calibrated probabilistic selection, rather than greedy choice, balances score magnitude and relative rank to handle noisy geometry scores.

Generative media

BEV maps enable human-friendly control for embodied navigation video generation

The authors introduce Map2Vid, a task that generates egocentric navigation videos conditioned on bird's-eye-view (BEV) maps, offering a more intuitive alternative to text prompts or camera trajectories. They built a dataset of 117,800 navigation sequences across 61 indoor scenes, each paired with videos, BEV maps, trajectories, and language instructions. A curriculum training strategy gradually weakens non-map inputs, achieving robust performance with BEV maps alone.

Generative media

Robust prediction principle yields better latent spaces for diffusion models

The authors revisit REPA-E, a prior approach for improving latent diffusability, and find that removing the diffusion objective while retaining semantic prediction from corrupted latents gives comparable results. This motivates a general principle—robust prediction—which they instantiate in RoAR-VAE using two complementary objectives: RobustAlign (predicting pretrained semantic features) and RobustRecon (reconstructing the clean image). RoAR-VAE improves average generation benchmark performance by 2.19 points over the strongest baseline and boosts small-text reconstruction accuracy by 16.93 percentage points over FLUX.1-VAE.