Vision

Image and video understanding, multimodal perception, detection, segmentation, 3D understanding

201 articles · page 4 of 5

Vision

Parallel diffusion decoding rivals autoregressive visual grounding at 4B scale

The authors introduce GroundAnything, a 4B-parameter visual grounding model that replaces serialized autoregressive token prediction with blockwise diffusion decoding. By allowing spatial hypotheses to be filled in parallel and refined iteratively, the model attains strong accuracy while cutting inference latency. An autoregressive sibling, GroundAnything-VLM, sets a new state of the art among similarly sized models.

Vision

Distribution drifting on spatial features enables efficient one-step super-resolution

The authors propose DriftSR, a one-step framework for real-world image super-resolution that uses a frozen diffusion model’s intermediate features. By treating spatial locations as samples for distribution drifting and using structure-modulated guidance, it achieves high perceptual quality without score distillation or adversarial training. The method outperforms previous one-step methods on real-world benchmarks while maintaining inference efficiency.

Vision

Probability rewards keep visual grounding reasoning aligned with predicted objects

Sun et al. study thinking drift in visual intention grounding, where a vision-language model identifies an object from a goal-oriented query and predicts its bounding box. They show that reinforcement learning based only on box overlap can improve localization while reinforcing reasoning that names a different object. Their proposed Rita framework reduces this mismatch without requiring labeled reasoning traces.

Vision

Encoder-free model unifies image and video understanding and generation in pixel space

The authors present PixelUMM, an encoder-free model that handles both image and video understanding and generation directly in pixel space. By representing images as spatial patches and videos as spatiotemporal tubelets, and using a Mixture-of-Transformers architecture, the model achieves competitive performance across tasks. This design removes the need for separate pretrained visual encoders (e.g., ViT for understanding, VAE for generation), simplifying the visual interface.

Vision

Image restoration can learn new tasks without forgetting by updating only critical filters

The authors propose Restoring without Forgetting (RwF), a continual learning method for image restoration that adapts models to sequential degradation tasks without accessing past data. RwF identifies a small set of degradation-critical filters using integrated gradients in parameter space, then generates task-specific filters from a filter bank using factorized low-rank transformations. On six restoration tasks, RwF avoids catastrophic forgetting and matches the quality of all-in-one models, while using about 10 times fewer additional parameters than LoRA-style adaptation.

Vision

Graded preference optimization improves limb-motion caption fidelity

The authors present a framework for improving fine-grained limb-motion captioning in Vision-Language Models (VLMs). They introduce FlexBench, a benchmark of 3,105 shots and 18,161 queries with human-verified person identities and per-shot limb action checklists, and the Graded Physical Alignment score (GPA), which awards credit for correct content and deducts for incorrect or fabricated actions. Building on this rubric, their Graded Margin Direct Preference Optimization (GM-DPO) assigns stronger preference margins and training weight to more severe action errors, achieving the highest GPA scores across three VLM backbones and reducing hallucination rates relative to standard DPO.

Vision

Large-scale CT dataset provides tumor masks for 9 organs

The authors present Merlin Plus, a public CT dataset containing radiologist-created tumor masks for 9 organs that previously lacked such annotations. Using a report-based active learning framework, they generated initial masks from radiology reports, which radiologists reviewed and corrected, reducing annotation time by 3-5 times. The dataset enables multi-tumor segmentation and detection models that outperform existing public models.

Vision

Lifted training with prototypes improves classifier accuracy without changing inference architecture

The authors propose ProtoSeam, a training method that splits a classifier into two subnetworks and inserts a learnable Gaussian prototype per class at the interface. During training, a consensus penalty pulls early embeddings toward their class prototype while the head is trained on samples drawn around those prototypes, with no gradient crossing the seam. At inference the prototypes are discarded, and the unmodified network is deployed. Across six dataset-architecture pairs, the approach consistently improves test accuracy over both a baseline and a matched end-to-end control, with gains of up to 5.2 percentage points.

Vision

Flow matching plus physics yields more trackable 3D human motion from video

Wang et al. present FlowHMR, a method for recovering physically plausible global 3D human motion from monocular video. They formulate motion capture as a video-conditioned generation problem, first pretraining a flow matching model on synthetic data, then post-training it via reinforcement learning with rewards for both fidelity to the video and trackability by a physics-based controller. On their new Wild-4K evaluation set, their method achieves a physical tracking success rate of 82.47%, compared to 62.82% for the strongest baseline, GVHMR.

Vision

Self-supervised tri-modal model perceives and forecasts any road obstacle without labels

The authors present TriO, an unsupervised world model for self-driving perception that uses camera, LiDAR and RADAR as both inputs and self-supervision, removing the need for human annotations. The model predicts 4D occupancy, obstacle segmentation, flow and future LiDAR, reporting state-of-the-art results including zero-shot road obstacle segmentation on Argoverse 2 and Spotting the Unexpected.

Vision

Continuous feature decoding improves fine-grained semantic matching between images

Choi examines why semantic correspondence systems, which locate matching semantic points across images, lose accuracy under strict spatial tolerances. The paper attributes much of this decline to patch-grid quantization and reports that continuous feature decoding improves fine-grained accuracy on SPair-71k and AP-10K, including a 6.2 percentage point gain over the previous state of the art at PCK@0.01 on SPair-71k.

Vision

Loop closure in streaming 3D reconstruction achieves kilometer-scale accuracy without drift

The authors present CLoSeR, a system that integrates loop closure into a feedforward streaming reconstruction backbone (LoGeR) to eliminate tracking drift on long trajectories. They detect loop candidates via global descriptor retrieval, construct loop-conditioned windows for relative pose estimation, and optimize all poses on the SE(3) manifold, leveraging the backbone's inherent scale consistency. On kilometer-scale monocular sequences, CLoSeR substantially reduces drift and produces consistent geometry, outperforming prior state-of-the-art methods.

Vision

One-step diffusion framework restores 1080p video in real time with efficient streaming attention

Chen et al. present FastVR, a one-step diffusion framework for streaming video restoration that processes 1080p video at 11 FPS on a single H20 GPU. By pairing a lightweight VAE with chunk-wise causal attention, the method targets the two bottlenecks that dominate once iterative sampling is compressed into a single step. The authors report state-of-the-art restoration quality on synthetic and real-world benchmarks while remaining faster than the diffusion baselines evaluated.

Vision

Training VLMs to construct reference frames improves perspective taking.

The authors introduce LeRF, a framework that trains VLMs to dynamically decide whether to construct a reference coordinate frame, ground the relevant entity, and render a directional frame on the image. Using supervised fine-tuning and reinforcement learning, the model learns to use these visual cues for perspective-taking tasks, achieving state-of-the-art performance among open-source VLMs on multiple benchmarks. The rendered frames are shown to be causally important for reasoning, particularly for queries requiring non-camera viewpoints.

Vision

Pruning visual tokens and decoder sublayers cuts multimodal model computation

Chen and colleagues target two sources of multimodal model overhead: redundant visual tokens and unnecessary computation inside the language-model decoder. Their training-free SPIDER framework retains complementary visual information from middle and deep vision layers, then selectively skips attention or feed-forward sublayers. On LLaVA-NeXT-7B, the authors report 79 percent fewer FLOPs while retaining 96 percent of baseline performance.

Vision

Self-distillation helps vision-language models reason with far fewer visual tokens

Jeddi et al. investigate why reasoning vision-language models falter when most visual tokens are removed. They find that pruning does not always erase the necessary evidence: repeated sampling from the same sparse representation often produces correct answers that greedy decoding misses. Their self-distillation method improves how reliably models use this surviving information, reaching 92.43% of unpruned performance with only 10% of visual tokens retained.

Vision

Fine-Grained Rewards Reduce Vision-Language Hallucinations Without Silencing Valid Detail

Long and colleagues address a failure mode of on-policy training for vision-language models: a model can reduce object hallucinations simply by making fewer claims, including fewer correct ones. Their framework instead supplies dense evidence about which objects are present or absent and assigns rewards separately to parts of a response, producing descriptions that the authors report are both more informative and more faithful.

Vision

Steering hidden states stabilizes vision-language reasoning against subtle visual changes

The authors identify answer flips—cases where nearly identical images cause vision-language models to produce different answers—and propose FlipDir, a training-free inference-time method that estimates a low-rank subspace of flip-inducing activations and selectively steers hidden states during decoding. A margin-based gate restricts steering to uncertain steps, recovering original predictions while leaving stable ones unchanged. Across 18 settings spanning scientific reasoning, robot-scene understanding, and medical VQA, FlipDir consistently outperforms existing methods on a combined metric of recovery and preservation.

Vision

Asymmetric certainty gains from optimization hinder multimodal classification

The paper identifies a flaw in multimodal learning: optimization yields asymmetric gains in predictive certainty between strong and weak modalities, driving imbalanced contributions. The authors propose Max Confidence Regularization (MaxCR) to dynamically intervene in modality confidence using a nonlinear sparsity measure, penalizing overconfident strong modalities and encouraging underconfident weak ones. Experiments on several datasets show MaxCR improves overall performance over state-of-the-art baselines.

Vision

Parallel tile inspection helps models find details in large images

Tao et al. address a recurring failure in high-resolution visual question answering: crucial evidence can disappear when an image is resized, yet a model may not know where to zoom from an initial overview. Their Visual Parallel Search framework first examines a grid of tiles concurrently, then lets a main agent combine the tile reports, zoom into selected or merged regions, and answer the question. Across five benchmark splits and three model sizes, this scaffold generally outperformed a sequential zoom-only approach.

Vision

Forensic tool use improves explainable zero-shot image forgery detection

Tan et al. present ATAR, an agentic system that investigates images through repeated visual inspection and calls to specialized forensic tools, rather than explaining a classifier’s predetermined output. Across six zero-shot image manipulation detection and localization benchmarks, ATAR achieved an average image-level F1 score of 78.5%, 11.8 percentage points above the strongest multimodal language model baseline evaluated.

Vision

Visual image rearrangement helps models when exact evidence matters most

Zhang et al. treat multi-image reasoning as a problem of re-representation: reorganising visual evidence so a model can inspect relationships that are difficult to infer from the original images alone. They compare textual reasoning with visual transformations, introduce the grounding-focused MosaicBench, and train MosaicAgent-8B to operate on images. Their central finding is that visual re-representation helps most when answers depend on precise alignment, orientation, or pixel-level comparison, rather than broad semantic interpretation.

Vision

Shared scene graphs connect interaction understanding with grounded robot planning

Zhang and colleagues introduce ChronoGraph, a representation that links actions on functional object parts, such as handles, to subsequent semantic and geometric changes in a scene. They use automatically annotated human videos and simulated robot trajectories to train vision-language models to reconstruct past interactions and predict future states, then demonstrate that the resulting plans can guide a mobile manipulator through existing robot skills.

Vision

Grounding long-video memories in visual identity lets QA systems track objects across days

The authors introduce Grounded Entity Biographies (GEB), a long-video memory framework that groups visually grounded observations of the same physical instance across clips into retrievable biographies. On four benchmarks, including day-long and week-long recordings, GEB outperforms prior memory frameworks in both multiple-choice and open-ended question answering. On EgoLifeQA it reaches 72.0%, 4.4 percentage points above the best published result.

Vision

Multi-agent framework generates spatial QA data via code execution

Ying et al. introduce Exemplar2VQA, a multi-agent framework that synthesizes large-scale 3D spatial question-answer pairs in simulated environments. By converting user-defined exemplar queries into deterministic code via specialized agents, the system avoids hallucinations common to LLM-based generation. Fine-tuning Qwen2.5-VL models on this synthetic indoor data improves performance on multiple spatial benchmarks, including outdoor and mixed-scene settings.