News without the noise
Image and video understanding, multimodal perception, detection, segmentation, 3D understanding
30 articles
The authors propose KeyRec, a training-free method that builds bounded visual memory by maintaining a dense recent cache and a structured event bank. It achieves the best compressed performance in 13 of 15 settings across four benchmarks using only 10% of the decoder-facing visual-token budget, outperforming compressed baselines by 2.21 to 18.37 points on real-time questions.
The authors conduct a controlled architecture-objective study and propose TT-VidT, which uses a DINOv3-initialized ViT-B/16 per-frame spatial path and a compact Temporal Transfer Layer, trained with Diff Compression to reconstruct frames from a first-frame anchor and motion tokens. They show TT-VidT leads fine-tuning on Jester, Something-Something V2, ARID, and Diving48, improving 54–121% over non-TT methods, with 48–55% fewer encoder FLOPs than DisMo and VideoMAE/V-JEPA2.
Chen and colleagues target two sources of multimodal model overhead: redundant visual tokens and unnecessary computation inside the language-model decoder. Their training-free SPIDER framework retains complementary visual information from middle and deep vision layers, then selectively skips attention or feed-forward sublayers. On LLaVA-NeXT-7B, the authors report 79 percent fewer FLOPs while retaining 96 percent of baseline performance.
Jeddi et al. investigate why reasoning vision-language models falter when most visual tokens are removed. They find that pruning does not always erase the necessary evidence: repeated sampling from the same sparse representation often produces correct answers that greedy decoding misses. Their self-distillation method improves how reliably models use this surviving information, reaching 92.43% of unpruned performance with only 10% of visual tokens retained.
Moon and colleagues address open-vocabulary 3D segmentation, in which users identify scene objects with unrestricted text rather than predefined labels. Their EviSplat system stores separate features from multiple camera views and selects among them only after receiving a query, instead of compressing potentially conflicting evidence into one descriptor beforehand.
The authors train a multimodal language model to alternate between chain-of-thought reasoning and tool calls that crop relevant regions from a table image. Across nine table benchmarks, InterTab raises its backbone model’s average accuracy from 68.28% to 73.17%, a gain of 4.89 percentage points.
Long and colleagues address a failure mode of on-policy training for vision-language models: a model can reduce object hallucinations simply by making fewer claims, including fewer correct ones. Their framework instead supplies dense evidence about which objects are present or absent and assigns rewards separately to parts of a response, producing descriptions that the authors report are both more informative and more faithful.
ProDyGS is a dynamic Gaussian splatting method for novel view synthesis from a single static camera video. By generating proxy images from estimated depth maps, it creates synthetic multi-view supervision. Experiments on the DyNeRF dataset show state-of-the-art quality, outperforming methods that rely on scene flow priors.
The authors identify answer flips—cases where nearly identical images cause vision-language models to produce different answers—and propose FlipDir, a training-free inference-time method that estimates a low-rank subspace of flip-inducing activations and selectively steers hidden states during decoding. A margin-based gate restricts steering to uncertain steps, recovering original predictions while leaving stable ones unchanged. Across 18 settings spanning scientific reasoning, robot-scene understanding, and medical VQA, FlipDir consistently outperforms existing methods on a combined metric of recovery and preservation.
Sunagad, Jesslen and Kortylewski dissected the training objectives behind DINO-style self-supervised vision transformers by removing or modifying individual components. They find that aligning two geometrically displaced global crops from the same image is sufficient for strong semantic correspondence, whereas local crops and patch masking do not independently create comparable structure.
The paper identifies a flaw in multimodal learning: optimization yields asymmetric gains in predictive certainty between strong and weak modalities, driving imbalanced contributions. The authors propose Max Confidence Regularization (MaxCR) to dynamically intervene in modality confidence using a nonlinear sparsity measure, penalizing overconfident strong modalities and encouraging underconfident weak ones. Experiments on several datasets show MaxCR improves overall performance over state-of-the-art baselines.
Tao et al. address a recurring failure in high-resolution visual question answering: crucial evidence can disappear when an image is resized, yet a model may not know where to zoom from an initial overview. Their Visual Parallel Search framework first examines a grid of tiles concurrently, then lets a main agent combine the tile reports, zoom into selected or merged regions, and answer the question. Across five benchmark splits and three model sizes, this scaffold generally outperformed a sequential zoom-only approach.
Tan et al. present ATAR, an agentic system that investigates images through repeated visual inspection and calls to specialized forensic tools, rather than explaining a classifier’s predetermined output. Across six zero-shot image manipulation detection and localization benchmarks, ATAR achieved an average image-level F1 score of 78.5%, 11.8 percentage points above the strongest multimodal language model baseline evaluated.
Zhang et al. treat multi-image reasoning as a problem of re-representation: reorganising visual evidence so a model can inspect relationships that are difficult to infer from the original images alone. They compare textual reasoning with visual transformations, introduce the grounding-focused MosaicBench, and train MosaicAgent-8B to operate on images. Their central finding is that visual re-representation helps most when answers depend on precise alignment, orientation, or pixel-level comparison, rather than broad semantic interpretation.
Zhang and colleagues introduce ChronoGraph, a representation that links actions on functional object parts, such as handles, to subsequent semantic and geometric changes in a scene. They use automatically annotated human videos and simulated robot trajectories to train vision-language models to reconstruct past interactions and predict future states, then demonstrate that the resulting plans can guide a mobile manipulator through existing robot skills.
Tao et al. introduce VPS, a visual parallel-search framework in which a main agent delegates tile inspection to parallel question-conditioned sub-agents and then adaptively zooms into relevant regions. Across five benchmark splits and three model sizes, VPS outperforms dedicated zoom-only search in 14 of 15 same-model comparisons, with gains up to 8.0 points. The authors also contribute a supervision pipeline with hint-free verification and a paired role-specific GRPO surrogate for post-training.
The authors introduce Grounded Entity Biographies (GEB), a long-video memory framework that groups visually grounded observations of the same physical instance across clips into retrievable biographies. On four benchmarks, including day-long and week-long recordings, GEB outperforms prior memory frameworks in both multiple-choice and open-ended question answering. On EgoLifeQA it reaches 72.0%, 4.4 percentage points above the best published result.
Ying et al. introduce Exemplar2VQA, a multi-agent framework that synthesizes large-scale 3D spatial question-answer pairs in simulated environments. By converting user-defined exemplar queries into deterministic code via specialized agents, the system avoids hallucinations common to LLM-based generation. Fine-tuning Qwen2.5-VL models on this synthetic indoor data improves performance on multiple spatial benchmarks, including outdoor and mixed-scene settings.
Ma et al. introduce an inference-time method that refines a short prefix of hidden states according to each token's estimated role. On two vision-language models, it improved macro accuracy over chain-of-thought prompting and beat output-space scaling baselines under matched candidate budgets.
Grange and Glantz equipped Qwen3.5-4B with callable tools for 3D object detection, metric depth, coordinate projection and deterministic spatial calculations. On ReVSI-Bench, the tool-augmented model improved markedly on three of four tasks, and experiments with ground-truth boxes showed that most remaining error came from visual perception rather than tool orchestration.
Shang et al. compared compact implicit neural representations, or INRs, with conventional U-Net pipelines for brain MRI segmentation across parameter budgets, augmentation settings and imaging sites. They find that INRs are strongest when models are small and augmentation is limited, but their advantage diminishes as U-Nets gain capacity and stronger training augmentation.
Milivojevic et al. present GraphWrit3R, a method that takes a 3D point cloud, Gaussian Splats, or a combination as input and directly outputs a complete scene graph as a structured JSON script. On the 3DSSG benchmark, the method achieves state-of-the-art performance on object class, predicate, and triplet recall, outperforming methods that rely on ground-truth object annotations during inference.
Li et al. propose SemMSA, a framework that uses LLM-generated latent semantics to compensate for missing or corrupted modalities in multimodal sentiment analysis. By iteratively refining semantic states and aligning them with visual and acoustic features without a fixed anchor modality, they report state-of-the-art performance on SIMS, MOSI, and MOSEI benchmarks.
The authors compare static and agentic pipelines for answering questions about long, visually complex documents using frontier and open-weight vision-language models. They find that the six-tool agent becomes competitive as the reader grows stronger, while top-k image retrieval offers the best token efficiency and three leading pipelines solve substantially different sets of questions.
The authors extended YOLOv2 with intermediate prediction layers (branched architecture) to allow early exit for easy cases, and introduced geometric features from Bayesian segmentation to distinguish visually similar signs. On a combined GTSDB/GTSRB dataset of ten classes, branched YOLOv2 achieved similar accuracy but slightly faster runtime (0.647 s vs 0.6607 s), and adding geometric verification during inference improved mAP from 0.680 to 0.713.
The authors introduce PersistBench, a dataset and metric suite using 360° videos as ground truth to test object permanence, motion continuity, and appearance preservation in 4D foundation models. Evaluating several models, they find that performance drops significantly after objects leave the field of view, concluding that current models 'see but do not remember.'
The authors introduce LatentStream, a progressive latent working-memory framework for multimodal LLMs processing streaming video. Instead of storing historical clips in an external memory bank and retrieving them as extra visual context, LatentStream converts retrieved evidence into compact, fixed-length latent memory tokens that evolve over time. Experiments show the method outperforms prior store-and-retrieve approaches on both online and offline video understanding benchmarks.
Xu and Chen isolate context construction as an optimization problem for long-video understanding. Their outer-loop system proposes, executes, and evaluates context-building programs around a frozen vision-language model, finding improvements from both a uniform-sampling start and a stronger hand-crafted baseline.
The authors propose Agentic Designer, a multi-agent system that generates interior furniture layouts by breaking the task into stages of proposal, evaluation, and refinement. Compared to one-shot methods, their approach enforces geometric constraints at each step, achieving superior structural adherence and functional coherence on a new benchmark, InStruct.
The authors introduce Boogu-Image-0.1, a family of open-source multimodal models that unify understanding and generation. By strengthening the system's understanding capability through a better multimodal encoder, agentic prompt rewriting, and inference-time scaling, the model matches or surpasses existing open-source models and approaches closed-source systems, all with a theoretical training cost of roughly $400K.