Sat 31 Oct
LiveGet it free
ZOTPAPER

News without the noise

Research

New AI papers from arXiv, picked daily and explained with their claims, benchmark results and code.

2566 articles · page 8 of 54 · The Research front page

AllAgentsLanguage modelsReinforcement learningVisionGenerative mediaSpeech & audioRobotics & embodied AISafety & securityEvaluation & benchmarksEfficiency & systemsCode & softwareAI for science & healthTheory
Viewgridlistcompact
Evaluation & benchmarks

Dual Exam benchmarks agents and world models separately, exposing weak coupled performance

The authors introduce Dual Exam, a two-track evaluation framework that tests multimodal agents under fixed environment responses and world models under fixed action interventions across physical, digital, social, and scientific domains. On 1,088 agent items and 400 world model cases, top scores were 31.3 for GPT 6 Astra and 21.7 for GPT-Image-2, while a 150-case coupled pilot achieved zero task successes.

Robotics & embodied AI

Sensorimotor alignment predicts robot task success offline

The authors propose Sensorimotor Alignment (S^2), a training-free offline metric that quantifies the alignment between a model's sensory (vision and language) representations and end-effector trajectories. Across vision-language-action models, raw vision-language models, and world action models in simulation and on real robots, higher S^2 scores correlate positively with downstream task success, offering a low-cost proxy for model selection before expensive closed-loop evaluation.

Generative media

Image generators create less distorted panoramas with fewer visible seams

The authors propose Squaring-the-Sphere, a framework for 360° panorama generation using CubeAtlas, a new representation that arranges cubemap faces into a single canvas. This preserves perspective fidelity while allowing a pre-trained diffusion transformer to model the full sphere in one forward pass, without modifying the backbone. The method achieves better geometric fidelity and cross-view consistency than prior work, with reduced attention computation via a layout-aware sparse mask.

Evaluation & benchmarks

Reinforcement learning yields user simulators that stay faithful and sound natural

The authors address a trade-off in user simulators for evaluating AI assistants: prompted LLMs follow complex intents but sound unnatural, while specialized simulators sound human but drift from task instructions. They propose a multi-turn reinforcement learning recipe that jointly rewards task intent coverage and behavioral naturalness, using a dynamically retrained discriminator to judge naturalness. The resulting model, UserLM-2.0, achieves valid conversations 46.2% of the time versus 14.1% for the strongest prior simulator.

Evaluation & benchmarks

AI agents struggle more with understanding what they see than reasoning

The authors introduce AgentVQA, a curated offline benchmark of 1,000 multiple-choice questions drawn from 23 datasets across five agentic domains: web agents, robotics, egocentric video, games, and spatial planning. Evaluating 22 vision-language models, they find the best reaches only 58.5% accuracy, and error analysis attributes most failures to inadequate grounding and tracking of visual state rather than to task misunderstanding or reasoning errors.

Efficiency & systems

Language models can give identical answers up to 2.78 times faster

The authors propose ValuSpec, a candidate valuation mechanism that filters low-quality branches from a candidate tree before target-side verification. It introduces transfer-based supervision to calibrate a filtering threshold using the target model's greedy trajectories, requiring no parameter training. The main finding is up to 2.38x end-to-end speedup in retrieval-based tree speculative decoding settings.

Agents

AI agents can outline organs in scans from plain-language requests

Isensee et al. combine the promptable 3D segmentation model nnInteractive with general-purpose AI agents via the Model Context Protocol, creating nninteractive-agent. Without additional training, the agents interpret natural-language requests, decide on targets, place prompts, review 3D results, and refine them. On a benchmark spanning CT, MRI, PET, and micro-CT of humans, a mouse, and an ant, the agents reach a mean Dice of 0.81–0.84 from the request alone and match oracle prompting when given target locations, while outperforming a text-prompted model in robustness across uncommon anatomies and non-human species.

Efficiency & systems

Rewriting softmax calculations fixes two rounding errors that distort training

Jonathan T. Barron, Dan Gnanapragasam and Michael Tschannen identify two floating-point errors in standard softmax and cross-entropy implementations, functions that turn model scores into probabilities and training losses. Their replacements preserve small gradient signals without higher-precision arithmetic and, in the reported experiments, remove training instabilities and unwanted drift during knowledge distillation.

Generative media

System streams infinite 3D worlds that remember and restore user edits

The authors present WorldStream, an open-ended 3D world model that concurrently extends terrain beyond any pre-specified map, acts on open-ended language in the running world, and restores visited places with their edits. It uses a hierarchical planner (Macro/Micro) over a typed decision model and deterministic local code to generate terrain, roads, settlements, and entities, all coordinated through a single persistent world record. The key finding is that streaming keeps per-frame cost bounded as explored extent grows, while the retained record ensures consistent state across planning, generation, interaction, and re-rendering.

Safety & security

Persistent memory in LLMs enables cross-session security breaches

The authors propose a general threat model for memory injection attacks on large language models (LLMs) that leverage persistent memory features. They show that by planting malicious instructions in personal memories, projects, skills, organization instructions, or Model Context Protocol (MCP) resources, attackers can achieve all NIST-defined attack goals, availability, integrity, privacy, and misuse, across ChatGPT Instant 5.3, Claude Sonnet 4.6, and Gemini 3 Flash. The attacks persist across sessions and, in the case of Claude, can spread to other users in an organization.

Evaluation & benchmarks

Cheap baselines often rival expensive LLM methods on graph node classification

Podstawski tunes four cheap baseline models on splits from three recent studies of LLM-based node classification on text-attributed graphs, comparing them to the best reported LLM accuracy. Correct & Smooth on TF-IDF features trails the reference by a median of 2.3 points and an MLP on frozen embeddings by 2.8 points. The authors also document several evaluation risks in released code, including test-set hyperparameter selection and label leakage.

Generative media

A single flow matching Transformer generates images and text jointly in pretrained encoder spaces

The authors introduce JUICE, a flow matching Transformer that models the joint distribution of images and text directly in frozen pretrained encoder spaces, with each modality having its own noise level. This lets text-to-image generation, captioning, VQA, and joint generation become paths through a square of noise levels. At 2.1B parameters, JUICE reaches up to 0.88 on GenEval and 73.1 on VQAv2, and coupled sampling raises joint route-and-text success on multi-solution mazes from 54.6% to 93.7%.

Safety & security

Embedding-free framework enables private topic discovery in LLM conversations

The authors introduce CALLIOPE, a framework that discovers topic hierarchies from large text corpora (e.g., LLM conversations) while supporting differential privacy. Unlike prior approaches that rely on clustering vector embeddings, CALLIOPE operates entirely in the text domain by leveraging LLMs to abstract documents and recursively build taxonomies. The authors report that this approach yields better utility than embedding-based methods, especially under tight privacy budgets.

Speech & audio

Audio-language model suite adds speaker-aware transcription and understanding

The authors present Qwen-Audio-3.1-ASR, a family of audio-language models that extend accurate speech recognition toward structured transcription and broader audio interpretation. The models retain support for 30 languages and 16 Chinese dialects, and introduce instruction-controlled features like historical-context conditioning and hotword customization. Most notably, they demonstrate joint speaker diarization and transcription, as well as audio understanding tasks such as temporal grounding and sound event detection.

Language models

Fine-tuning small models outperforms large ones for query clarification in RAG

The authors introduce ClariEdge, a dataset of about 3,900 examples for query clarification in retrieval-augmented generation (RAG). They show that a fine-tuned small LoRA model substantially outperforms prompting of much larger models on both classification and generation of clarifications. Hard negatives, queries that look ambiguous but are resolvable from context, are critical for performance.

Vision

BlazeVGGT speeds up VGGT 13x by selecting representative views without retraining

The authors introduce BlazeVGGT, a training-free plug-in for VGGT that sparsifies cross-view attention at the frame level. By analyzing camera tokens, they found that shallow layers capture local view topology, while deeper layers encode global geometry. Using a greedy determinantal point process to select complementary views, BlazeVGGT achieves up to 13x faster inference at 2000 frames with a lower Chamfer distance (0.41 to 0.32).

Agents

Aggregating internal signals reveals hidden agreement dynamics in LLM conversations

The authors study two LLMs discussing subjective binary topics over multiple turns, using a linear classifier on hidden states to measure internal agreement. They find that while internal agreement fluctuates wildly per conversation, aggregating across conversations reveals a clean first-order Markov structure for behavioral agreement and a power-law relation between aggregated internal and external agreement.

Vision

OACap turns video captioning into active, tool-guided observation

Shi et al. propose OACap, an agent that treats an evolving video caption as a testable account, distinguishing gaps that can be revised from existing evidence from gaps that require targeted observation via tools such as cropping, dense sampling, and OCR. Across five backbones and three video-understanding benchmarks, OACap consistently improves caption-based question answering, and the authors introduce FSCap-Bench with two new metrics to diagnose factual and semantic preservation.

Vision

VLMs mostly err by ignoring visual evidence they possess

The researchers propose a method to decompose VLM errors into two types: those where necessary visual information is missing (representation failure) and those where it is encoded but not used (utilization failure). Using a lightweight rank-based probe (VELens) on over 6,000 images, they estimate that utilization failures account for the vast majority of errors, suggesting that improving how VLMs use existing visual information is more urgent than improving visual encoding.

Evaluation & benchmarks

First benchmark evaluates coding agents on XR development tasks

The authors introduce GenXR-Bench, the first benchmark for end-to-end evaluation of coding agents in extended reality (XR) software development. It comprises 112 tasks spanning SDK engineering and application generation. Evaluating 10 model configurations, the strongest achieves 50.9% overall Pass@1; failure analysis reveals recurring misunderstandings in coordinate transformations, asynchronous sensor lifecycles, and bimanual interaction semantics.

Language models

Draft-training objective gains shrink when target regenerates training responses

The authors compare three draft-training objectives on external versus target-regenerated responses for a vision-language speculative decoding pipeline. Distillation and a verification gate show large gains on external responses, but after greedy regeneration, distillation adds only a small gain and the gate little or none. The shrinkage persists across controls, corpora, and a text-only replication.

Agents

A verified memory interface makes agent security depend on explicit confirmations

Jiang and colleagues present PerMIT, an architecture that treats information from tools and other external sources as untrusted, even after it enters an agent’s persistent memory. They prove that, within its interface, such values can reach sensitive actions only after explicit confirmation, then test whether automated verification and accumulated community endorsements make that boundary practical.

Efficiency & systems

Lower precision beats structured pruning until GPU quantization limits are reached

Xiuyi Qin compares two ways to accelerate large language models: reducing numerical precision and imposing 2:4 sparsity, which retains at most two values in each hardware-defined group of four. In the tested configurations, lower precision was both faster and more accurate whenever another precision step was available. At the hardware’s lowest precision, sparsity delivered additional speed, but preserving model quality required retraining.

Agents

Executable environments enable policy benchmarks and iterative guard training

The authors present AgentPolicyForge, a framework that uses executable environments to create policy-compliance benchmarks and iteratively train policy guards through coevolution of attacks and defenses. Using this framework they built PolicyGym, a dataset of 952 scenarios across ten policy families. The final trained guard achieved an 8.22% attack success rate in-domain, a 40–44% relative reduction compared to policy-aware task agent self-regulation.

Generative media

Two classifier heads align few-step diffusion with external rewards.

The authors propose DualHeads, which uses two lightweight binary classifiers on a frozen diffusion backbone to tilt a few-step generator toward higher rewards. The reward head ranks samples by their black-box scores, and the reference head separates generated samples from reference images; the generator maximizes a weighted sum of their logits. A four-step student matches or exceeds the reward of 40-step reinforcement learning methods while maintaining image quality and incurring no extra inference cost.

Language models

LongZero trains LLMs on self-written long-context tasks, no external data

Wan and Chen propose LongZero, in which a language model synthesizes question, evidence, and answer as a compact structured specification, then expands it into a long multi-document context, and trains with GRPO. Across two base models and five long-context reasoning benchmarks, LongZero outperforms RL on synthetic data built from external corpora, generalizing to real tasks without any external data.

Vision

Unified visual encoder delivers state, language, and reconstruction for world models

The authors pretrain a ViT-B/16 from scratch using language similarity, pixel reconstruction, and spatial correspondence objectives, then refine it with cross-view self-distillation. On fifteen-encoder benchmarks, ALPS achieves the lowest LIBERO state-probe error among ViT-B models, matching the 3.5x larger V-JEPA 2-L, while retaining strong language alignment and enabling pixel-perfect reconstruction. Frozen ALPS tokens further achieve 85.5% mean policy success on LIBERO with a native text tower.

Language models

A language model using only self-conditioning outperforms existing diffusion models

The authors introduce LAW (Loopholing All the Way), a continuous generative model that relies purely on self-conditioning, without auxiliary conditioning mechanisms. Starting from uniformly random tokens, LAW refines the expected embedding of the clean token under the denoiser's prediction at each step and decodes to discrete tokens only after the final step. On Sudoku, Maze, and GSM8K, LAW outperforms prior discrete and continuous diffusion models, and on OpenWebText it approaches the quality of autoregressive models with 64 function evaluations.