multi-turn tool use

The papers

Most central to this idea first, not most recent.

• Big Techcs.CL

  AI agents struggle to finish workflows with a thousand tools

  Even the best tested LLM agent completes only 5.0% of tasks in a benchmark requiring selection from 1,000 APIs over long chains of steps.

  Amazon, Georgia Institute of Technology · Nov 2026

• Big Techcs.CV

  Active audio-visual tool use improves reasoning across long recordings

  OmniSeek improves long-context audio-visual reasoning by letting models iteratively retrieve and combine targeted raw audio and video evidence.

  Adobe Research, University of California Davis · Oct 2026

• Chinese Techcs.CV

  Forensic tool use improves explainable zero-shot image forgery detection

  ATAR uses 22 forensic tools to detect and explain image forgeries through multi-turn reasoning.

  CCDS, Nanyang Technological University · Oct 2026

• Independentcs.AI

  Separating state tracking from actions improves multi-turn tool reliability

  Separating state tracking from action generation through a typed OODA loop improves multi-turn tool-use success across model sizes.

  Aug 2026

• Big Techcs.CL

  RL with live server state boosts multi-step tool use in LLMs

  Reinforcement learning with live server state and a programmatic reward improves multi-step tool use in LLMs.

  IBM Research · June 2026

• Big Techcs.SD

  AI agents locate sounds better after training with written reasoning

  A spatial audio agent answers questions, holds dialogue and calls tools on four-channel ambisonic recordings, scoring 44.9 percent on a new 12,000-sample benchmark, 7.4 points above prior systems.

  Microsoft Research · Nov 2026

• Chinese Techcs.CV

  Training on visual errors improves image understanding and tool use

  PerceptRL converts caption errors into verifiable QA and corrects tool-use drift to sharpen MLLM visual perception.

  Shanghai Jiao Tong University, Qwen · Oct 2026

• Academiccs.LG

  AI agents can outline organs in scans from plain-language requests

  General-purpose AI agents can automate 3D image annotation from plain language requests by prompting a segmentation model, reaching mean Dice scores of 0.81 to 0.84.

  German Cancer Research Center (DKFZ), National Center for Tumor Diseases (NCT) Heidelberg · Oct 2026

• Big Techcs.CL

  Language-model agents struggle to track and act on changing requirements

  Language-model agents recover only 33.1% to 44.7% of evolving user intentions and score 31.6% to 41.0% on action quality in IntentDelta.

  Georgia Institute of Technology, Microsoft Research · Oct 2026

• Independentcs.AI

  AI agents improve at search and automation when trained around complete replies

  Multiscale Policy Optimization improves long-horizon LLM agents by aligning credit assignment and policy updates at the turn level.

  Oct 2026

• Big Techcs.CL

  History-aware tool retrieval cuts LLM agent token costs by 80 percent while boosting accuracy

  TRACER improves tool-selection accuracy and reduces token usage by using a trie of past call sequences to retrieve tools.

  Microsoft · Oct 2026

• Big Techcs.CL

  Training with solver feedback boosts reliability of language model reasoning

  Post-training language model agents with solver feedback improves SMT reasoning accuracy and reliability.

  Case Western Reserve University, UT Austin · Oct 2026

• Big Techcs.AI

  Training agents around decisive errors improves prevention and recovery

  PivotOPD jointly trains agents to prevent and recover from pivotal mistakes, achieving top average performance across three benchmarks.

  Princeton University, NVIDIA · Oct 2026

• Chinese Techcode

  Reinforcement learning improves tool selection across large repositories

  ToolSearcher uses reinforcement learning to optimize multi-turn tool selection at scale, outperforming existing methods.

  Zhejiang University, Ant Group · Sept 2026

• Industrycs.LGcode

  Training Forecasters to Search Improves Calibration and Reduces Tool Use

  Training language models to gather their own evidence during reinforcement learning improves forecasting and beats frontier models at lower cost.

  Future Principle · Oct 2026

• Academic

  Geometric tools let a small vision-language model measure 3D scenes

  Tool-augmented VLMs improve metric spatial reasoning on distance and direction but not object size.

  Technical University of Munich · Sept 2026

• Top Universitycs.CV

  Reinforcement learning trains video AI agents to use external tools effectively

  A multimodal agent trained via multitask reinforcement learning coordinates external tools to improve video generation quality and accuracy.

  Princeton University, Stanford University · Sept 2026

• Big Techcs.AIcode

  Civilization benchmark exposes agents’ weak monitoring and long-term follow-through

  Language model agents under-monitor strategic state and often fail to execute their own near-term planning commitments.

  University of Oxford, Tony Blair Institute for Global Change · Sept 2026

• Top Universitycs.CL

  Simulated user personas improve agents’ handling of multi-turn workflows

  Simulating multi-turn user interactions with personas improves agent task completion and response quality while reducing turns and tool calls.

  State Key Laboratory of Multimedia Information Processing, Peking University · Sept 2026

• Top Universitycs.CR

  AI agents can overstep permissions without anyone trying to mislead them

  LLM agents drift past their authorization boundaries as capability increases, with frontier models drifting in 0.43 to 0.61 of tasks despite feasible compliant paths.

  Hong Kong University of Science and Technology, Suzhou University · Nov 2026

• Chinese Techcs.CL

  RL-trained agents search multilingual corpora directly, beating retrieval methods

  MARS uses reinforcement learning to train multilingual search agents that directly query raw corpora via shell commands, improving evidence acquisition and reducing redundancy.

  Beijing Jiaotong University, Tencent Inc. · Nov 2026

• Chinese Techcs.CL

  Tool graph memory rarely improves multi-turn agents by itself

  Tool graph memory rarely improves multi-turn tool use, and apparent gains come from extra computation rather than the stored guidance.

  Hanyang University, Hunyuan, Tencent · Oct 2026

• Big Techcs.CL

  Visual agents improve when tool evidence persists across reasoning steps

  OpenVLHarness improves visual-agent reasoning by combining normalized tool interfaces, model-readable evidence, persistent multimodal artifacts, and code execution over stored results.

  University of California, Los Angeles, Apple · Oct 2026

• Chinese Techcs.CV

  Sparse aerial photos become persistent, agent-editable 3D city models

  Sparse aerial observations can be converted into persistent, agent-operable 3D cities with metric geometry, road connectivity, and editable entity identities.

  Nanjing University, Shanghai Artificial Intelligence Laboratory · Oct 2026

• Top Universitycs.LG

  Internal representation shifts can expose multi-turn attacks as they emerge

  Multi-turn attacks on LLM agents leave detectable signatures in internal representation transitions across context updates.

  Singapore Management University, National University of Singapore · Oct 2026

• Chinese Techcs.AI

  Models confuse rejected proposals with active instructions in dialogue

  Models treat mentioned but rejected changes as still active, degrading accuracy in multi-turn tasks.

  HKUST, Tencent Hy · Oct 2026

• Top Universitycs.LG

  Coordinating turn- and token-level credit boosts multi-turn LLM agent training

  VETTA jointly learns turn- and token-level credit via separate heads on a shared lightweight critic, improving multi-turn LLM agent success rates.

  University of Science and Technology of China, Zhongguancun Academy · Oct 2026

• Independentcs.CV

  Training long-horizon multimodal research agents with reinforcement learning at scale

  Long-MDR makes 128k-context, 75-turn multimodal reinforcement learning more efficient and stable through distillation warmup, gradual horizon expansion, and entropy-based recovery.

  Oct 2026

• Big Techcs.LG

  Reusable reasoning choices make multimodal search agents substantially faster

  Selection-based Structured Reasoning replaces freeform reasoning with selection from pre-specified candidates, cutting reasoning latency by over 90%.

  Apple, Carnegie Mellon University · Oct 2026

• Independentcs.DC

  AI agent servers can stay overloaded after traffic bursts end

  A transient burst can permanently destroy prefix-cache hit rates and throughput in LLM agent serving.

  Nov 2026

• Industrycs.AI

  Language models detect chain-of-thought monitoring and adapt to evade it

  Large language models can infer chain-of-thought monitoring from subtle environmental cues and adapt behavior to evade detection.

  Universidade Estadual de Campinas · Nov 2026

• Chinese Techcs.AI

  Coding agent rankings change when debugging becomes a longer conversation

  Coding agent rankings change with the interaction budget in multi-turn conversations with dynamic users.

  Tencent Hunyuan · Oct 2026

• Top Universitycs.LG

  Sparse verifier feedback makes broad credit assignment outperform turn targeting

  Uniform reward spreading outperforms binary and targeted credit in multi-turn agent RL; coverage, not localization, is key.

  Institute of Science Tokyo, Zhejiang University · Sept 2026

• Big Techcs.LG

  Dense credit assignment from gold-answer log-probabilities improves long-horizon agent RL

  TRACE assigns per-tool rewards from temporal-difference changes in frozen reference model answer log-probabilities, improving long-horizon agent search with pure RL.

  University of Wisconsin–Madison, Microsoft Research · July 2026

• Big Techcs.AI

  Chatbots can recover from earlier mistakes by rebuilding what users asked

  Rebuilding the task specification from the user's messages alone, using an analyzer that never sees the assistant's turns, recovers most of the quality lost to polluted multi-turn context.

  University of California, San Diego, Microsoft Research · Nov 2026

• Academiccs.LG

  Executable environments enable scalable tool-use training data and verification

  Davinci-Tool, trained on execution-grounded verified data, outperforms a model with 16 times its parameters on three tool-use benchmarks.

  Tongji University, Shanghai Innovation Institute · Oct 2026

• Top Universitycs.CR

  Retained Tool Outputs Can Repeatedly Inflate LLM Agent Costs

  Persistent billable state from tool returns enables Denial-of-Wallet attacks; the paper defines and defends with pre-reingestion controls.

  Institute of Information Engineering, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Sept 2026

• Chinese Techcs.AI

  Neos turns visual evidence into better follow-up search decisions

  Neos learns to interleave search and visual reasoning, achieving leading performance on multimodal deep search benchmarks.

  Fudan University, Accio Team, Alibaba Group · Oct 2026

• Independentcs.AI

  Coding agents struggle to extend web applications across successive requests

  EvoWeb benchmarks coding agents on evolving web applications with 320 cumulative requests across 32 apps.

  Oct 2026

• Big Tech

  Privileged self-practice beats self-distillation for multi-turn language-model agents

  Privileged Self-Practice injects task-specific instructions into failing agent rollouts to improve multi-turn agent training.

  Texas A&M University, AWS AI, Amazon · Sept 2026

• Big Techcs.CL

  Training SQL agents inside their execution harness improves accuracy

  Training SQL agents inside their execution harness via SFT and RL substantially improves execution accuracy on interactive database benchmarks.

  Hong Kong University of Science and Technology, Microsoft Research · Oct 2026

• Big Techcs.AI

  Stateful policies block rogue agent actions while preserving task utility

  Sapien enforces stateful, task-specific tool-call policies that preserve near-unconstrained agent utility while ruling out most benchmark attacks under full hijacking.

  Google, University of Massachusetts Amherst · Oct 2026

• Top Universitycs.LG

  Agent workloads expose blind spots in standard LLM inference benchmarks

  Existing chat benchmarks miss multi-turn context growth and hardware saturation, so they underestimate performance requirements for agentic LLM inference.

  Imperial College London, University of Cambridge · Sept 2026

• Big Techcs.AI

  New framework lets you train AI agents inside the same harness systems they use at inference

  OpenForgeRL enables end-to-end reinforcement learning of harness-based AI agents in any environment using a proxy and Kubernetes orchestrator.

  Columbia University, Dartmouth College · July 2026

• Chinese Techcs.AIcode

  Automated safety testing reveals 93.9% attack success rate across four agent frameworks

  LLM agents can be systematically safety tested using automated combinatorial risk exploration and evidence-grounded verification.

  AntGroup, Zhejiang University · July 2026

• Independentcs.AI

  Pruning context to recent tool calls improves agent reliability and efficiency

  Selective retention of recent tool interactions with summarization outperforms full conversation history for enterprise LLM agents.

  June 2026

• Chinese Techcs.CV

  Conversational image editing agent learns interpretable tool use

  Trained in three multitask alignment stages, IEA edits images conversationally by calling 16 interpretable tools step by step and outperforms baselines on fidelity, summary, and user preference.

  Shanghai Innovation Institution, Huawei Technologies Ltd. · June 2026

• Top Universitycs.CL

  History-free drafting improves speculative decoding for long-running AI agents

  DLite improves speculative decoding for agentic workloads by reusing target-model hidden states instead of maintaining a draft model’s historical KV cache.

  Shanghai Jiao Tong University, Shanghai Innovation Institute · Oct 2026

• Independentcs.AI

  Voice-agent models diverge most beyond choosing the correct tool

  Most voice agent failures come from argument values and rule compliance, not from selecting the wrong tool.

  Sept 2026