Agent Harness Optimization
Techniques for optimizing the code, instructions, and training frameworks surrounding LLM agents to improve performance and reduce cost without changing model weights.
235 papers · 5 months
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
Results across this thread
58 reported results from the papers in this thread.
The table is for subscribers. Subscribe to see every reported number side by side.
How this thread developed
May 2026 · Columbia University, Google
Proposing a workflow store to harden AI agents against failure
Proposes a workflow store as an alternative to on-the-fly planning to harden agents against failure.
June 2026 · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Tongyi Lab , Alibaba Group
Evolutionary training harness co-evolves with LLM policies for RL
Proposes co-evolution of training harness and LLM policies, treating the harness as an evolvable component.
released code
3 further papers
June 2026 · TokenRhythm Technologies, Infinigence AI
New benchmark standardizes evaluation of coding agent harnesses
Benchmark and adapter protocol for evaluating coding agent harnesses, revealing adapter design impact.
released code, data
June 2026
Latent communication across different LLMs matches or beats text-based transfer at lower compute.
Latent communication via KV-cache alignment reduces compute compared to text-based transfer between heterogeneous LLMs.
3 further papers
July 2026 · Allen Institute for AI, University of Washington
Re-evaluation shows harness evolution for agents may not outperform simple test-time scaling
Re-evaluation showing harness evolution may not outperform simple test-time scaling.
released code
2 further papers
July 2026 · Columbia University, Dartmouth College
New framework lets you train AI agents inside the same harness systems they use at inference
Framework for training agents inside the same harness used at inference.
Aug 2026 · Google DeepMind
Cost-aware model routing matches exhaustive estimation with fewer expensive checks
Models AI routing as search with inspection costs, achieving cost-aware routing across multiple LLM setups.
3 further papers
Aug 2026 · AWS
Switching Models Mid-Task Imposes a Cost and Quality Penalty
Quantifies the 'handoff tax' in mid-task model switching, informing harness design decisions about model selection.
4 further papers
Aug 2026
Separating state tracking from actions improves multi-turn tool reliability
Introduces a structured tool-use policy that separates state tracking from actions, improving reliability.
3 further papers
Aug 2026 · Tencent
Self-improving context programs help frozen models understand long videos
Treats context construction as an optimizable outer-loop program, improving performance of frozen models on long-video understanding without weight changes.
Aug 2026 · The Chinese University of Hong Kong, Individual Researcher
Self-evolving harness turns general AI agents into stronger RCA specialists
Demonstrates harness that learns from previous diagnoses to specialize general agents for root cause analysis, advancing harness optimization for domain adaptation.
Aug 2026 · LV-NUS Lab
Generating Agent Harnesses on Demand Improves Models Across Benchmarks
Shows that generating task-specific harnesses improves performance across benchmarks, a direct advance in harness optimization.
Aug 2026 · X-LANCE Lab, Shanghai Jiao Tong University
Structured software interfaces outperform screenshot-and-click control for AI agents
Replaces screenshot-and-click with structured JSON interfaces, optimizing agent tool use and runtime interaction for better performance.
1 further paper
Aug 2026 · Dartmouth College, University of Cambridge
LLM agents struggle to infer wellbeing from long-term wearable data
Benchmark for wellbeing prediction from wearable data evaluates agent temporal reasoning.
1 further paper
Aug 2026 · New York University
Frontier LLMs design competitive algorithms for well-specified operations problems
LLMs design algorithms for operations problems in sandbox environment.
4 further papers
Sept 2026 · Infineon Technologies AG
Tool-call traces expose extraction failures that source fidelity misses
Tool-call trace analysis for failure attribution in datasheet extraction.
1 further paper
Sept 2026 · DreamX Team, Alibaba Group
Models remain weak at steering coding agents through long tasks
Benchmark revealing controller weakness in steering coding agents.
released code
4 further papers
Sept 2026 · Stanford University
API benchmark scores poorly predict performance in consumer chatbot interfaces
Shows that consumer chat interfaces yield lower accuracy than API testing, highlighting harness effects on agent evaluation.
2 further papers
Sept 2026
Refresh schedules, not cache age, determine agent answer staleness
Optimizing refresh schedules improves answer freshness and system efficiency.
2 further papers
Sept 2026 · UMass Amherst, Zoom Video Communications
Coding-agent harness choices should match model skill and context budget
Shows that harness choices (planning, action space, context management) must match model skill and context budget, guiding harness optimization.
1 further paper
Sept 2026 · The Chinese University of Hong Kong, University of Edinburgh
Task-specific plans and terminal checks create different value for LLM agents
Comparing task-specific plans against shuffled text reveals how meaningful planning signals add value, informing instruction design in agent harnesses.
2 further papers
Sept 2026 · Northeastern University, Seattle
Serving stacks can distort local language-model tool-use evaluations
Identifies and analyzes how serving-stack rejections can distort tool-use evaluations, offering insights for improving agent harness reliability.
Sept 2026 · Nanyang Technological University
Typed selective control cuts strong-model calls while preserving agent success
Introduces a selective control architecture that preserves agent success while drastically reducing calls to strong models, optimizing harness efficiency.
2 further papers
Sept 2026
Benchmark tests whether coding agents improve governance without breaking repositories
Provides a benchmark for assessing coding agent improvements to governance without breaking repositories.
2 further papers
Sept 2026 · National University of Singapore
DynBranch starts and reuses agent work before branches resolve
Introduces DynBranch, which starts and reuses agent work before branch resolution, reducing latency by up to 32%.
6 further papers
Sept 2026 · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences
Agents Learn Reusable Control Code Instead of Rebuilding Context
Trains an agent harness from execution failures to learn reusable control code, reducing context rebuilding.
Sept 2026 · The Chinese University of Hong Kong, New Laboratory of Pattern Recognition (NLPR)
More Harness Programs Do Not Necessarily Produce Useful Specialization
Shows that generated harness programs may not provide true specialization, informing harness design.
released code
3 further papers
Oct 2026 · Carnegie Mellon University, AWS AI Labs
Models Learn to Allocate Context and Scale Inference More Effectively
Presents Hermes harness that gives models control over context allocation, advancing harness design.
Oct 2026 · Georgia Institute of Technology, AWS AI Labs
MILO co-evolves agent harnesses and the strategies that discover them
Co-evolves harness designs and discovery strategies, enhancing both frontier and open-weight models.
1 further paper
Oct 2026 · Xi’an Jiaotong University
Agents can budget external checks without sacrificing task quality
Formulates external state checking as a resource allocation problem, improving efficiency without quality loss.
Oct 2026 · POSTECH, KAIST
Adaptive training curricula produce stronger optimized agent harnesses
Treats scenario selection as an adaptive curriculum within agent harness optimization, improving efficiency and coverage
1 further paper
Oct 2026 · Nanyang Technological University
Agent performance depends on matching models, harnesses and tasks
Studies how matching models, harnesses, and tasks affects agent performance, highlighting harness importance.
released code, data
Oct 2026 · The Chinese University of Hong Kong, Shenzhen, Tianjin University
Evidence-driven harness edits improve frozen GUI agents across benchmarks
Introduces automated harness edits that improve frozen GUI agent performance across benchmarks.
released code
1 further paper
Oct 2026 · Harvard University
Simple LRU rivals sophisticated eviction for LLM prefix caching
Demonstrates that LRU is competitive with sophisticated eviction policies, reducing inference costs for agent infrastructures.
Oct 2026 · Amazon, Columbia University
Canopy learns when tree-based shortcuts fail in LLM inference
Proposes CANOPY, a multi-fidelity bandit algorithm for tree-structured LLM inference search that reduces cost without assuming similar branch values.
4 further papers
Oct 2026 · Meta AI, Columbia University
Separating research choices from execution improves long-horizon agent performance
improves long-horizon agent performance by decoupling investigation decisions from execution steps
2 further papers
Oct 2026 · Tencent, Fudan University
Workspace-grounded synthesis scales verifiable training for long-horizon work agents
introduces WorkForge pipeline for creating verifiable training environments for long-horizon workspace agents
Oct 2026 · Nankai University, Huawei Technologies Co., Ltd.
Diagnostic tools speed agentic root cause analysis, but accuracy varies
Controlled experiments on tool designs for agentic root cause analysis show diagnostic recommendations speed diagnosis.
3 further papers
Oct 2026 · Samsung Semiconductor, Rice University
Personalized LLMs can share most adaptation capacity across users, needing only tiny per-user code.
Finds that most adapter capacity in personalized LLMs can be shared across users, enabling tiny per-user code
Oct 2026 · Amazon, Apple
Separating agent checks from reasoning improves weaker models’ reliability
Separates action approval and termination decisions from the reasoning model, improving weaker models' reliability on agent benchmarks.
1 further paper
Oct 2026 · Fastino Labs
Commercial LLM routers fail to beat carefully chosen random routing
evaluates commercial LLM routers and finds they fail to beat carefully chosen random routing, highlighting limitations of current routing methods.
2 further papers
Oct 2026 · Shandong University
Coding harnesses change token costs more reliably than success rates
Shows coding harnesses affect token costs more reliably than success rates on SWE-bench, highlighting harness effects.
1 further paper
Oct 2026 · AWS AI Labs, Georgia Institute of Technology
Agent evolvers can learn their own search strategies across tasks
Learns to optimally allocate resources and search over agent harness candidates, directly optimizing the surrounding training framework.
1 further paper
Oct 2026 · Harbin Institute of Technology (Shenzhen)
Bidirectional protocol connects LLM agent workflows with inference engines
Proposes a bidirectional protocol between agent harness and inference engine to improve workflow efficiency and output quality.
Oct 2026 · University of California San Diego, Meta AI
Personal LLM agents improve recommendations using cross-platform history
uses cross-platform history to mediate platform recommendations, improving personalization
Oct 2026 · Amazon Web Services, University of the Bundeswehr Munich
Step-by-step process delivery boosts LLM predictability and auditability
optimizes instruction delivery structure for agent compliance
4 further papers
Oct 2026 · Tsinghua University, Huawei Noah's Ark Lab
Learning the refinement loop improves web generation beyond final pages
Introduces WebLoop framework training generator, critic, and refiner as roles of a shared policy for functional web generation.
3 further papers
Oct 2026 · Microsoft Research, University of Maryland
Code understanding, not edit volume, limits coding agents
Provides a controlled task generation framework (CABRA) to isolate and measure code understanding vs edit volume, showing understanding is the key bottleneck in coding agents.
6 further papers
Oct 2026
CopRoute reduces inference calls by exploiting complementary failures among language models
Orders models by complementary coverage and calibrates a conformal cutoff to reduce inference calls.
Oct 2026 · Amazon
Expert feedback for LLM agents should target decisions, not whole tasks
Allocates expert feedback to individual decision points, improving agent accuracy at lower annotation cost.
2 further papers
Oct 2026 · FAIR at Meta, Harvard
Shared message search lets LLM agent groups scale without flooding readers
this paper introduces a shared datastore with search and hyperlinks to scale multi-agent communication without information overload.
Oct 2026 · Riemann Lab, Huawei Technologies
Later LLM handoffs rarely beat choosing the model upfront
provides empirical evidence that later model handoffs in multi-turn tasks rarely outperform upfront model selection, informing agent harness design.
2 further papers
Oct 2026 · Apple
Tool discovery paradigm crossover: Retrieve-then-Plan wins on simple queries, Plan-then-Retrieve on complex ones at scale
Compares Retrieve-then-Plan and Plan-then-Retrieve tool discovery and shows the optimal paradigm flips with query complexity.
4 further papers
Oct 2026 · NVIDIA
Long-context midtraining improves coding agents after identical supervised tuning
Shows that inserting a long-context midtraining stage between pretraining and SFT improves coding agent performance.
3 further papers
Oct 2026 · Hanyang University, Hunyuan, Tencent
Tool graph memory rarely improves multi-turn agents by itself
Examines the utility of tool graph memory as an agent harness component, revealing negligible improvements under current designs.
2 further papers
Oct 2026 · IDSIA (SUPSI), DFKI
Recovered design docs guide agents to rebuild original module structure, not test results.
recovers design docs to guide agents in understanding code structure, improving context optimization
released code, data
8 further papers
Oct 2026 · Carnegie Mellon University
Fast AI agents read states correctly yet still choose wrong actions
Provides insights into state reading errors to inform agent instruction and framework optimization.
3 further papers
Oct 2026 · Amazon, New York University
Most harness-evolution gains come from overfitting or extra compute
Provides Harness-Delta Attribution to diagnose why harness evolution works.
2 further papers
Oct 2026 · Northeastern University, Alibaba Group
Visual skill representations reduce exposure-driven action bias in language model agents
Identifies and mitigates exposure-driven action bias using visual skill representations.
Oct 2026
CUA-Lite standardises computer use agent development across environments and models
Standardises abstractions for computer use agents, reducing development overhead across environments.
Oct 2026 · University of Illinois Urbana-Champaign, AWS AI Labs
Global coordination of proof agents beats local repair in formal theorem proving
Introduces a coordinator-worker protocol where structured failure reports replace local subgoal repair in Lean proof development.
3 further papers
Oct 2026 · George Washington University, Amazon Web Services
Predicted GUI screens add little beyond direct action ranking
Separates screen prediction from action ranking in GUI agents, finding predicted screens provide no additional ranking benefit.
Oct 2026 · IBM Research
Token accounting shrinks evolutionary search’s apparent advantage over sampling
Reevaluates the apparent advantage of evolutionary search over sampling by accounting for token cost, offering a practical insight for optimizing agent search strategies.
1 further paper
Oct 2026 · University of California, Davis, AWS AI Labs
Governing agents' evidence with deterministic harness rules beats free-form trajectories
Proposes AgInt, a deterministic harness that governs evidence admission and grading, outperforming free-form trajectories.
1 further paper
Oct 2026 · FuriosaAI, Yonsei University
Agent harness optimization mostly improves execution reliability, not task reach
Systematically distinguishes reliability improvements from task reach improvements, showing harness optimization primarily enhances execution reliability.
2 further papers
Oct 2026 · Amazon
Formatting loss in document ingestion makes agents confidently wrong
Exposes formatting loss in document ingestion as a source of confident errors, motivating optimization of the document handling pipeline in agents.
Oct 2026 · Columbia University, Google Inc.
Coding agents often miss crashes and misread changes that prevent them
reveals that coding agents miss crashes and misread changes, highlighting evaluation weaknesses.
4 further papers
Oct 2026 · Amazon, Technische Universität Dresden
Models can build their own agent software, but improvements generalize unreliably
Tests whether language models can improve their own interaction software without human code edits, showing unreliable generalization.
2 further papers
Oct 2026 · ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems
Context compaction spares benchmark scores but silently drops task constraints
Shows that context compaction strategies silently drop task constraints, revealing a failure mode in agent optimization.
7 further papers
Oct 2026 · Apple
AI agents learn more from varied tools than extra tasks
Shows that allocating synthetic training data to tool diversity outperforms task diversity for agent learning.
Oct 2026 · The Hong Kong University of Science and Technology (Guangzhou), The Hong Kong University of Science and Technology
AI agents can drop from 85% to 19% success with rearranged information
Shows post-training causes action preferences to become specific to the training harness, leading to large drops when observation layout changes.
7 further papers
Oct 2026 · Kyutai, Meta FAIR
Language models gain from varied answers, not necessarily altered weights
Shows that adding random biases to model forward pass can improve reasoning via voting, a low-cost harness-level enhancement.
9 further papers
Nov 2026 · Scale AI
Small decision models can match large language model judges at a fraction of the cost
Reduces evaluation cost by replacing large language model judges with lightweight decision models in agent pipelines.
Nov 2026 · Hunan Agricultural University
Prompt optimization yields no gains on strong reasoning models
Negative result bounding the harness line: automated prompt compilers give no statistically significant gains on frontier reasoning models.
2 further papers
Nov 2026
Terminal prompts make AI agents report success without doing the work
Reveals that command-line confirmation prompts cause agents to misreport success, a harness design gap for interactive tool execution.
2 further papers
Nov 2026 · University of Washington, NVIDIA
Simple prompt trick stops AI agents from faking progress in robot tasks
Decouples description and assessment in VLMs to eliminate confirmation bias in robot task evaluation.
6 further papers
Nov 2026 · UC Berkeley, Bosch
Robot coding agents learn faster with smarter feedback, not more data
Shows that optimizing feedback type and training curricula improves robot coding agent skill acquisition more than scaling data.
Nov 2026 · Mila – Quebec AI Institute, Université de Montréal
Small model predicts per-query cost of AI reasoning models, saving up to a third on compute
Introduces a scout model that predicts per-query compute cost to selectively skip expensive reasoning steps, saving up to a third on inference.
1 further paper
Nov 2026 · Amazon Web Services
AI agents can keep needed context yet fail to use it
Reveals that context availability does not guarantee its use by agents, highlighting a key gap in agent system design.
3 further papers
Nov 2026 · Zhejiang University, University of Oxford
External workflow control keeps AI agents on procedure, letting small models beat large ones
Uses an external finite-state machine to control agent workflows, preventing drift and allowing small models to outperform larger ones.
80 of 235 papers shown