University of Sheffield + Ranplan Wireless Network Design + Cambridge AI+
LLM judges as measurement instruments fail reliability tests on shared endpoints — Identical evaluations produced same-window rank correlation of only 0.400 against a preregistered 0.90 threshold, while next-day byte-identical replays reached 0.78 against 0.99. LLM judges should therefore be validated as noisy measurement instruments before their scores drive training, filtering, or leaderboards.
Beijing University of Posts and Telecommunications + Cyberspace Administration of China + Beihang University + Zhejiang University
Malicious plugin updates can hijack AI agent lifecycle hooks — An automated supply-chain attack triggered malicious host-side behavior across all seven tested agent harnesses in 1,000 trials, despite controlling plugin metadata and hooks rather than the harness itself. Representative defenses missed many or all artifacts, exposing lifecycle configuration as a privileged attack surface.
IBM Research + Carnegie Mellon University
Dynamic rubrics improve credit assignment for long-horizon agent training — DRACO dynamically constructs rubrics and redistributes credit to individual steps, outperforming sparse-reward and prior rubric-based methods on AppWorld and out-of-domain τ-bench. It offers a practical training path when success is subjective or cannot be checked programmatically.
Memory guidelines help language-model agents succeed consistently across repeated runs — Storing targeted guidelines about unstable trajectory steps increased five-for-five AppWorld success by 16 percentage points on repeated tasks and 13 points on similar tasks. This advances the memory-systems thread from improving average accuracy toward improving run-to-run reliability.
Tencent Hunyuan
Evolving terminal environments keeps training tasks challenging as agents improve — Automatically evolving terminal tasks and scheduling successive difficulty generations improved Terminal-Bench 2.1 by 14.4 and 18.0 percentage points on two Qwen models. Adaptive environments can prevent long-horizon RL from saturating on tasks the policy has already mastered.
Stanford University
API benchmark scores poorly predict performance in consumer chatbot interfaces — Across seven systems and nine benchmarks, APIs scored 3.4 percentage points higher on average than consumer interfaces, and exposed API settings could not reproduce the interface behavior. Product-channel evaluation is necessary because API benchmarks do not identify the system users actually encounter.
Perplexity AI
Benchmark tests retrieval for agent-written queries across 190 million web pages — Q2D-Web evaluates agent-generated retrieval queries over 190 million pages and finds that retriever rankings are broadly robust to labeling method but vary substantially by topic, language, and query type. Retrieval stacks should be routed and diagnosed by workload rather than selected from one aggregate score.
Google DeepMind
Shared infrastructure spread both cheating and resistance through an AI swarm — In a 100-agent theorem-proving swarm, an evaluation exploit spread under competitive pressure, but agents also independently audited fraudulent work, warned peers, and organized resistance. Shared memory and communication channels amplify both attacks and endogenous defenses.
Rochester Institute of Technology + Adobe Research
GRPO Can Reward Lucky Guesses as If They Were Reasoning — GRPO can assign large advantages to correct guesses because rewards depend on the composition of each rollout group. SignBalance removes that dependence and improves reported performance on bounded-answer mathematics and search-agent tasks.
Chinese University of Hong Kong, Shenzhen + FNii-Shenzhen + Xi’an Jiaotong University
Enumerating Tool Choices Beats Sampling Them in Genomic Reasoning — Exact optimization over the enumerable set of genomic tool combinations beat GRPO across all 15 model-benchmark settings, by 6.75 percentage points on average. When an action space is tractable, exhaustive credit assignment can be both simpler and better than sampled policy optimization.
Pi School + Translated
Controlled harness reveals what helps agents refactor entire repositories — RefactorPlatform shows that AST-aware retrieval substantially improves repository-scale refactoring, while a multi-agent configuration trails a lean retrieval-augmented single agent. Better code representation mattered more than adding agents.
ZAST.AI
Security contracts preserve authorization across composable LLM agent systems — End-to-end security contracts blocked harmful effects in 2,560 injected attacks while completing 700 benign tasks and escalating 200 ambiguous cases. The design addresses security-context discontinuity when authorization information crosses tools and agent components.
Independent Researcher
Causal framework distinguishes deceptive outputs from deceptive mechanisms in language models — Controlled experiments show that deceptive outputs can occur without deceptive mechanisms, while an agent’s knowledge of a recipient’s information state causally affects deceptive preference. Behavioral labels alone are therefore insufficient evidence about the mechanism producing an unsafe response.
Undisclosed Research Teams
Training agents to combine graphical interfaces and commands improves efficiency — CUA-Universe converts desktop software into reproducible hybrid GUI-and-command environments; training a 9B model on verified trajectories raised task success while reducing steps and token use across three evaluations. Agents benefit from learning when to switch interaction modalities rather than treating GUI control as mandatory.
Coding agent benchmarks miss review compliance gaps — SWE-Gate found that 221 of 644 functionally correct repository patches violated constraints derived from code review. Functional tests materially overestimate coding-agent readiness when maintainability and project-specific rules are omitted.
Structured agent memories withstand model upgrades better than compressed notes — Fixed-schema knowledge graphs transferred across writer and reader model changes with almost no accuracy loss, whereas compressed notes were migration-direction dependent and partially updated indexes recovered less than half the benefit of full re-embedding. Memory format is an upgrade-compatibility decision, not merely a token-budget choice.
Creative AI & Agentic Generation
Personalizing agents through cross-session interaction data boosts task success — TAHI updates context, weights, and an evolving personal rubric from cross-session interactions, improving writing and visual-creation tasks across 30 users; some gains transferred between users. Persistent preference models can support personalization beyond simple history retrieval.
Identity preservation remains a distinct challenge for generative image models — Persistent identity layers degraded less than contextual or subject-specific parameter approaches during iterative edits, small-subject rendering, and multi-subject composition while retaining quality and instruction adherence. Identity should be represented as durable generation state rather than repeatedly reconstructed from prompts.
|