The paper introduces Long-Transduction, a diagnostic to test whether language models can sustain repeated state-dependent operations (arithmetic, sorting, variable lookups, table transformations) over long generations. The benchmark independently varies context length, input format, and local task complexity to isolate failure modes.
Evaluating seven open-weight models, the authors find a 62.8% accuracy decrease when scaling context from 4K to 128K tokens, a 36.5% decrease when input format changes (e.g., ordered vs. shuffled record IDs), and a 39.9% decrease when local task complexity increases (e.g., more operands or dictionary entries). These results show that models fail to stay on task even for simple operations, exposing critical reliability limits in long-horizon agentic systems.