Long context windows do not ensure reliable sustained execution

Across seven open-weight models, performance deteriorated independently with longer contexts, harder operations and less structured input formats.

Big Tech
Jeffrey Willette · Krishna C. Puvvada · Boris Ginsburg

NVIDIA

Research Digest··2 min read

In one line

Long-Transduction reveals that models suffer 62.8% accuracy loss when context length scales to 128K tokens in long-horizon agentic workflows.

Before you read this

Three ideas this paper builds on
Read it as

The paper introduces Long-Transduction, a diagnostic to test whether language models can sustain repeated state-dependent operations (arithmetic, sorting, variable lookups, table transformations) over long generations. The benchmark independently varies context length, input format, and local task complexity to isolate failure modes.

Evaluating seven open-weight models, the authors find a 62.8% accuracy decrease when scaling context from 4K to 128K tokens, a 36.5% decrease when input format changes (e.g., ordered vs. shuffled record IDs), and a 39.9% decrease when local task complexity increases (e.g., more operands or dictionary entries). These results show that models fail to stay on task even for simple operations, exposing critical reliability limits in long-horizon agentic systems.

How it works

before and after
STANDARD LONG-CONTEXT EVALUATIONLong Input: up to 128K tokensLong Inputup to 128K tokensModel: standard inferenceModelstandard inferenceFinal Score: overall accuracyFinal Scoreoverall accuracyLONG-TRANSDUCTION DIAGNOSTICvariedinputsgeneratesrecordsproducesscoresTask Design: 4 families w/ controlledTask Design4 families w/controlledModel: open-weight modelsModelopen-weightmodelsRecord Scoring: per-record alignmentRecordScoringper-recordalignmentFailure Metrics: error types & locationsFailureMetricserror types &locationsInput formats varied: ordered IDs, shuffled IDs, or no IDs.
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe