A language model that answers questions over long documents updates a small internal memory with every chunk of text it reads. But when the model treats all chunks equally, irrelevant information crowds out what is needed for the answer, making the memory worse. Researchers developed Select-TTT, a method that scores each chunk's relevance to the question using the model's own existing output layer, then writes relevant chunks more strongly. On two long-context question answering tests, Select-TTT consistently outperformed the uniform write approach and the baseline model.
Query-guided memory updates improve language models’ answers over long contexts
Select-TTT uses a model’s existing output probabilities to prioritize relevant context during test-time weight updates, improving question answering across two benchmarks and two model backbones.
Chinese Tech
Yue Wu · Yang Zhang · Weinan Gan · Chongming Gao · Xinsen Guo · Xiaoyuan Li · +1 more
University of Science and Technology of China · National University of Singapore · Huawei Noah’s Ark Lab
Research Digest··2 min read
In one line
Query-conditioned selective writes improve In-Place test-time training for long-context question answering.
Before you read this
An idea this paper builds onRead it as
Wu and colleagues report that updating a language model equally on every chunk of a long context can dilute information needed to answer a particular question.
The authors built on In-Place Test-Time Training, which turns existing projection matrices inside a language model into fast weights: parameters updated while processing incoming context.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.