Long-context understanding with LLMs is hard because relevant evidence is sparse and scattered among irrelevant information. The authors propose a method called Highlight-Then-Summarize (H2S) that first identifies and highlights question-relevant evidence from a long document, then summarizes that evidence into a compact representation, and finally answers the question. They trained a 14B parameter model (H2S-14B) on a new dataset and evaluated it on a seven-task benchmark. H2S-14B outperformed larger open-source models and retained high performance even with a very small output budget, indicating that explicit evidence compression leads to better and more efficient reasoning.
Selecting evidence before summarizing improves long-context reasoning efficiency
A 14-billion-parameter model trained to extract and compress relevant passages outperformed larger open-source models while using shorter outputs.
Chinese Tech
Zhaoyuan Xia (Peking University) · Qinghongbing Xie (Tsinghua University) · Yung Xiang Hue (Tsinghua University) · Jianguang Jiang (Baidu Inc) · Gaofeng Lu (Baidu Inc) · Zhenyu Jiao (Baidu Inc) · +4 more
Research Digest··2 min read
In one line
Highlighting and summarizing relevant evidence before answering improves long-context reasoning and allows more compact output.
Before you read this
Three ideas this paper builds oncontext compactionCompressing or summarizing an LLM's past context into a smaller representation to extend memory and improve reasoning within limited token budgets.79 papers →evidence retrievalPulling specific, relevant information from a large corpus to support an answer, decision, or reasoning step.107 papers →process-level evaluationA granular assessment of a system's intermediate steps or subgoals, rather than just its final output, to diagnose specific failures.75 papers →
Read it as
Xia et al. train a model to handle long inputs in three explicit stages: identify question-relevant evidence, combine it into a compact summary, then answer.
The authors developed Highlight-Then-Summarize, or H2S, a compress-then-reason procedure for documents, conversations, and code.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.