, a list of assignments like 'K = apple, B = K, D = B, print(D)').
A tiny LoRA intervention unlocks deep reference-following in transformers
Thirteen base models reliably follow only 1.4–3.6 lines; a rank-8 LoRA at one early layer boosts exact accuracy on 24-line chains from 15.5% to 99% in Qwen3-8B.
Academic
Zehao Jin · Ruixuan Deng · Junran Wang
Georgia Institute of Technology
Research Digest··2 min read
Thread:LoRA Optimization Geometry
The authors show that pretrained transformers underuse their depth for in-context reference following, with thirteen models reliably tracking only 2-3 lines.
Why this paper
From Georgia Institute of Technology · Part of LoRA Optimization Geometry, now 5 papers
In one line
A tiny rank-8 LoRA at one early layer lets pretrained transformers follow 24-line reference chains with 99% accuracy instead of 15.5%.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks (3 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§