5 Super-120B-A12B.
Long-context midtraining improves coding agents after identical supervised tuning
A documented recipe combining natural software-engineering data with limited agent traces improved two Nemotron models across coding benchmarks and agent harnesses.
Big Tech
Soumye Singhal · Joseph Jennings · Ka Shun Shum · Alexander Bukharin · Khushi Bhardwaj · Zirui Liu · +8 more
NVIDIA
Research Digest··3 min read
Singhal and colleagues insert a 256K-context training stage between conventional pretraining and supervised fine-tuning, or SFT, using hundreds of billions of tokens from software repositories, development histories and agent interactions.
Why this paper
From NVIDIA
In one line
A midtraining stage of natural software data and a small share of agentic traces primes base models for agentic post-training, improving benchmarks on two Nemotron models.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (4 noted)
- ✓Reports numbers on named benchmarks (5 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§