Most harness-evolution gains come from overfitting or extra compute

Across 21 successful configurations, the authors attribute an average 54% of improvement to benchmark overfitting and 14% to increased inference-time computation.

Big Tech
Wenxuan Ding · Qin Lu · Changlong Yu · Sha Li · Shuowei Jin · Xin Liu · +2 more

Amazon · New York University

Research Digest··2 min read
Ding et al.

The authors evaluated Meta-Harness, SkillOpt and AHE using 11 executor models.

§