The authors evaluated Meta-Harness, SkillOpt and AHE using 11 executor models.
Most harness-evolution gains come from overfitting or extra compute
Across 21 successful configurations, the authors attribute an average 54% of improvement to benchmark overfitting and 14% to increased inference-time computation.
Big Tech
Wenxuan Ding · Qin Lu · Changlong Yu · Sha Li · Shuowei Jin · Xin Liu · +2 more
Amazon · New York University
Research Digest··2 min read
Ding et al.
§