HERO addresses two linked problems in program generation.
Execution-verified feedback improves LLM harness optimization for SQL, Cypher, and Python generation
HERO combines offline prompt revision validated by execution results with test-time refinement of generated programs, improving pass@1 by 7.6 points offline and 2.0 additional points at test time.
Big Tech
Mingyu Lu · Timothy Ossowski · Tristan Naumann · Hoifung Poon · Peniel Argaw
University of Washington · Microsoft Research · Recursion
Research Digest··2 min read
The authors introduce HERO, a two-stage framework for optimizing the harness (prompts and scaffolding) around LLM program generation.
Why this paper
From Microsoft Research and 2 others
In one line
HERO improves pass@1 by 7.6 points offline and an additional 2.0 points via test-time refinement on text-to-code benchmarks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§