The authors constructed DivAlgo, a large-scale dataset containing algorithmically distinct human solutions for programming problems from CodeContest+.
LLMs capture only a fraction of human diversity in code generation tasks
New benchmark DivAlgo reveals that even advanced strategies achieve only 70% of human algorithmic diversity.
Top University
Anubhav Jangra · Smaranda Muresan
Columbia University · Barnard College
Research Digest··2 min read
Jangra and Muresan introduce DivAlgo, a benchmark of algorithmically diverse human solutions to programming problems, and evaluate 24 large language models (LLMs) for code generation.
§