Companies claim new coding models are better because they score higher on tests. But no study had checked if that means developers actually get more code accepted into projects.
When a top model was banned for 19 days, users were forced to switch to a lower-scoring model. Researchers tracked their pull requests, proposed code changes, and compared merge rates, the share of requests accepted within a week, to a control group that never switched.
The higher-scoring model raised merge rates by only 1.4 percentage points. This change was not statistically significant, meaning benchmark improvements did not translate into more real work.
