When developers use (LLMs) to write code, they usually check only whether the code is correct, not how fast or efficient it runs. In production, inefficient code wastes time, memory, and energy, but most existing ignore these factors or test only a few programming languages.
PerfArena is a new, open-source system that solves this by running each LLM's code in a controlled environment and measuring three things: how long it takes to run (latency), how much memory it uses at its peak (peak memory), and how much energy it consumes. It currently tests 13 LLMs across 7 languages using two sets of coding problems: 99 problems from LeetCode and 10 heavy computation tasks from the Computer Language Benchmarks Game.
The results show that correct code is not always efficient. The slowest LLM's correct solutions took on average 1.6 times longer than the fastest on the LeetCode problems, and 3.6 times longer on the heavier tasks. The fastest LLM varied by language, so developers should pick an LLM based on its per-language performance rather than just correctness or cost.
