Code search benchmarks often claim that dense encoders (neural models that turn code into numerical vectors) outperform simpler lexical search like BM25 (a keyword-matching algorithm). Yet many rely on grep and succeed, suggesting the advantage may be an illusion.
The authors built RepoPremium, a more realistic benchmark using real repositories and pull-request queries. They compared dense encoders against BM25 under fair conditions: indexing whole files instead of chunks and excluding test files from relevance labels. They also tested adding rare code identifiers to queries using a large language model (LLM).
Under fair conditions, BM25’s disadvantage dropped from 0.121 to 0.034 in the nDCG@10 metric. When queries contained the code’s own rare words, BM25 matched dense retrieval. This shows the dense advantage is largely a measurement artifact; simpler methods can be just as effective when evaluated properly.
