Data selection for language model pretraining faces a trade-off between quality and diversity. Standard quality filtering often causes diversity collapse by favoring data similar to high-quality references, which hurts performance on underrepresented domains. Existing diversification methods are either indirect or computationally expensive.
Leverage Score Sampling (Lev) directly optimizes geometric diversity by iteratively selecting samples that maximally expand the determinantal volume of the embedded data. It uses leverage scores, a computationally efficient criterion that avoids costly covariance matrix recomputation, enabling scalable selection.
Experiments show Lev delivers up to 72x speedup over the strong baseline DiSF, improves dataset diversity (Vendi score) by 9.2%, and raises accuracy on seven CommonCrawl tasks by up to 1.31%. On code data (StarCoderData), Lev reduces bits-per-byte by 3.08% over DiSF. The paper also finds that quality filtering via DCLM-fastText fails to retain sufficient code content, leading to worse code performance compared to Lev-selected data.