6B to 14B, plus three Llama models, using a dense grid over the learning rate and the reference-policy regularisation parameter beta.
DPO hyperparameters do not transfer across model scales or datasets; validation loss is unreliable
A dense sweep across five Qwen3 and three Llama models shows learning-rate optima shift with size and configuration rankings vary sharply between neighboring models.
Industry
Hannan Mahadik · David Salinas · Jacek Golebiowski · Aaron Klein
ELLIS Institute Tübingen · Prior Labs · distil labs
Research Digest··3 min read
The authors ran dense hyperparameter sweeps for direct preference optimisation (DPO) across five Qwen3 model sizes and three Llama models, evaluating on eleven benchmarks and two LLM judges.
What it released
Code
§