Combining separately fine-tuned AI models ("experts") into one is a key step in building capable systems. A popular fine-tuning technique, LoRA, introduces a "rank" setting that controls an expert's capacity. Current practice forces this rank to be the same for all experts.
The researchers trained experts on three domains (mathematics, instruction following, and multilingual) using different ranks. They merged these experts both with a uniform rank and with each expert at its own best rank.
For simple averaging, using diverse ranks improved accuracy by up to 1.8 percentage points. For conflict-resolution methods like TIES, the same change cost up to 4 points. The constraint is not in merging itself, but in methods that select components by magnitude. Choosing a rank-insensitive merge method removes the need for rank coordination.