When training language models with automatic checks, multiple requirements must be satisfied. Combining all checks into a single score discards useful information about partial progress.
This work introduces POLPO, which uses Pareto dominance: one response is better only if it passes every check the other passes and at least one more. It learns from these comparisons without forcing a single ranking.
POLPO achieves higher rates of complete requirement satisfaction and learns more efficiently than methods using aggregated scores. It works for code generation and instruction following.
