Small language models are often used as judges in agent systems, scoring a response against a criterion. But when training on pointwise data, where each response is labeled independently, the model sees responses that differ in many ways at once, so it cannot easily learn which features actually matter for the score.
CPA fixes this by taking each training example and generating a second response to the same query that should receive the opposite label. An independent auditor confirms the flipped label, and the pair is used as a pairwise training example that forces the model to focus on the feature that changes the verdict. The judge is fine-tuned on both the original pointwise data and these counterfactual pairs.
On six binary criteria, CPA lifted average pointwise accuracy by 2.5 points over pointwise-only training and surpassed a frontier zero-shot judge by 11.8 points. This makes small judges significantly more reliable for real-time agent decisions where cost and latency matter.