Large language models frequently express high confidence in wrong answers, undermining their reliability. The paper proposes CoCal (Companion ), a method that learns confidence estimation concurrently with capability improvement during reinforcement learning from verifiable rewards (RLVR). Unlike existing approaches that combine confidence and task learning into shared policy parameters, CoCal separates the two by training a small companion model on rollout hidden states and verifier-derived correctness signals, leaving the main policy unchanged.
Experiments on Qwen3-8B and Qwen3-14B show that CoCal improves confidence calibration compared to both RL-based concurrent methods (RLCR and DCPO) and matched post-hoc calibration, while maintaining task performance. In five mathematical datasets, CoCal outperformed RLCR by 9.9 percentage points and DCPO by 5.0 points in accuracy. The companion also generalizes across different domains and policy shifts, and the benefits hold at both model scales.
This is interesting because it offers a practical way to get reliable confidence estimates during training without additional sampling, and without interference between the two learning objectives. The approach leverages the natural feedback from RLVR to improve self-assessment, which is critical for autonomous LLM applications.