Companion model sharpens LLM confidence without harming task performance

The authors introduce CoCal, which trains a lightweight companion from rollout hidden states and verifier-derived correctness supervision while leaving task optimization unchanged.

Research Lab
Shiyu Ni · Keping Bi · Jiafeng Guo · Yilong Xu · Jingtong Wu · Zengxin Han · +1 more

State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Research Digest··3 min read

In one line

CoCal trains a lightweight companion on rollout hidden states to calibrate LLM confidence without sacrificing task performance.

Before you read this

Two ideas this paper builds on
Read it as

Large language models frequently express high confidence in wrong answers, undermining their reliability. The paper proposes CoCal (Companion ), a method that learns confidence estimation concurrently with capability improvement during reinforcement learning from verifiable rewards (RLVR). Unlike existing approaches that combine confidence and task learning into shared policy parameters, CoCal separates the two by training a small companion model on rollout hidden states and verifier-derived correctness signals, leaving the main policy unchanged.

Experiments on Qwen3-8B and Qwen3-14B show that CoCal improves confidence calibration compared to both RL-based concurrent methods (RLCR and DCPO) and matched post-hoc calibration, while maintaining task performance. In five mathematical datasets, CoCal outperformed RLCR by 9.9 percentage points and DCPO by 5.0 points in accuracy. The companion also generalizes across different domains and policy shifts, and the benefits hold at both model scales.

This is interesting because it offers a practical way to get reliable confidence estimates during training without additional sampling, and without interference between the two learning objectives. The approach leverages the natural feedback from RLVR to improve self-assessment, which is critical for autonomous LLM applications.

How it works

before and after
EXISTING CONCURRENT METHODSRLVR policy update: learns capabilityRLVR policy updatelearns capabilityShared confidence head: inside policy parametersShared confidenceheadinside policyparametersCOCAL (OURS)hidden statescorrectnesslabelsconfidencescorePolicy rollouts: hidden states + answersPolicyrolloutshidden states +answersVerifier labels: correctness supervisionVerifierlabelscorrectnesssupervisionCompanion network: lightweight, separateCompanionnetworklightweight,separateCalibrated confidence: for abstention or helpCalibratedconfidencefor abstentionor helpThe companion trains on accumulated rollout data after each RLVR iteration, adapting to policy shifts…
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe