Language models can reject harmful requests without sacrificing general reasoning

The authors report safer reasoning and answers, alongside fewer refusals of benign requests, by training models on separate safety objectives while protecting reasoning performance.

Chinese Tech

Yichen Sun · Qirui Liu · Yan Wang · Jianan Lin · Linbo Jiang · Shiyu Wang · +2 more

Zhejiang University · Chongqing Ant Consumer Finance Co., Ltd. · Alibaba Group

Research Digest·6 November 2026·2 min read

Illustration: Language models can reject harmful requests without sacrificing general reasoning

In one line

Decomposing safety into four verifiable reward signals and preserving reasoning gradients gives safer large reasoning models with fewer over-refusals and no loss of reasoning.

Before you read this

Three ideas this paper builds on

safety alignmentProcess of training models to recognize and refuse harmful requests, ensuring outputs remain safe.98 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.152 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.279 papers →

Read it as

Large reasoning models solve hard problems by writing out a long chain of thought before answering. The visible reasoning can contain harmful details even when the final answer is a refusal. Current safety training checks only the final answer, so it misses reasoning dangers and causes over-refusals.

The method trains on four separate checks: safe reasoning, non-malicious response, reasoning-answer consistency, and helpfulness. A grader scores each check. A gradient projection step updates the model only when safety changes do not hurt reasoning abilities.

Across models from 1.5 billion to 671 billion parameters, it raised safety and lowered over-refusals while keeping reasoning performance. Fine-grained checks are harder to game than one overall score and generalize better to new attacks.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

safety alignment reinforcement learning with verifiable rewards large language model chain of thought llm as a judge supervised fine-tuning