Object hallucination—where describe objects not present in an image—remains a significant problem. Existing on-policy methods often reduce hallucinations by making fewer claims (saying less), which sacrifices informativeness. This paper proposes a fine-grained alignment framework to produce both faithful and informative descriptions. The framework includes two key components: the DOPA dataset, which provides dense annotations of object presence and absence across an expanded vocabulary, and the SCAPO algorithm, which assigns credit at the subsentence level rather than the entire response. This ensures that valid claims are rewarded and hallucinations are penalized independently, preventing local errors from suppressing correct outputs. The method improves the faithfulness and informativeness of generated captions, and also boosts performance on downstream discriminative tasks.
Fine-Grained Rewards Reduce Vision-Language Hallucinations Without Silencing Valid Detail
The authors combine dense object annotations with subsentence-level reinforcement learning to reward accurate detail while penalizing localized hallucinations.
State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhongguancun Academy
In one line
Fine-grained alignment using dense reward signals and subsentence-level credit assignment reduces object hallucination in vision-language models.
Before you read this
Three ideas this paper builds onThe authors construct Dense Object Presence and Absence (DOPA), a dataset that exhaustively labels the deterministic presence or absence of concepts from an expanded object vocabulary.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.