The authors identified two bottlenecks in automated annotation: existing benchmarks use single labels per instance, failing to capture valid inter-annotator disagreement, and current annotation systems (single-agent or static multi-agent) struggle with complex, detailed guidelines.
Multi-agent framework with contrastive knowledge improves LLM annotation consistency
The authors introduce a benchmark with 5 human annotations per item and show their framework matches human agreement within 2 points of Fleiss' kappa.
Chinese Tech
Xiaopeng Ke · Zhixin Wang · Xuebo Liu · Xinyu Ma · Xing Zheng · Bowen Song · +2 more
ANT Group · Harbin Institute of Technology (Shenzhen)
Research Digest··3 min read
The authors construct ANT-A, a multi-annotator benchmark spanning 7 datasets across domains and modalities, each with detailed guidelines and 5 independent human labels per instance.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.