Large language models (LLMs) are often used as judges to replace human annotators, but deciding when replacement is justified is difficult. Previous tests required the same humans to label every item, which is rare in practice.
The authors propose AltRA-Test, a statistical method that resamples items and annotations to compare agreement between all-human and mixed human-LLM panels. It works even when different items are labeled by different people.
In tests across 20 datasets, about 100 items were often enough to decide. LLMs passed the test in about half of the setups, depending on the task. This means LLM annotators must be validated for each specific task.
