Large language models are now used to evaluate other AI outputs, but they only see what they are given. When judging a multi-step task like a web search, important details may be missing from the provided log.
Agent-as-a-Judge systems turn the evaluator into an active agent that can search for missing information, run code, or check facts before making a decision. This makes the evaluation more thorough and grounded in actual evidence.
The main finding is that these active judges often improve accuracy, but not consistently across all tasks. The approach is most valuable when the missing evidence is crucial, such as verifying code output or live web content.
