Automated machine learning (AutoML) systems try to find the best way to build a predictive model with little human help. Some modern systems use an LLM () critic to give suggestions after each try, hoping to speed up the search. But this paper shows that a weak critic can actually make things worse.
The researchers tested what happens when you add a critic to an AutoML search loop. They compared a naive 8-billion-parameter critic against a search with no critic at all. On 7 out of 8 datasets, the critic-guided search found worse models than random guessing.
The main finding is that a critic can misdirect the search, wasting the limited number of tries. The worst loss was 0.245 AUC (a measure of model quality, where higher is better). The practical lesson is to always test a critic against a no-critic baseline before trusting it.
