Artificial intelligence-generated images often misrepresent marginalized communities. Current evaluation tools check image quality or prompt accuracy, not whether communities feel represented. This can reinforce harmful stereotypes.
The authors created Community Scorers. They worked with several disability communities to define what good representation looks like. They collected ratings of AI-generated images from community members and trained prediction systems based on those ratings.
The Community Scorers matched community preferences significantly better than standard evaluation metrics. This gives communities a way to hold AI systems accountable. It also offers developers a tool to ensure their images are respectful.