When a computer learns to classify images or text, it uses a loss function to measure how wrong its predictions are. The most common loss function, called cross entropy, is simple and fast. However, a newer loss function called the familywise loss has better theoretical guarantees for learning, but it was thought to be too slow because it required sorting the predictions.
This paper proves that the familywise loss and its gradient, the direction the model adjusts to improve, can be computed just as quickly as cross entropy. The key insight is that the calculation only needs the top few predictions, not an entire sorted list. The authors built custom software kernels that run this computation efficiently on both CPUs and GPUs.
In their tests, the optimized familywise loss ran faster than cross entropy on both CPU and GPU hardware. This means the familywise loss can now replace cross entropy in large-scale classification tasks, potentially leading to better learning without any speed penalty.
