PyTorch 2.15 tackles the 'one bad GPU' problem with rewritten NCCL backend
New flight recorder and straggler handling aim to make training jobs more resilient
The latest release of PyTorch, version 2.15, introduces a rewritten backend for Nvidia's Collective Communications Library (NCCL) designed to improve fault diagnosis and recovery when GPUs work together; one bad GPU may no longer mean restarting an entire training job.