The latest release of PyTorch, version 2.15, introduces a rewritten backend for Nvidia's Collective Communications Library (NCCL) designed to improve fault diagnosis and recovery when GPUs work together; one bad GPU may no longer mean restarting an entire training job.
The updated NCCL backend, known as ProcessGroupNCCL, enforces a strict rule: one process group gets only one communicator during initialization. By default, this rule eliminates crosstalk between multiple communicators. The previous version could spawn dozens of communicators per process group, leading to intermittent deadlocks and unpredictable performance.
PyTorch core developer Tristan Rice, a Meta software engineer specializing in distributed computing, said during a livestream on Thursday that the old interface was drowning in technical debt and needed a rewrite. The new backend also handles stragglers more effectively by allowing a third-party orchestrator to exclude slow or crashed GPUs and assign replacements.
A new “flight recorder” captures all operations in snapshots that can be inspected after a failure. Previously, users would get a generic message if a cluster of GPUs stalled, perhaps from a timeout or deadlock. Rice noted the cleaned-up backend gives developers a blank slate for adding future features.
Other improvements in PyTorch 2.15 include a new Python transport API for asynchronous one-way data movement, parallel processing enhancements for Apple hardware, and support for Nvidia’s next-generation Blackwell GPUs. The legacy ProcessGroupNCCL remains as a rollback option, but if the new backend works as expected, the older version is planned for removal in version 2.16, planned for early next year.
“Every major ML lab (OpenAI, DeepMind, Meta, Nvidia) uses PyTorch. It powers everything from diffusion models and LLMs to recommender systems and robotics. This widespread adoption isn’t accidental,” said Will Stewart, co-founder of AI software provider Northflank, in a blog post. The update comprised 3,117 commits from 583 contributors.
Analysis
Why This Matters
- Reduces wasted compute and time in large-scale AI training by allowing recovery from individual GPU failures.
- Improves observability for ML ops teams with the new flight recorder for post-mortem debugging.
- Signals PyTorch’s maturation from a research tool to a production-grade infrastructure platform.
Background
PyTorch is an open-source machine learning framework initially developed by Meta in 2016. NCCL (Nvidia Collective Communications Library) provides efficient inter-GPU communication primitives, essential for training large neural networks across multiple nodes. The 2.15 release continues a pattern of hardening the framework for enterprise deployment.
What to Watch
- Adoption of the new NCCL backend by major AI labs and cloud providers.
- Removal of the legacy ProcessGroupNCCL in PyTorch 2.16, planned for early next year.
- Further refinements to the flight recorder and straggler-handling capabilities as real-world feedback arrives.
Checked
12 claims in this story checked against 1 source: 12 supported.
Show each claim
- SupportedPyTorch 2.15’s rewritten NCCL backend may mean one bad GPU no longer requires restarting an entire training job.www.theregister.com - Articles: “PyTorch 2.15 tackles the 'one bad GPU' problem. ML admins will be happy to learn that one bad GPU may no longer mean starting an entire training job from scratch.”
- SupportedA new flight recorder and better straggler handling aim to make training jobs more resilient.www.theregister.com - Articles: “The updated library also better handles stragglers by allowing for easier reconfiguration. ... A “flight recorder” has also been added to give admins a better idea of what went wrong.”
- SupportedPyTorch 2.15 includes a rewritten NCCL backend designed to improve fault diagnosis and recovery when GPUs work together.www.theregister.com - Articles: “The latest 2.15 release of PyTorch includes a rewritten backend for Nvidia's Collective Communications Library (NCCL), designed to improve fault diagnosis and recovery when GPUs work together.”
- SupportedRice said the old interface was drowning in technical debt and needed a rewrite.www.theregister.com - Articles: “The interface to the networking library in PyTorch needed a rewrite – it was drowning in technical debt, said PyTorch core developer Tristan Rice, a Meta software engineer specializing in distributed computing, during a live YouTube livestream on Thursday.”
- SupportedThe new backend’s default rule gives each process group one communicator during initialization and eliminates crosstalk between multiple communicators.www.theregister.com - Articles: “The new version of the NCCL backend comes with a strict rule: One process group gets only one communicator during initialization. By default, this rule eliminates crosstalk between multiple communicators”
- SupportedThe previous backend could spawn dozens of communicators per process group, leading to intermittent deadlocks and unpredictable performance.www.theregister.com - Articles: “The previous PyTorch backend for NCCL, called ProcessGroupNCCL, could spawn dozens of communicators per process group, leading to intermittent deadlocks and unpredictable performance.”
- SupportedA third-party orchestrator can exclude a crashed or slow GPU and assign a replacement worker.www.theregister.com - Articles: “If a GPU crashes or runs slowly, a third-party orchestrator can exclude the laggard and even assign a replacement worker.”
- SupportedThe flight recorder captures operations in snapshots that can be inspected after a failure.www.theregister.com - Articles: “The new flight recorder captures all operations in a series of snapshots that can be inspected after a failure occurs.”
- SupportedPreviously, a stalled GPU cluster could leave users with only a generic message, perhaps because of a timeout or deadlock.www.theregister.com - Articles: “Previously, if a cluster of GPUs stalled—perhaps from a timeout or deadlock—the user would just get a generic message.”
- SupportedThe story quotes Will Stewart saying that every major ML lab listed uses PyTorch and that its adoption is not accidental.www.theregister.com - Articles: ““Every major ML lab (OpenAI, DeepMind, Meta, Nvidia) uses PyTorch. It powers everything from diffusion models and LLMs to recommender systems and robotics. This widespread adoption isn’t accidental,” noted Will Stewart, co-founder of AI software provider Northflank, in a blog post.”
- SupportedThe legacy ProcessGroupNCCL remains in the 2.15 package as a rollback option, and its removal in 2.16 is conditional on the new backend working as expected; 2.16 is planned for early next year.www.theregister.com - Articles: “The legacy ProcessGroupNCCL is part of the 2.15 package as a rollback, but if the new NCCL backend works as expected, the older version will be removed in the 2.16 release, planned for early next year.”
- SupportedThe update comprised 3,117 commits from 583 contributors.www.theregister.com - Articles: “Overall, the update consisted of 3,117 commits from 583 contributors.”
