Repository navigation
Conversation
Import the fault-tolerant collectives library from NCCL's contrib/fault_tolerant_collectives (tstamler/nccl ft-nccl-upstream at fe221ad16) unchanged, so that the follow-up build integration is reviewable on its own. The library provides timeout + active-mask AllGather, AllReduce, AllToAll, AllToAllv, AllGatherv and ReduceScatterv kernels on the NCCL Device API (LSA, plus experimental GIN for AllGather/AllReduce), the ft_converge/ft_rejoin membership operations, and an ft_nccl torch.distributed backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: tstamler <tstamler@nvidia.com>
Point the ft_handle and ft_handle_bench Makefiles at the repository's makefiles/common.mk and default NCCL_HOME to the vendored third_party/nccl build, as nccl_ep does; pin C++17 to match. Add a build target so the root Makefile can drive nccl_ft like the other extensions (make nccl_ft.build), list it in the root README, and update the nccl_ft README's build, install and test paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: tstamler <tstamler@nvidia.com>
kwen2501
self-requested a review
November 3, 2026 20:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Solution to issue mentioned in #32
Full, performant implementations of fault-tolerant collectives built on top of NCCL communicators.
Fault tolerance does not cost performance in the regime that matters most for inference. On GB200 NVL at 4–32 GPUs, the fault-tolerant collectives beat NCCL for small and medium messages: AllReduce by 1.5–2× and AllGather by 2.5–3.8× below 256 KB, and AllReduce by 1.1–1.35× at 1–8 MB. We attribute this to direct communication. Each rank writes its data straight into every peer's symmetric memory over NVLink in a single step, avoiding the multi-step pipelined schedules NCCL uses, so latency stays low when messages are small. The same direct, all-to-all pattern is what limits us at large messages, where NCCL leads: roughly above 4–33 MB for AllGather and 64–128 MB for AllReduce. Every rank must push a separate copy to each peer through its own NVLink egress, so throughput is capped by per-rank egress bandwidth. Above that point NCCL switches to bandwidth-optimal algorithms (ring, and log-depth PAT trees for AllGather) that spread traffic across links and both link directions. A/B tests support this: forcing NCCL to use NVLS, the hardware-multicast form of the same direct pattern, brings it within about 14% of our AllGather, so the remaining gap is the choice of algorithm rather than the hardware. All-to-all is currently slower than NCCL, and the PR documents this as a known limitation.
The contribution includes a correctness suite covering every collective in eager and CUDA-graph modes, fault-injection, convergence and rejoin tests. Under symmetric timeouts the model confirms that all ranks agree and that rejoin resynchronizes generations; under asymmetric timeouts the safety properties still hold. We have also demonstrated functionality with vLLM and plan to upstream fault-tolerance support there once these collectives are published.