Skip to content

Fault tolerant collectives - #33

Draft
tstamler wants to merge 2 commits into
NVIDIA:mainfrom
tstamler:fault-tolerant-collectives
Draft

tstamler wants to merge 2 commits into
NVIDIA:mainfrom
tstamler:fault-tolerant-collectives

Conversation

@tstamler

@tstamler tstamler commented Nov 2, 2026

Copy link
Copy Markdown

Solution to issue mentioned in #32

Full, performant implementations of fault-tolerant collectives built on top of NCCL communicators.

Fault tolerance does not cost performance in the regime that matters most for inference. On GB200 NVL at 4–32 GPUs, the fault-tolerant collectives beat NCCL for small and medium messages: AllReduce by 1.5–2× and AllGather by 2.5–3.8× below 256 KB, and AllReduce by 1.1–1.35× at 1–8 MB. We attribute this to direct communication. Each rank writes its data straight into every peer's symmetric memory over NVLink in a single step, avoiding the multi-step pipelined schedules NCCL uses, so latency stays low when messages are small. The same direct, all-to-all pattern is what limits us at large messages, where NCCL leads: roughly above 4–33 MB for AllGather and 64–128 MB for AllReduce. Every rank must push a separate copy to each peer through its own NVLink egress, so throughput is capped by per-rank egress bandwidth. Above that point NCCL switches to bandwidth-optimal algorithms (ring, and log-depth PAT trees for AllGather) that spread traffic across links and both link directions. A/B tests support this: forcing NCCL to use NVLS, the hardware-multicast form of the same direct pattern, brings it within about 14% of our AllGather, so the remaining gap is the choice of algorithm rather than the hardware. All-to-all is currently slower than NCCL, and the PR documents this as a known limitation.

The contribution includes a correctness suite covering every collective in eager and CUDA-graph modes, fault-injection, convergence and rejoin tests. Under symmetric timeouts the model confirms that all ranks agree and that rejoin resynchronizes generations; under asymmetric timeouts the safety properties still hold. We have also demonstrated functionality with vLLM and plan to upstream fault-tolerance support there once these collectives are published.

tstamler and others added 2 commits November 2, 2026 16:03
Import the fault-tolerant collectives library from NCCL's
contrib/fault_tolerant_collectives (tstamler/nccl ft-nccl-upstream at
fe221ad16) unchanged, so that the follow-up build integration is reviewable
on its own.

The library provides timeout + active-mask AllGather, AllReduce,
AllToAll, AllToAllv, AllGatherv and ReduceScatterv kernels on the NCCL
Device API (LSA, plus experimental GIN for AllGather/AllReduce), the
ft_converge/ft_rejoin membership operations, and an ft_nccl
torch.distributed backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: tstamler <tstamler@nvidia.com>
Point the ft_handle and ft_handle_bench Makefiles at the repository's
makefiles/common.mk and default NCCL_HOME to the vendored
third_party/nccl build, as nccl_ep does; pin C++17 to match. Add a
build target so the root Makefile can drive nccl_ft like the other
extensions (make nccl_ft.build), list it in the root README, and update
the nccl_ft README's build, install and test paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: tstamler <tstamler@nvidia.com>
@kwen2501
kwen2501 self-requested a review November 3, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant