[BUG]DeepSpeed latest version fails during NCCL process-group initialization with CUDA error 600, while 0.17.2 works

[janelu9]

Describe the bug

With the latest DeepSpeed version, launching a single-GPU training job fails during torch.distributed.new_group() with an NCCL CUDA error 600: device not ready.

The failure occurs during DeepSpeed-launched process initialization, before the model forward pass or any training computation.

Importantly, the same environment can successfully initialize both the default NCCL process group and an additional NCCL process group without DeepSpeed. Downgrading DeepSpeed to 0.17.2 makes the exact same training command work correctly.

This suggests a regression or compatibility issue in the latest DeepSpeed launcher/runtime related to NCCL process-group initialization.

To Reproduce

1. Environment

The issue was reproduced with:

PyTorch: 2.13.0+cu129
CUDA: 12.9
NCCL: 2.29.7
GPU: NVIDIA GeForce RTX 3060 Laptop GPU
GPU capability: 8.6
Python: 3.12

2. Launch a training job with the DeepSpeed launcher

deepspeed --module jllm.train_pipe \
    --model Qwen3.5-27B \
    --num_train_epochs 1 \
    --gradient_accumulation_steps 1 \
    --train_data s10_Qwen3.5-27B/ \
    --init \
    --partition_method '0,6'

DeepSpeed launches:

world_size=1
local_rank=0
CUDA_VISIBLE_DEVICES=0

3. The job fails during model-parallel initialization

The relevant traceback is:

torch.distributed.new_group()
    -> _new_process_group_helper()
    -> eager_backend.eager_connect_single_device()
    -> NCCL error

Error:

torch.distributed.DistBackendError: NCCL error in:
.../NCCLUtils.cpp:243, unhandled cuda error

ncclUnhandledCudaError: Call to CUDA function failed.

Last error:
Cuda failure 600 'device not ready'

The failure occurs at:

jllm/core/parallel_state.py
    create_group()
        -> torch.distributed.new_group()

before the model starts training.

Expected behavior

The DeepSpeed launcher should successfully initialize the NCCL process group, as it does with DeepSpeed 0.17.2.

Since the job uses only one GPU (world_size=1), NCCL process-group initialization should not fail with CUDA error 600.

ds_report output

ds_report output:

[Please paste the output of `ds_report` here]

Screenshots

Not applicable. The complete traceback is provided above.

System info

• OS: Ubuntu on WSL2
• GPU count and types: 1 × NVIDIA GeForce RTX 3060 Laptop GPU
• GPU capability: 8.6
• Interconnects: N/A (single GPU)
• Python: 3.12
• PyTorch: 2.13.0+cu129
• CUDA runtime: 12.9
• NCCL: 2.29.7
• DeepSpeed: latest version where the issue occurs

Launcher context

The experiment is launched using the DeepSpeed launcher:

deepspeed --module jllm.train_pipe ...

DeepSpeed reports:

WORLD INFO DICT: {'localhost': [0]}
nnodes=1
num_local_procs=1
node_rank=0
dist_world_size=1
CUDA_VISIBLE_DEVICES=0

Docker context

Not using Docker.

Additional context

I verified that the underlying PyTorch/NCCL environment works correctly without DeepSpeed.

The following minimal test succeeds in the same Python environment:

import torch
import torch.distributed as dist

print("torch:", torch.__version__)
print("cuda:", torch.version.cuda)
print("nccl:", torch.cuda.nccl.version())
print("gpu:", torch.cuda.get_device_name(0))
print("capability:", torch.cuda.get_device_capability(0))

torch.cuda.set_device(0)

dist.init_process_group(
    backend="nccl",
    init_method="tcp://127.0.0.1:29501",
    rank=0,
    world_size=1,
)

print("default PG OK")

g = dist.new_group(
    ranks=[0],
    backend="nccl",
)

print("new_group OK")

dist.destroy_process_group()

Output:

torch: 2.13.0+cu129
cuda: 12.9
nccl: (2, 29, 7)
gpu: NVIDIA GeForce RTX 3060 Laptop GPU
capability: (8, 6)
default PG OK
new_group OK

Therefore, the CUDA/NCCL stack itself appears to be functional.

DeepSpeed version comparison

The exact same training environment and command work correctly after downgrading DeepSpeed to:

DeepSpeed 0.17.2

So the issue appears to be specific to newer DeepSpeed versions.

In summary:

DeepSpeed latest
    -> torch.distributed.new_group()
    -> NCCL eager_connect_single_device()
    -> CUDA error 600: device not ready
    -> FAIL

DeepSpeed 0.17.2
    -> same environment
    -> same training command
    -> SUCCESS

Could this be related to a change in the DeepSpeed launcher or distributed/NCCL initialization path in versions newer than 0.17.2?

[pengdurice]

@janelu9 , thank you for reporting this issue! A couple of asks to help us debug more;-)

1. is there any chance that you can do the tests on two GPUs? This helps isolate if this is a single GPU issue or also applies to two GPUs.
2. For the failure case, would you mind paste the full raw call trace with with every File "...", line N, in ... frame? This help us understand what is really breaking.
3. Would you mind apply PR and test again to see if that fixes your issue?
4. The code snippet in Additional context may not be very informative since PyTorch does lazy initialization without device_id, you can try

dist.init_process_group(
    backend="nccl",
    init_method="tcp://127.0.0.1:29501",
    rank=0,
    world_size=1,
    device_id=torch.device("cuda", 0),      # <-- the only change
)

and see what happens.
can you also try NCCL_DEBUG=INFO when doing the tests and paste the full log?
Thank you!

[janelu9]

@janelu9 , thank you for reporting this issue! A couple of asks to help us debug more;-)

1. is there any chance that you can do the tests on two GPUs? This helps isolate if this is a single GPU issue or also applies to two GPUs.
2. For the failure case, would you mind paste the full raw call trace with with every File "...", line N, in ... frame? This help us understand what is really breaking.
3. Would you mind apply PR and test again to see if that fixes your issue?
4. The code snippet in Additional context may not be very informative since PyTorch does lazy initialization without device_id, you can try

dist.init_process_group(
    backend="nccl",
    init_method="tcp://127.0.0.1:29501",
    rank=0,
    world_size=1,
    device_id=torch.device("cuda", 0),      # <-- the only change
)

and see what happens.
can you also try NCCL_DEBUG=INFO when doing the tests and paste the full log?
Thank you!

Thanks, I will try again. It's well when running on two GPUs.

[pengdurice]

@janelu9 , thank you for reporting this issue! A couple of asks to help us debug more;-)

1. is there any chance that you can do the tests on two GPUs? This helps isolate if this is a single GPU issue or also applies to two GPUs.
2. For the failure case, would you mind paste the full raw call trace with with every File "...", line N, in ... frame? This help us understand what is really breaking.
3. Would you mind apply PR and test again to see if that fixes your issue?
4. The code snippet in Additional context may not be very informative since PyTorch does lazy initialization without device_id, you can try

dist.init_process_group(
    backend="nccl",
    init_method="tcp://127.0.0.1:29501",
    rank=0,
    world_size=1,
    device_id=torch.device("cuda", 0),      # <-- the only change
)

and see what happens.
can you also try NCCL_DEBUG=INFO when doing the tests and paste the full log?
Thank you!

Thanks, I will try again. It's well when running on two GPUs.

Thank you! LMK when you get new results. It is interesting to learn that it works on 2 GPUs and do not work on one.

[pengdurice]

I did some investigations on my side:
settings:
h200 gpu, torch 2.8.0+cu128, NCCL 2.27.3, Linux (I don't have the wsl2 as the @janelu9 has):
I tried 4 different code versions: 0.17.2, 0.17.3; master and the fix version with PR 8269.
Also, I tried a simple toy example training as well as @janelu9 's jllm codebase.
The tldr is that, I can generate some intermediate results that can be indicative but not the final failure (maybe because I don't have the same machine or lib versions).
The mechanism is that: when a device_id is given (introduced in 0.17.3), the process group init happens eagerly and when a new group is requested, it will do the commSplit of the existing process groups. But when a device id is not given, the init is lazy and there is no com split, and it goes a different path. I can reproduce the number of comm split:

tree	bound_device_id	ncclCommSplit
0.17.2	None	0
0.17.3	cuda:0	1
master	cuda:0	1
PR 8269	None	0

So the remaining question becomes if the comm split causes issues specifically on customer's hardware and software.

This PR may be a walk-around at best to avoid the trigger of the issue rather than solving it if comm split is the root cause.
As of world size = 1 vs world size =2: I am still investigating on that.

Look forward to the full raw stack of errors from the customer and if the fix will resolve the issue and what happens with world size = 2.

[pengdurice]

@janelu9 , would you also mind paste the logs of world-size = 2? that helps understand the diffs of ws=1 vs 2. Thank you!

[janelu9]

@janelu9 , would you also mind paste the logs of world-size = 2? that helps understand the diffs of ws=1 vs 2. Thank you!

My code is similar to Megatron-DeepSpeed, and the problem occurs when building the 3D parallel topology. The error only occurs when world_size=1; there are no errors when world_size>1.
The error occurs here.
https://github.com/janelu9/EasyLLM/blob/11589b1b177c06b8dcbdc6ada59043de7f6cce8b/jllm/core/parallel_state.py#L183

[rank0]: Traceback (most recent call last):
[rank0]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank0]:   File "<frozen runpy>", line 88, in _run_code
[rank0]:   File "/mnt/e/nlp/jllm/train_pipe.py", line 761, in <module>
[rank0]:     main(args)
[rank0]:   File "/mnt/e/nlp/jllm/train_pipe.py", line 658, in main
[rank0]:     parallel_state.initialize_model_parallel(args.tensor_parallel_size,
[rank0]:   File "/mnt/e/nlp/jllm/core/parallel_state.py", line 1151, in initialize_model_parallel
[rank0]:     group = create_group(
[rank0]:             ^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/nlp/jllm/core/parallel_state.py", line 183, in create_group
[rank0]:     return torch.distributed.new_group(**kwargs,device_id=torch.device("cuda", 0))
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/py312/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
[rank0]:     func_return = func(*args, **kwargs)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/py312/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 5863, in new_group
[rank0]:     return _new_group_with_tag(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/py312/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 6056, in _new_group_with_tag
[rank0]:     pg, pg_store = _new_process_group_helper(
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/py312/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2335, in _new_process_group_helper
[rank0]:     eager_backend.eager_connect_single_device(device_id)
[rank0]: torch.distributed.DistBackendError: NCCL error in: /__w/pytorch/pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:243, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.29.7
[rank0]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank0]: Last error:
[rank0]: Cuda failure 600 'device not ready'
[rank0]:[W827 10:36:47.199915054 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[2026-08-27 10:37:01,454] [INFO] [launch.py:335:sigkill_handler] Killing subprocess 978