Repository navigation
Conversation
The async self-copy tests added in #29792 constructed CudaStream with own_flag=false and a null external cuBLAS handle, so the constructor's cublasSetStream threw CUBLAS_STATUS_NOT_INITIALIZED. Construct the streams with own_flag=true so CudaStream creates its own cuBLAS/cuDNN handles, and drop the explicit cudaStreamDestroy calls since the owning CudaStream now destroys the stream. Fixes #33930 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Removing explicit stream cleanup leaks CUDA streams in both async tests under USE_CUDA_MINIMAL because CudaStream’s destructor omits cudaStreamDestroy in that configuration.
1 open issue
What changed in this PR
This test-only PR addresses #33930 by fixing stream ownership in the two asynchronous same-buffer copy regression tests added in #29792. No product code changes are included.
Changes:
- Set
own_flag=trueso eachCudaStreamcreates valid library handles. - Remove explicit stream destruction in favor of destructor cleanup.
- Preserve synchronization and capture-test stream lifetimes.
| File | Description |
|---|---|
onnxruntime/test/providers/cuda/test_cases/gpu_data_transfer_test.cc |
Updates stream ownership and cleanup in both async same-buffer copy tests. |
Review effort: Balanced
~CudaStream compiles out cudaStreamDestroy under USE_CUDA_MINIMAL, so the async same-buffer tests would leak their streams in that configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
GPU execution was not validated locally, so both async tests, including the capture-validity assertion, still need confirmation in Linux CUDA, TensorRT, and Windows CUDA CI.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
|
[Agentic Response]: Closing as superseded. #33916 already landed the same test fix on main ( |

Description
Fixes the two async same-buffer tests added in #29792 (
onnxruntime/test/providers/cuda/test_cases/gpu_data_transfer_test.cc):GPUDataTransferSameBufferTest.CopyTensorAsyncSelfCopyIsNoOpGPUDataTransferSameBufferTest.CopyTensorAsyncSelfCopyDuringCaptureKeepsCaptureValidBoth constructed
CudaStreamwithown_flag=falseandnullptrexternal cuBLAS/cuDNN handles. In that modeCudaStream::CudaStream(core/providers/cuda/cuda_stream_handle.cc) callscublasSetStream(cublas_handle_, stream)on the null handle, which throws:Test-only change:
own_flag=true(matchingcuda_execution_provider_test.cc) soCudaStreamcreates its own cuBLAS (and cuDNN, if enabled) handle.cudaStreamDestroy(cuda_stream)calls, because an owningCudaStreamdestroys the stream in its destructor (avoids a double destroy).cudaStreamEndCaptureand graph destruction. The first test still synchronizes the stream before scope exit.No product code changes.
GPUDataTransfer::CopyTensorAsyncdoes not callCudaStream::Flush(), so owning the stream doesn't add a capture-illegal synchronize.Motivation and Context
Fixes #33930. These tests fail on main in Test Linux CUDA x64 Release, Test Linux TensorRT x64 Release, and Windows GPU CUDA CI Pipeline Test Job.
Validation
clang-format --dry-run --Werror onnxruntime/test/providers/cuda/test_cases/gpu_data_transfer_test.cc(VS clang-format 22.1.3): passed.