Skip to content

Fix CUDA same-buffer CopyTensorAsync tests to own their CudaStream - #33934

Closed
adrastogi wants to merge 2 commits into
mainfrom
adrastogi/fix-cuda-same-buffer-test
Closed

adrastogi wants to merge 2 commits into
mainfrom
adrastogi/fix-cuda-same-buffer-test

Conversation

@adrastogi

Copy link
Copy Markdown
Contributor

Description

Fixes the two async same-buffer tests added in #29792 (onnxruntime/test/providers/cuda/test_cases/gpu_data_transfer_test.cc):

  • GPUDataTransferSameBufferTest.CopyTensorAsyncSelfCopyIsNoOp
  • GPUDataTransferSameBufferTest.CopyTensorAsyncSelfCopyDuringCaptureKeepsCaptureValid

Both constructed CudaStream with own_flag=false and nullptr external cuBLAS/cuDNN handles. In that mode CudaStream::CudaStream (core/providers/cuda/cuda_stream_handle.cc) calls cublasSetStream(cublas_handle_, stream) on the null handle, which throws:

CUBLAS failure 1: the library was not initialized ... expr=cublasSetStream(cublas_handle_, stream)

Test-only change:

  • Construct both streams with own_flag=true (matching cuda_execution_provider_test.cc) so CudaStream creates its own cuBLAS (and cuDNN, if enabled) handle.
  • Remove the explicit cudaStreamDestroy(cuda_stream) calls, because an owning CudaStream destroys the stream in its destructor (avoids a double destroy).
  • The capture test still keeps the stream alive through cudaStreamEndCapture and graph destruction. The first test still synchronizes the stream before scope exit.

No product code changes. GPUDataTransfer::CopyTensorAsync does not call CudaStream::Flush(), so owning the stream doesn't add a capture-illegal synchronize.

Motivation and Context

Fixes #33930. These tests fail on main in Test Linux CUDA x64 Release, Test Linux TensorRT x64 Release, and Windows GPU CUDA CI Pipeline Test Job.

Validation

  • clang-format --dry-run --Werror onnxruntime/test/providers/cuda/test_cases/gpu_data_transfer_test.cc (VS clang-format 22.1.3): passed.
  • I didn't run the CUDA tests locally because this machine has no NVIDIA GPU. Validation depends on the CUDA CI jobs above: Test Linux CUDA x64 Release, Test Linux TensorRT x64 Release, and Windows GPU CUDA CI Pipeline Test Job.

The async self-copy tests added in #29792 constructed CudaStream with own_flag=false and a null external cuBLAS handle, so the constructor's cublasSetStream threw CUBLAS_STATUS_NOT_INITIALIZED. Construct the streams with own_flag=true so CudaStream creates its own cuBLAS/cuDNN handles, and drop the explicit cudaStreamDestroy calls since the owning CudaStream now destroys the stream.

Fixes #33930

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 06:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Removing explicit stream cleanup leaks CUDA streams in both async tests under USE_CUDA_MINIMAL because CudaStream’s destructor omits cudaStreamDestroy in that configuration.

1 open issue
What changed in this PR

This test-only PR addresses #33930 by fixing stream ownership in the two asynchronous same-buffer copy regression tests added in #29792. No product code changes are included.

Changes:

  • Set own_flag=true so each CudaStream creates valid library handles.
  • Remove explicit stream destruction in favor of destructor cleanup.
  • Preserve synchronization and capture-test stream lifetimes.
File Description
onnxruntime/​test/​providers/​cuda/​test_cases/​gpu_data_transfer_test.cc Updates stream ownership and cleanup in both async same-buffer copy tests.

Review effort: Balanced

~CudaStream compiles out cudaStreamDestroy under USE_CUDA_MINIMAL, so the async same-buffer tests would leak their streams in that configuration.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

GPU execution was not validated locally, so both async tests, including the capture-validity assertion, still need confirmation in Linux CUDA, TensorRT, and Windows CUDA CI.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@adrastogi

Copy link
Copy Markdown
Contributor Author

[Agentic Response]: Closing as superseded. #33916 already landed the same test fix on main (own_flag=true and no manual cudaStreamDestroy in both async GPUDataTransferSameBufferTest tests). This PR's only extra change was an explicit stream destroy under USE_CUDA_MINIMAL, which only matters for CUDA-minimal builds with the CUDA EP internal tests enabled. Not worth a separate PR for now.

@adrastogi adrastogi closed this Nov 6, 2026
@adrastogi
adrastogi deleted the adrastogi/fix-cuda-same-buffer-test branch November 6, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CUDA CI: GPUDataTransferSameBufferTest async tests fail on main since #29792 (null cuBLAS handle passed to CudaStream)

2 participants