[amd] MI350 tests are failing

[xuzhao9]

Last good commit: f7c1d69401e9f09050451f30776562954b05e850
Pytorch: 2.12.0.dev20260412+rocm7.2
Last good workflow: https://github.com/meta-pytorch/tritonbench/actions/runs/24361986643/job/71143820345

First bad commit: f7c1d69401e9f09050451f30776562954b05e850
Pytorch: 2.12.0.dev20260412+rocm7.2
First bad workflow: https://github.com/meta-pytorch/tritonbench/actions/runs/24363474649/job/71148992904

Plan to run a bisect to pinpoint the rest of the failures.

Tring to bisect on test_gpu_tritonbench_grouped_gemm:

Reproduce command:

TORCHINDUCTOR_COMPILE_THREADS=1 python -m unittest test.test_gpu.main -k test_gpu_tritonbench_grouped_gemm

Last good Triton commit: f7c1d69401e9f09050451f30776562954b05e850
First bad Triton commit: ce5391b2a49839a5f8f018f4ff9941559e3ce624

[xuzhao9]

Bisect workflow try 1: https://github.com/meta-pytorch/tritonbench/actions/runs/25162717614

Try 2: https://github.com/meta-pytorch/tritonbench/actions/runs/25163505833

Try 3: https://github.com/meta-pytorch/tritonbench/actions/runs/25163973685

Try 4 (clean up both $HOME/.triton and build/ in pre-flight checks): https://github.com/meta-pytorch/tritonbench/actions/runs/25164547848

Try 5 (clean up $HOME/.triton in tritonparse) : https://github.com/meta-pytorch/tritonbench/actions/runs/25172055722

Try 6 (skip compile fail): https://github.com/meta-pytorch/tritonbench/actions/runs/25220921781

[xuzhao9]

Bisect points to triton-lang/triton@df82d98

[xuzhao9]

Kicking off another bisect with gdpa:

Reproduce command:

TORCHINDUCTOR_COMPILE_THREADS=1 python -m unittest test.test_gpu.main -k test_gpu_tritonbench_gdpa

Last good Triton commit: f7c1d69401e9f09050451f30776562954b05e850
First bad Triton commit: ce5391b2a49839a5f8f018f4ff9941559e3ce624

https://github.com/meta-pytorch/tritonbench/actions/runs/25226177908

[zhanglx13]

@xuzhao9 It's caused by llvm bug and should be fixed by llvm/llvm-project#193499

[xuzhao9]

Thanks @zhanglx13 , we pick-up nightly updates from triton-lang/triton. Once the upsteam includes the llvm fix, the trunk should reflect it on the next day.

[xuzhao9]

Pin AMD docker to a known good commit for now: a3b5df5