Bump to pytorch 25.05 container along with TE update#13899

Merged

chtruong814 merged 41 commits intomainfrom

chtruong/bump-pytorch-25-05

Jul 6, 2025

Collaborator

chtruong814 commented Jun 12, 2025 •

edited

Loading

What does this PR do ?

Bump to pytorch 25.05 container along with TE update
Also, remove the torch accelerator patch. Doesn't seem necessary given the current version. And update the triton patch. The fix for triton was not in the latest pytorch container.
Set two tests as optional for now:
- L2_VLM_HF_Transformer_PEFT_4bit - automodel test that is failing due to bitsandbytes not compiled for cuda 12.9. Will see if I can resolve later but shouln't be affected in container.
- Optional_L2_Speech_Batch_Size_OOMptimizer_Canary - this is flaky
Added additional import guard for Mcore imports to resolve arm/mac install issues

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

chtruong814 added 4 commits

June 11, 2025 19:31


          Update base container to be pytorch:25.05-py3

5a981c7

Signed-off-by: Charlie Truong <chtruong@nvidia.com>


          Update TE to 2.4

80f5009

Signed-off-by: Charlie Truong <chtruong@nvidia.com>


          Remove torch accelerator patch

d47d040

Signed-off-by: Charlie Truong <chtruong@nvidia.com>


          Update triton patch

3437c93

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 requested a review from ko3n1g

June 12, 2025 00:59

chtruong814 added the Run CICD label

chtruong814 had a problem deploying to test

June 12, 2025 01:00

— with

GitHub Actions Error


          Bump TE and Mcore commits

5d0754f

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

ko3n1g added Run CICD and removed Run CICD labels

ko3n1g temporarily deployed to test

June 12, 2025 01:13

— with

GitHub Actions Inactive

github-actions bot removed the Run CICD label

chtruong814 added the no-fail-fast label


          Fix triton patch

2be52f5

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added the Run CICD label

chtruong814 temporarily deployed to test

June 13, 2025 01:19

— with

GitHub Actions Inactive

github-actions bot removed the Run CICD label

chtruong814 added 2 commits

June 15, 2025 01:41


          Fix triton patch

a7803e5

Signed-off-by: Charlie Truong <chtruong@nvidia.com>


          No fail fast

4904f2a

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 requested review from pablo-garay and thomasdhc as code owners

June 15, 2025 01:42

github-actions bot added the CI label

chtruong814 added the Run CICD label

ko3n1g added Run CICD and removed Run CICD labels

github-actions bot removed the Run CICD label


          Update trt-llm to 0.20.0

9275b23

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added the Run CICD label

github-actions bot removed the Run CICD label


          Merge remote-tracking branch 'origin/main' into chtruong/bump-pytorch…

cd5ca33

…-25-05

github-actions bot removed the Run CICD label

chtruong814 added 3 commits

July 3, 2025 00:47


          Merge remote-tracking branch 'origin/main' into chtruong/bump-pytorch…

782d352

…-25-05


          Ensure top_p is float in nemo_export test script

c470906

Signed-off-by: Charlie Truong <chtruong@nvidia.com>


          Set Optional_L2_Speech_Batch_Size_OOMptimizer_Canary to truly be opti…

34a8a0c

…onal

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added the Run CICD label


          Fix top_k and top_p types in megatronllm_deployable

7275eaf

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

ko3n1g added Run CICD and removed Run CICD labels

chtruong814 added Run CICD and removed Run CICD labels

github-actions bot removed the Run CICD label

chtruong814 added 2 commits

July 5, 2025 14:31


          Revert "Skip failing test_rnnt_logprobs_random after pytorch bump"

fb29767

This reverts commit c6c3a76.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>


          Fix optional export test

67da2f5

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

github-actions bot removed the ASR label

chtruong814 added the Run CICD label

github-actions bot removed the Run CICD label

Contributor

github-actions bot commented Jul 6, 2025

[🤖]: Hi @chtruong814 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

//cc @chtruong814 @ko3n1g @pablo-garay @thomasdhc


          Revert unnecessary changes

0d2d02e

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added the Run CICD label

chtruong814 temporarily deployed to test

July 6, 2025 14:58

— with

GitHub Actions Inactive

Contributor

github-actions bot commented Jul 6, 2025

[🤖]: Hi @chtruong814 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

//cc @chtruong814 @ko3n1g @pablo-garay @thomasdhc

github-actions bot removed the Run CICD label

ko3n1g approved these changes

View reviewed changes

chtruong814 merged commit 0339181 into main

296 checks passed

chtruong814 deleted the chtruong/bump-pytorch-25-05 branch

July 6, 2025 23:58

chtruong814 added the r2.4.0 label

ko3n1g added a commit that referenced this pull request


          Bump to pytorch 25.05 container along with TE update (#13899)

636f9f2

* Update base container to be pytorch:25.05-py3

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update TE to 2.4

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Remove torch accelerator patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Bump TE and Mcore commits

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* No fail fast

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update trt-llm to 0.20.0

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix test_sched_config_parse_reduce_on_plateau

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add no build isolation to TE

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update trt-llm dependencies

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update manifest

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert "Enable LoRA for TELinear layers (#13929)"

This reverts commit 7d9f40f.

* update mcore with wd_mult key fix

Signed-off-by: oliver könig <okoenig@nvidia.com>

* Revert "Revert "Enable LoRA for TELinear layers (#13929)""

This reverts commit 5a1da6c.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix nemo install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix nemo install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix export image build

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Remove unnecessary sed for torch_tensorrt

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update TE and Mcore commits

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add optional tests

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Ensure test script arg types are correct for top_p and top_k

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Increase export deploy timeouts

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Skip failing test_rnnt_logprobs_random after pytorch bump

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Skip coverage artifact config-3.12.py

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Include more config files ot exclude during coverage

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update dependencies

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Ensure top_p is float in nemo_export test script

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Set Optional_L2_Speech_Batch_Size_OOMptimizer_Canary to truly be optional

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix top_k and top_p types in megatronllm_deployable

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert "Skip failing test_rnnt_logprobs_random after pytorch bump"

This reverts commit c6c3a76.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix optional export test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert unnecessary changes

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

---------

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>

chtruong814 added a commit that referenced this pull request


          Cherry pick `Bump to pytorch 25.05 container along with TE update (13…

2afaf47

…899)` into `r2.4.0` (#14145)

* Bump to pytorch 25.05 container along with TE update (#13899)

* Update base container to be pytorch:25.05-py3

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update TE to 2.4

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Remove torch accelerator patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Bump TE and Mcore commits

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* No fail fast

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update trt-llm to 0.20.0

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix test_sched_config_parse_reduce_on_plateau

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add no build isolation to TE

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update trt-llm dependencies

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update manifest

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert "Enable LoRA for TELinear layers (#13929)"

This reverts commit 7d9f40f.

* update mcore with wd_mult key fix

Signed-off-by: oliver könig <okoenig@nvidia.com>

* Revert "Revert "Enable LoRA for TELinear layers (#13929)""

This reverts commit 5a1da6c.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix nemo install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix nemo install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix export image build

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Remove unnecessary sed for torch_tensorrt

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update TE and Mcore commits

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add optional tests

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Ensure test script arg types are correct for top_p and top_k

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Increase export deploy timeouts

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Skip failing test_rnnt_logprobs_random after pytorch bump

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Skip coverage artifact config-3.12.py

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Include more config files ot exclude during coverage

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update dependencies

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Ensure top_p is float in nemo_export test script

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Set Optional_L2_Speech_Batch_Size_OOMptimizer_Canary to truly be optional

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix top_k and top_p types in megatronllm_deployable

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert "Skip failing test_rnnt_logprobs_random after pytorch bump"

This reverts commit c6c3a76.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix optional export test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert unnecessary changes

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

---------

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>

* Set L2_NeMo_2_Export_Deploy_Query_In_Framework to be optional (#13946)

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

---------

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

AmirHussein96 pushed a commit to AmirHussein96/NeMo that referenced this pull request


          Bump to pytorch 25.05 container along with TE update (NVIDIA-NeMo#13899)

63b3cb7

* Update base container to be pytorch:25.05-py3

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update TE to 2.4

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Remove torch accelerator patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Bump TE and Mcore commits

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* No fail fast

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update trt-llm to 0.20.0

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix test_sched_config_parse_reduce_on_plateau

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add no build isolation to TE

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update trt-llm dependencies

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update manifest

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert "Enable LoRA for TELinear layers (NVIDIA-NeMo#13929)"

This reverts commit 7d9f40f.

* update mcore with wd_mult key fix

Signed-off-by: oliver könig <okoenig@nvidia.com>

* Revert "Revert "Enable LoRA for TELinear layers (NVIDIA-NeMo#13929)""

This reverts commit 5a1da6c.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix nemo install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix nemo install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix export image build

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Remove unnecessary sed for torch_tensorrt

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update TE and Mcore commits

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add optional tests

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Ensure test script arg types are correct for top_p and top_k

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Increase export deploy timeouts

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Skip failing test_rnnt_logprobs_random after pytorch bump

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Skip coverage artifact config-3.12.py

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Include more config files ot exclude during coverage

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update dependencies

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Ensure top_p is float in nemo_export test script

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Set Optional_L2_Speech_Batch_Size_OOMptimizer_Canary to truly be optional

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix top_k and top_p types in megatronllm_deployable

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert "Skip failing test_rnnt_logprobs_random after pytorch bump"

This reverts commit c6c3a76.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix optional export test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert unnecessary changes

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

---------

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Amir Hussein <amhussein@nvidia.com>

AmirHussein96 pushed a commit to AmirHussein96/NeMo that referenced this pull request


          Bump to pytorch 25.05 container along with TE update (NVIDIA-NeMo#13899)

7017b8f

* Update base container to be pytorch:25.05-py3

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update TE to 2.4

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Remove torch accelerator patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Bump TE and Mcore commits

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix triton patch

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* No fail fast

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update trt-llm to 0.20.0

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix test_sched_config_parse_reduce_on_plateau

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add no build isolation to TE

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update trt-llm dependencies

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update manifest

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert "Enable LoRA for TELinear layers (NVIDIA-NeMo#13929)"

This reverts commit 7d9f40f.

* update mcore with wd_mult key fix

Signed-off-by: oliver könig <okoenig@nvidia.com>

* Revert "Revert "Enable LoRA for TELinear layers (NVIDIA-NeMo#13929)""

This reverts commit 5a1da6c.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix nemo install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix nemo install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix export image build

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Remove unnecessary sed for torch_tensorrt

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update TE and Mcore commits

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add optional tests

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix install

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Ensure test script arg types are correct for top_p and top_k

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Increase export deploy timeouts

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Skip failing test_rnnt_logprobs_random after pytorch bump

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Skip coverage artifact config-3.12.py

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Include more config files ot exclude during coverage

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Update dependencies

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Ensure top_p is float in nemo_export test script

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Set Optional_L2_Speech_Batch_Size_OOMptimizer_Canary to truly be optional

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix top_k and top_p types in megatronllm_deployable

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert "Skip failing test_rnnt_logprobs_random after pytorch bump"

This reverts commit c6c3a76.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Fix optional export test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Revert unnecessary changes

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

---------

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Amir Hussein <amhussein@nvidia.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI core NLP no-fail-fast r2.4.0 skip-linting