Skip to content

Merge branch 'dev' into optimize_hybrid_ep #4584

Merge branch 'dev' into optimize_hybrid_ep

Merge branch 'dev' into optimize_hybrid_ep #4584

Triggered via push December 12, 2025 02:33
Status Success
Total duration 1h 15m 55s
Artifacts 17

cicd-main.yml

on: push
is-not-external-contributor
11s
is-not-external-contributor
pre-flight  /  pre-flight
10s
pre-flight / pre-flight
cicd-wait-in-queue
3s
cicd-wait-in-queue
cicd-container-build
7m 43s
cicd-container-build
cicd-parse-unit-tests
6s
cicd-parse-unit-tests
Matrix: cicd-unit-tests-latest
cicd-parse-integration-tests
12s
cicd-parse-integration-tests
Matrix: cicd-integration-tests-latest
Nemo_CICD_Test
9s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
ko3n1g
deleted by ko3n1g Dec 12, 2025
(Environment deleted)

Automatically approved by queue manager

Annotations

33 errors and 13 warnings
cicd-container-build
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_model_configs.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/data/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/data/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/pipeline_parallel/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/pipeline_parallel/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_fp8_param.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_fp8_param.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/distributed/fsdp/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/distributed/fsdp/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/moe/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/moe/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/data/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/pipeline_parallel/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/test_fp8_param.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/distributed/fsdp/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/moe/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.

Artifacts

Produced during runtime
Name Size Digest
coverage-unit-test-20154334586-0899c8f4-e3c8-4ff6-9598-79589bc02eaa Expired
126 KB
sha256:160d6ee420de15d919f7e3fef00ae099f96ab27b646f38c78b413ef41f846104
coverage-unit-test-20154334586-2240421b-b33e-44ca-a607-3f024cf29d7c Expired
132 KB
sha256:48926002e896d2df403b958532aef918046d2bdd2b8c4b217df165f8bc8cb6b8
coverage-unit-test-20154334586-2d92623e-390a-4d07-8c98-18af46993137 Expired
130 KB
sha256:a7834b46d449400fef9c37da8156bcee5dad90fe015a3a125a417dc424763eee
coverage-unit-test-20154334586-52bdfab9-dacf-4215-888d-d35bcbc13d0b Expired
128 KB
sha256:2d433ed384a3a5491a58c8a7bd89927b804b42151709f2471467ae18a2d51e10
coverage-unit-test-20154334586-5449b1cb-d6e2-4115-bdaa-d242b2de5dd9 Expired
128 KB
sha256:02db1fc212da422b3f5001da51593586a284854da51d3a695fb07c894d7708ce
coverage-unit-test-20154334586-5611d633-c2d0-4c24-a23a-febd2e9cb29c Expired
130 KB
sha256:7b2b7423bc8376780deb423188a73c9113223a036fb735aa6aa8fd8edeb37140
coverage-unit-test-20154334586-59ec41b1-8878-42d4-bd68-359aa645aac8 Expired
130 KB
sha256:9ad2b275d1b6af08bf481c6ac510fe82f8285c5ebf7fca623300426b75b5b966
coverage-unit-test-20154334586-7d704310-6107-4044-944c-2ed8257326ae Expired
129 KB
sha256:d6afecf22f3e6f2b9a451d18dd9a718c9377d770a105826a4dd3e45b35b3fffb
coverage-unit-test-20154334586-8da6f141-83b7-4d54-86f8-7a3162d9b7c4 Expired
125 KB
sha256:70cade62e9ae97d3a1c70931a128af646ee4adc3a740cfdcb2a7fe3088038f72
coverage-unit-test-20154334586-8fe86add-7915-4835-a4ef-b9fdb2809516 Expired
128 KB
sha256:cb6dcc9e532b4898830a6c8891ba3643ca4e0a7248e5ebdd6207d73148162d02
coverage-unit-test-20154334586-a9c6fed0-3dc1-44a0-a563-eed1a464b1a6 Expired
130 KB
sha256:a6ce245eb3af2e89e1cb7b4181b299bf503d3d8e00fd353957445bd05dc37c77
coverage-unit-test-20154334586-b575e9b1-542e-4c52-ade6-a908ae127360 Expired
129 KB
sha256:578224ab8ac5de1cb089186e9cdfb0557f0ab8828149247ec52f49ffcbe6e390
coverage-unit-test-20154334586-cb6dd896-255c-4c70-b4ff-c89d4281288b Expired
137 KB
sha256:34c6d3fe315aea178ca3a938bf84bb051de2ca9fee1de60dab84b75a789ead70
coverage-unit-test-aggregated Expired
28.9 KB
sha256:504bc8601340d6c088731d33207ae4a454d603ae99dedac420156aca0890f16b
logs-gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer-20154334586-2a62ca93-969d-436f-a33e-ae28f06124ba Expired
55.8 KB
sha256:954c6a63a63aa7dd90edcbedb753951adf01e22c613488ae0ffb2626c88f9eea
logs-gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather-20154334586-3adfbc84-9a7d-48b4-8b17-63a72eb9e058 Expired
128 KB
sha256:98b6a867916e7ee51eec2e725bf68d78d459fdac53fe87127933b91c9a45d036
logs-gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer-20154334586-47bfdc39-e589-47f8-945b-8dde5c95407c Expired
131 KB
sha256:8a03daf87d863c2d2a34441c531c83a33359ba452de451da035f7913ddea157c