[ROCm] Keep GLM-5.2 on MRV1 and disable default breakable cudagraph - #53155
Open
Rohan138 wants to merge 5 commits into
Open
[ROCm] Keep GLM-5.2 on MRV1 and disable default breakable cudagraph#53155Rohan138 wants to merge 5 commits into
Rohan138 wants to merge 5 commits into
Conversation
vllm-project#52861 routed the DSA models to the V2 model runner (MRV2) and breakable CUDA graphs, excluding DeepseekV32/DeepseekV4 from MRV2 on ROCm ("unsupported by MRV2 or slower with MRV2 on AMD GPUs") but missing GlmMoeDsaForCausalLM. GLM-5.2 (FP8 and MXFP4) thus became the only DSA model defaulting to MRV2 + breakable cudagraph on ROCm, regressing decode latency on gfx950 (~30-37% TPOT at batch 1). Two ROCm-only changes: - Add GlmMoeDsaForCausalLM to the MRV2 exclusion so GLM-5.2 stays on the compiled MRV1 path like the other DSA models. - Stop defaulting any architecture to breakable CUDA graphs on ROCm, where it currently regresses performance. Opt-in via VLLM_USE_BREAKABLE_CUDAGRAPH=1 is unchanged. Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Rohan138
requested review from
ProExpertProg,
WoosukKwon,
houseroad,
mgoin,
robertgshaw2-redhat,
tlrmchlsmth,
yewentao256 and
youkaichao
as code owners
August 20, 2026 18:59
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Collaborator
Author
AndreasKaratzas
approved these changes
Aug 24, 2026
Member
|
/ci run |
|
✅ @Rohan138, CI is now available for this PR.
|
|
✅ Triggered Buildkite CI #85387 for commit |
…able This PR keeps GlmMoeDsaForCausalLM on the compiled MRV1 path and disables breakable cudagraphs by default on ROCm. Update the config tests to match: - GLM-5.2 no longer defaults to breakable cudagraph on ROCm (test_dsa_breakable_cudagraph_platform_default: GlmMoeDsa/is_rocm=True now expects False). - GLM-5.2 is excluded from both the V2 model runner and breakable cudagraph default sets on ROCm (extend test_rocm_keeps_compiled_deepseek_defaults). Signed-off-by: Rohan Potdar <rohan.potdar@amd.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Should revisit the perf numbers here after #53712, and maybe also let DSV4 go back on MRV2 |
Member
|
/ci run |
Member
|
/amd-ci run |
|
✅ Triggered Buildkite CI #85730 for commit |
|
✅ Triggered Buildkite AMD CI #12389 for commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[ROCm] Keep GLM-5.2 on MRV1 and disable default breakable cudagraph
Purpose
#52861 routed the DSA models to the V2 model runner (MRV2) and breakable CUDA
graphs. On ROCm it excluded
DeepseekV32ForCausalLM/DeepseekV4ForCausalLMfrom MRV2 (the
TODO(rocm)notes these are "unsupported by MRV2 or slower withMRV2 on AMD GPUs") but missed
GlmMoeDsaForCausalLM. GLM-5.2 (both FP8 andMXFP4) thus became the only DSA model defaulting to MRV2 + breakable cudagraph on
ROCm, which regresses decode latency on gfx950 (MI355X) by ~30–37% TPOT at
batch 1 in nightly perf-eval.
This PR, ROCm-only:
GlmMoeDsaForCausalLMto the MRV2 exclusion so GLM-5.2 stays on thecompiled MRV1 path like the other DSA models;
currently regresses performance. Opt-in via
VLLM_USE_BREAKABLE_CUDAGRAPH=1is unchanged.
No effect on non-ROCm platforms.
Not a duplicate
gh pr list --repo vllm-project/vllm --state open --search "52861 in:body"and asearch for the GLM/ROCm model-runner area returned no open PR addressing this;
the only related open PR (#41834) is NVIDIA SM12x DeepSeek-V4 enablement.
Perf results
GLM-5.2-MXFP4, gfx950 (MI355X), TP8,
--load-format dummy, in/out 1024,vllm bench serve --dataset-name random --max-concurrency 1 --num-prompts 10 --ignore-eos(warmup pass discarded), median TPOT:VLLM_USE_BREAKABLE_CUDAGRAPH=0only (still MRV2)Decomposition: MRV2 accounts for ~3.0 ms of the ~4.3 ms regression and breakable
cudagraph for ~1.2 ms, so both changes are needed to fully recover; disabling
breakable alone (env workaround) leaves ~23% on the table.
Testing
Using V2 Model Runner/Breakable CUDA graph enabledstartup logs are absent with this PR.tests/test_config.pycovers the ROCm MRV2 / breakable defaults; thesubmitter should run
pytest tests/test_config.py -k "rocm or dsa or breakable"in a built env (not runnable in the perf container used here).
AI assistance
This change was prepared with AI assistance (Claude) and reviewed by the
submitter.