[DSV4] Fuse norm and router for low latency scenario by jeejeelee · Pull Request #41263 · vllm-project/vllm

jeejeelee · 2026-04-29T17:37:45Z

Purpose

Test Plan

performance

DSV4PRO TP8 on B300

Conc	Req/s	Tok/s	Total tok/s	P50 TTFT (ms)	P50 TPOT (ms)
1	0.05 / 0.05	79.59 / 81.24	610.22 / 622.87	464.78 / 465.56	12.27 / 12.01
2	0.10 / 0.10	149.32 / 153.65	1144.79 / 1177.97	744.53 / 739.69	12.89 / 12.52
4	0.17 / 0.18	264.94 / 270.77	2031.17 / 2075.87	1058.05 / 1052.65	14.42 / 14.09
8	0.29 / 0.29	444.52 / 448.46	3407.96 / 3438.20	1182.50 / 1175.03	17.07 / 16.98
16	0.44 / 0.44	681.00 / 677.57	5220.99 / 5194.73	1238.56 / 1179.68	22.39 / 22.65
32	0.61 / 0.61	933.24 / 935.77	7154.81 / 7174.21	1242.79 / 1234.85	33.07 / 33.08

Accuracy

DSV4PRO TP8 on B300

Dataset	ACC
GSM8K	flexible-extract=0.9484 /strict-match=0.9492
AIME25	exact_match=0.9833
GPQA-D	exact_match=0.8952

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

claude

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}

gemini-code-assist

Code Review

This pull request implements a fused RMSNorm and router GEMV kernel specifically optimized for DeepSeek V4 (DSV4-Pro) to reduce kernel launch overhead. The changes include the CUDA kernel implementation, a new NormGatedLinear layer for runtime dispatching between fused and unfused paths, and updates to the DeepSeek V4 model to utilize this optimization. A benchmark script is also provided for performance and correctness verification. Feedback was provided regarding the SM version check in the kernel entry point, which may be overly restrictive for future architectures supported by the build configuration.

gemini-code-assist · 2026-04-29T17:43:25Z

+  TORCH_CHECK(sm >= 90 && sm <= 103,
+              "dsv4_norm_router_gemm requires SM_90 <= CUDA ARCH <= SM_103");


The runtime check restricts the kernel to SM versions up to 103. However, CMakeLists.txt allows building for SM 11.0 and 12.0 (if CUDA 13.0 is used). If the kernel is compiled for these future architectures, it will fail at runtime despite being SM90+ compatible. Consider relaxing the upper bound if the kernel is expected to be forward-compatible.

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Signed-off-by: jeejeelee <jeejeelee@verda-b300-05.datacrunch.io>

claude

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}

) Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: jeejeelee <jeejeelee@verda-b300-05.datacrunch.io> Co-authored-by: jeejeelee <jeejeelee@verda-b300-05.datacrunch.io> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

) Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: jeejeelee <jeejeelee@verda-b300-05.datacrunch.io> Co-authored-by: jeejeelee <jeejeelee@verda-b300-05.datacrunch.io> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Liuweixiong0118 <lwx34158427@gmail.com>

) Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: jeejeelee <jeejeelee@verda-b300-05.datacrunch.io> Co-authored-by: jeejeelee <jeejeelee@verda-b300-05.datacrunch.io> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Init

185b4f5

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

jeejeelee requested review from LucasWilkinson, mgoin, pavanimajety and tlrmchlsmth as code owners April 29, 2026 17:37

claude Bot reviewed Apr 29, 2026

View reviewed changes

jeejeelee marked this pull request as draft April 29, 2026 17:37

Merge branch 'main' into fused-norm-gate-dsv4

1b60ea6

mergify Bot added ci/build deepseek Related to DeepSeek models performance Performance-related issues labels Apr 29, 2026

gemini-code-assist Bot reviewed Apr 29, 2026

View reviewed changes

jeejeelee mentioned this pull request Apr 30, 2026

[Roadmap] DeepSeek V4 #40902

Closed

32 tasks

jeejeelee added 5 commits April 30, 2026 08:03

Move

3de29fd

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Address conflict

c64361b

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

FIX

9d262c3

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

FIX

c669398

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

FIX

24b1c7b

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

wuhuikx mentioned this pull request May 6, 2026

[Performance]: Deepseek-V4 Support and Optimization on ROCm Backend #41820

Open

22 tasks

jeejeelee and others added 3 commits May 12, 2026 08:18

Address conflict

545bbdf

Signed-off-by: jeejeelee <jeejeelee@verda-b300-05.datacrunch.io>

Merge remote-tracking branch 'origin/main' into fused-norm-gate-dsv4

2fcb9f0

Merge branch 'main' into fused-norm-gate-dsv4

302403f

jeejeelee marked this pull request as ready for review May 13, 2026 14:45

jeejeelee requested review from Harry-Chen and zyongye as code owners May 13, 2026 14:45

claude Bot reviewed May 13, 2026

View reviewed changes

zyongye approved these changes May 14, 2026

View reviewed changes

zyongye added the ready ONLY add when PR is ready to merge/full CI is needed label May 14, 2026

mergify Bot and others added 2 commits May 14, 2026 03:05

Merge branch 'main' into fused-norm-gate-dsv4

5e8c456

Merge branch 'main' into fused-norm-gate-dsv4

9700ef1

vllm-bot merged commit 0a65d46 into main May 14, 2026
162 of 164 checks passed

vllm-bot deleted the fused-norm-gate-dsv4 branch May 14, 2026 12:11

weizhoublue mentioned this pull request May 15, 2026

Fix : crash in DeepSeek V4 _forward_rocm due to stale ffn_norm reference after norm-gate fusion #42711

Closed

tjtanaa mentioned this pull request May 16, 2026

[ROCm] [Bugfix] Fix DeepSeek V4 Functionality and Accuracy #42810

Merged

4 tasks

maeehart mentioned this pull request May 17, 2026

[ROCm][DSv4] Functional fixes for DeepSeek V4 on MI300X (gfx942) #42893

Closed

jamestangg mentioned this pull request May 19, 2026

[Tracker]: DeepSeek-V4 Support and Optimization on ATOM ROCm/ATOM#707

Open

Navigation Menu

Uh oh!

Uh oh!

[DSV4] Fuse norm and router for low latency scenario - #41263

[DSV4] Fuse norm and router for low latency scenario#41263
vllm-bot merged 12 commits into
mainfrom
fused-norm-gate-dsv4

jeejeelee commented Apr 29, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 29, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		TORCH_CHECK(sm >= 90 && sm <= 103,
		"dsv4_norm_router_gemm requires SM_90 <= CUDA ARCH <= SM_103");

Uh oh!

Uh oh!

Conversation

jeejeelee commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

performance

Accuracy

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jeejeelee commented Apr 29, 2026 •

edited

Loading