sm12x: add tuned FP8 W8A8 block config for N=4096,K=12288 - #37
Conversation
DeepSeek-V4-Flash uses an N=4096, K=12288 dense FP8 block GEMM that has no
tuned config for RTX PRO 6000 Blackwell, so every boot logs:
Using default W8A8 Block FP8 kernel config. Performance might be
sub-optimal! Config files not found at [...N=4096,K=12288,
device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition...]
Tuned with benchmarks/kernels/benchmark_w8a8_block_fp8.py on 2x RTX PRO 6000
Blackwell Max-Q Workstation Edition (SM120), covering the full default batch
sweep M = 1..4096. Filed under the non-Max-Q device name to match the existing
configs from e8b14e2 and the Max-Q/Server aliases in
_SM12X_TUNED_CONFIG_DEVICE_ALIASES -- same silicon and memory bandwidth, so one
config serves the family.
Measured honestly: this removes the warning and replaces a heuristic tile choice
with measured ones, but a same-session A/B showed NO throughput gain beyond
run-to-run drift (decode: tuned 285.5 avg, baseline 286.3 avg, first-run-of-
session 303.1 -- the apparent win was ordering drift). This shape is evidently
not hot enough in the step time to matter. Submitted as correctness/tidiness for
the config set, not as a performance improvement.
Signed-off-by: Alex Bilichenko <abilichenko@gmail.com>
|
Merged as de114f4 — thank you. Verified before merging: 18 entries spanning M=1..4096, full key set on every To be explicit about what this merge does and does not vouch for: correctness The part worth calling out is that you caught your own result. A +5.9% that Also picked up your DSv4 illegal-memory-access report on |
…esult jasl/vllm#37 (alexbi29, merged 2026-08-03) added the same N=4096,K=12288 shape for RTX PRO 6000 Blackwell (SM120), five days before this work, using the same upstream tuner and the same 18-point grid. It does not make this repo redundant: vLLM's _SM12X_TUNED_CONFIG_DEVICE_ALIASES maps only the RTX PRO 6000 Max-Q and Server editions onto the Workstation name, and NVIDIA_GB10 is absent, so GB10 resolves to its own filename and finds nothing. Verified against the loader in the build this repo targets. Their result independently corroborates the headline finding: an order-controlled A/B on different SM12x silicon also measured no throughput gain, tracing an apparent +5.9% to first-benchmark-of-session drift -- the same confound encountered here. Two parties, two devices, same conclusion.
2 participants
Adds the one dense FP8 block-GEMM shape DeepSeek-V4-Flash uses that has no tuned config for RTX PRO 6000 Blackwell, so every boot logs
Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal!.Tuned with
benchmarks/kernels/benchmark_w8a8_block_fp8.pyon 2x RTX PRO 6000 Blackwell Max-Q Workstation Edition (SM120), full default sweep M = 1..4096 (18 entries).Filed under the non-Max-Q device name to match the configs added in
e8b14e236dand the aliases in_SM12X_TUNED_CONFIG_DEVICE_ALIASES— same silicon and memory bandwidth, so one config serves the family.Measured honestly: no throughput gain
A same-session A/B, order-controlled:
The tuned re-run matches baseline — the apparent +5.9% was first-benchmark-of-session drift, not the config. Prefill carries the same confound and I did not control for it, so I don't claim those numbers either.
So this is submitted as completeness for the config set (removes the warning, replaces a heuristic tile choice with measured ones, correctness verified), not as a performance improvement. Happy to drop it if you'd rather only carry configs with a demonstrated win.
Context: found while root-causing the DSv4 illegal-memory-access reported in vllm-project#41834 (comment).