[Selena Yang (selenayang888)]

Summary

• Reject the known-wrong cuDNN SDPA path for sequence_length_q=1, unequal Q/KV head counts and head size greater than 128 on cuDNN older than 9.9.0. Use the existing attention fallback.
• This is an intended default correctness guard on affected libraries, not an opt-in numerical speed optimization. Unaffected shapes and newer libraries retain their existing selection.
• Qwen3.5-2B VLM package v4 only; model files are unchanged. This description does not cover Qwen3.5-2B-Text or Qwen3-VL.

Previous PR: #34212 (Iteration 1: Add opt-in wide INT4 M1 GEMV with uint8 zero points). The source changes touch disjoint files, so this PR targets main; its accumulated numbers were measured with every previous PR applied.

Series: B0 baseline → #34212 wide GEMV → #34213 cuDNN guard (this PR) → #34214 XQA idle CTAs → #34215 chunked prefill.

Companion: ORT #33728 supplies the wide INT8 kernel included in every accumulated configuration from iteration 1 on.

Performance: Qwen3.5-2B

Accumulated package/runtime results, not isolated gains from this PR. Baseline = B0: rt-base (ORT main 2bf400a), all switches off, decoder capture off. Optimized = #34212 + #34213 (with #33728's INT8 kernel): rt-it1b, wide INT4 + wide INT8 switches on, decoder capture on. Change = Optimized decode relative to Baseline (first table) or llama.cpp (second table). The increment over the previous PR and isolated controls follow each GPU's tables; prefill is reported separately.

A100-SXM4-80GB

Baseline: a100-review-B0; optimized: a100-review-it1bw48cap.

Tokens	Baseline TTFT ms	Optimized TTFT ms	Baseline decode tok/s	Optimized decode tok/s	Change
128	14.99	15.21	213.63	300.62	+40.72%
512	37.59	37.73	214.16	301.80	+40.92%
2048	147.27	147.34	213.97	302.30	+41.28%
4096	293.83	293.97	213.63	304.15	+42.37%
7680	539.18	539.51	213.16	302.34	+41.84%

Same optimized series versus llama.cpp a100-review-llama (identical input IDs, same rounds):

Tokens	llama.cpp TTFT ms	Optimized TTFT ms	llama.cpp decode tok/s	Optimized decode tok/s	Change
128	18.26	15.21	284.83	300.62	+5.54%
512	42.04	37.73	285.05	301.80	+5.88%
2048	157.00	147.34	282.95	302.30	+6.84%
4096	311.30	293.97	279.95	304.15	+8.64%
7680	586.87	539.51	274.41	302.34	+10.18%

• Increment over Add opt-in wide INT4 M1 GEMV with uint8 zero points #34212, this guard plus decoder capture (it1w48 → it1bw48cap): decode +28.71% to +29.93%, TTFT +0.08% to +2.00%; decode regressions at tokens: none.
• Default path, all switches and capture off (B0 → it1boff): decode +0.59% to +1.15%, TTFT -0.44% to -0.01%; decode regressions at tokens: none.
• Prefill ms at 128/512/2048/4096/7680 tokens, measured separately (synchronized append only): baseline 14.89 / 37.49 / 147.17 / 293.71 / 539.08; optimized 15.10 / 37.62 / 147.23 / 293.86 / 539.38; llama.cpp 17.84 / 41.56 / 156.44 / 310.70 / 586.29. Prefill tok/s: fresh/BENCHMARKS.md.

H200

Baseline: h200-review-B0; optimized: h200-review-it1bw48cap.

Tokens	Baseline TTFT ms	Optimized TTFT ms	Baseline decode tok/s	Optimized decode tok/s	Change
128	9.48	9.66	322.73	491.45	+52.28%
512	23.54	23.67	322.07	494.49	+53.54%
2048	89.84	90.29	324.29	490.48	+51.25%
4096	189.82	191.37	321.88	490.17	+52.28%
7680	356.13	356.14	318.82	488.29	+53.15%

Same optimized series versus llama.cpp h200-review-llama (identical input IDs, same rounds):

Tokens	llama.cpp TTFT ms	Optimized TTFT ms	llama.cpp decode tok/s	Optimized decode tok/s	Change
128	12.21	9.66	421.57	491.45	+16.58%
512	24.46	23.67	416.70	494.49	+18.67%
2048	87.68	90.29	417.14	490.48	+17.58%
4096	173.50	191.37	411.14	490.17	+19.22%
7680	325.13	356.14	405.95	488.29	+20.28%

• Increment over Add opt-in wide INT4 M1 GEMV with uint8 zero points #34212, this guard plus decoder capture (it1w48 → it1bw48cap): decode +35.01% to +37.25%, TTFT +0.00% to +2.08%; decode regressions at tokens: none.
• Default path, all switches and capture off (B0 → it1boff): decode -0.06% to +1.07%, TTFT -0.51% to +0.00%; decode regressions at tokens: 2048.
• Prefill ms at 128/512/2048/4096/7680 tokens, measured separately (synchronized append only): baseline 9.39 / 23.44 / 89.74 / 189.67 / 355.99; optimized 9.58 / 23.58 / 90.19 / 191.24 / 356.00; llama.cpp 11.77 / 24.07 / 87.00 / 172.49 / 324.23. Prefill tok/s: fresh/BENCHMARKS.md.

Shared-host medians are descriptive, not statistical significance; negative rows are kept. Small increments must be remeasured on an isolated rebased PR head.

Controls And Configuration

Optimized: rt-it1b with both wide switches=1 and decoder capture on (GenAI enable_cuda_graph=1 on the decoder only; package unchanged). Capture enters the accumulated configuration here because this guard makes H200 captured output match uncaptured (40/50 to 50/50 bit-exact); its speedup comes from the existing GenAI option, not this PR's code. Default-path control: rt-it1b with all switches and capture off. cuDNN 9.8.0 is pinned on both hosts. Timing prompts have >=128 tokens and do not exercise the S=1 first-prompt defect; this is a correctness PR, not a speedup claim.

Published Qwen3.5-2B VLM package v4: 172 INT4 and 15 INT8 MatMulNBits projections, block128/uint8 zero points. FP16 KV, FPA1, context 8192, batch 1, eight CPU threads pinned to 32-39. A100 physical GPU 2; H200 physical GPU 1. llama.cpp uses Q4_K_M, CUDA graphs, flash attention and FP16 KV. Quantization/quality are not asserted equivalent.

Fresh unique-source workload: token prefixes 128/512/2048/4096/7680 of benchmark_src/16384-cache-capacity-diagnostics.txt (text SHA256 3496ea55dead1c0ca9cd72aa6b1ff4bfcfca4433262a228d193a33885d33c49d). Actual token counts/hashes come from the model tokenizer, not the manifest's historical API counts. Both engines consume identical input IDs. Exactly 128 generated tokens; fresh generator, no prefix reuse, tokenization and generator creation excluded. Two warmups + five measurements x three rotating rounds; medians of 15 measured requests per row. TTFT is synchronized native GenAI, not HTTP. Decode uses 127 first-to-last intervals. This is a new workload; do not combine it with historical repeated-seed rows.

Known Limitations

The accumulated decode increase at this step comes from decoder capture, not from the guard; the guard is the H200 correctness prerequisite for capture. Timing prompts start at 128 tokens and do not reproduce the S=1 first-prompt defect. The operator regression has not run against this private no-pybind build; retained characterization does not replace a failing-before/passing-after test. The unrelated A100 captured cached-S=8 mismatch remains unresolved.

The staged runtime binaries are frozen experiments, not benchmarks of a newly split/rebased PR head. B0 is source-built ORT baseline 2bf400a, not the published wheel or Foundry service. Full revision/build pinning, corresponding llama.cpp per-benchmark quality, CPU performance, padded/multimodal coverage and agreed RAM/VRAM budgets remain outstanding. Shared-host median ratios are descriptive; no statistical-significance, cross-GPU scaling or service-level win is claimed.

Review required: retain draft status and default-off numerical optimizations. Do not promote from these timings alone; resolve the iteration-specific failures and every WORK.md gate below.

Existing Work And Duplication

Open ORT #34111 changes cuDNN auto-selection for unshared KV buffers, and open #33424 falls back when the cuDNN graph cannot be built; both are different conditions from this cuDNN<9.9 S=1 grouped-head wrong-output guard but touch the same selection path. Merged #34164 changed cached-plan/capture handling in the same file; this branch is rebased onto it and the guard applies unchanged. The bounded search is not exhaustive duplicate clearance. cuDNN 9.9 already fixes S_q=1 issues; this guard protects older supported libraries.

Iteration-Specific Actions

Extract only the version gate and regression test. Keep it independent of the wide-kernel PR, justify the default behavior change as a correctness fix, and attach reproducer results for affected and unaffected libraries.

Quality Results

Benchmark	llama.cpp Q4_K_M	This exact candidate	Relative loss	Gate
MMLU	Not measured	Not measured	Unknown	BLOCKED
AI2D (multimodal package)	Not measured	Not measured	Unknown	BLOCKED

Use the same evaluation inputs/protocol and compute 100 * max(0, 1 - candidate_score / llama_score); require strictly less than 1 for each benchmark. This report does not transfer scores from other exports or models.