[Selena Yang (selenayang888)]

Draft. This branch applies the iteration's files onto main f4cd337; it has not yet been built, tested or measured at this head. Numbers below come from the frozen staged runtimes described under Known Limitations.

Summary

• Stream 16-byte INT4 weights with multiple columns per warp and FP32 accumulation for eligible FP16 M=1 decode GEMVs. Keep the existing path as the default and fallback.
• INT4 and reused INT8 wide switches remain default-off. The accumulated table enables both (the INT8 kernel is [CUDA] MatMulNBits 8-bit: wide fp16 M=1 kernel for decode GEMVs #33728's, as applied in iteration 1); the INT4-only control is listed under each GPU. Decoder capture is not enabled at this step.
• Qwen3.5-2B VLM package v4 only; model files are unchanged. This description does not cover Qwen3.5-2B-Text or Qwen3-VL.

Previous PR: none; this is the first PR in the series. The B0 baseline is reported below and was not opened as a separate PR.

Series: B0 baseline → #34212 wide GEMV (this PR) → #34213 cuDNN guard → #34214 XQA idle CTAs → #34215 chunked prefill.

Companion: ORT #33728 supplies the wide INT8 kernel included in every accumulated configuration from iteration 1 on.

Performance: Qwen3.5-2B

Accumulated package/runtime results, not isolated gains from this PR. Baseline = B0: rt-base (ORT main 2bf400a), all switches off, decoder capture off. Optimized = #34212 (with #33728's INT8 kernel): rt-it1, wide INT4 + wide INT8 switches on, decoder capture off. Change = Optimized decode relative to Baseline (first table) or llama.cpp (second table). The increment over the previous PR and isolated controls follow each GPU's tables; prefill is reported separately.

A100-SXM4-80GB

Baseline: a100-review-B0; optimized: a100-review-it1w48.

Tokens	Baseline TTFT ms	Optimized TTFT ms	Baseline decode tok/s	Optimized decode tok/s	Change
128	14.99	14.91	213.63	233.56	+9.33%
512	37.59	37.50	214.16	233.16	+8.87%
2048	147.27	147.11	213.97	233.49	+9.12%
4096	293.83	293.74	213.63	234.09	+9.58%
7680	539.18	539.09	213.16	232.84	+9.23%

Same optimized series versus llama.cpp a100-review-llama (identical input IDs, same rounds):

Tokens	llama.cpp TTFT ms	Optimized TTFT ms	llama.cpp decode tok/s	Optimized decode tok/s	Change
128	18.26	14.91	284.83	233.56	-18.00%
512	42.04	37.50	285.05	233.16	-18.21%
2048	157.00	147.11	282.95	233.49	-17.48%
4096	311.30	293.74	279.95	234.09	-16.38%
7680	586.87	539.09	274.41	232.84	-15.15%

• INT4 wide only, without [CUDA] MatMulNBits 8-bit: wide fp16 M=1 kernel for decode GEMVs #33728's INT8 kernel (B0 → it1w4): decode +7.03% to +7.36%, TTFT -0.54% to -0.01%; decode regressions at tokens: none.
• Prefill ms at 128/512/2048/4096/7680 tokens, measured separately (synchronized append only): baseline 14.89 / 37.49 / 147.17 / 293.71 / 539.08; optimized 14.80 / 37.40 / 147.01 / 293.64 / 538.98; llama.cpp 17.84 / 41.56 / 156.44 / 310.70 / 586.29. Prefill tok/s: fresh/BENCHMARKS.md.

H200

Baseline: h200-review-B0; optimized: h200-review-it1w48.

Tokens	Baseline TTFT ms	Optimized TTFT ms	Baseline decode tok/s	Optimized decode tok/s	Change
128	9.48	9.47	322.73	364.01	+12.79%
512	23.54	23.50	322.07	363.51	+12.87%
2048	89.84	89.85	324.29	360.15	+11.06%
4096	189.82	189.75	321.88	357.82	+11.16%
7680	356.13	356.13	318.82	355.78	+11.59%

Same optimized series versus llama.cpp h200-review-llama (identical input IDs, same rounds):

Tokens	llama.cpp TTFT ms	Optimized TTFT ms	llama.cpp decode tok/s	Optimized decode tok/s	Change
128	12.21	9.47	421.57	364.01	-13.65%
512	24.46	23.50	416.70	363.51	-12.76%
2048	87.68	89.85	417.14	360.15	-13.66%
4096	173.50	189.75	411.14	357.82	-12.97%
7680	325.13	356.13	405.95	355.78	-12.36%

• INT4 wide only, without [CUDA] MatMulNBits 8-bit: wide fp16 M=1 kernel for decode GEMVs #33728's INT8 kernel (B0 → it1w4): decode +6.22% to +7.87%, TTFT +0.01% to +1.08%; decode regressions at tokens: none.
• Prefill ms at 128/512/2048/4096/7680 tokens, measured separately (synchronized append only): baseline 9.39 / 23.44 / 89.74 / 189.67 / 355.99; optimized 9.38 / 23.42 / 89.74 / 189.62 / 356.01; llama.cpp 11.77 / 24.07 / 87.00 / 172.49 / 324.23. Prefill tok/s: fresh/BENCHMARKS.md.

Shared-host medians are descriptive, not statistical significance; negative rows are kept. Small increments must be remeasured on an isolated rebased PR head.

Controls And Configuration

Optimized: rt-it1 with ORT_MATMUL_NBITS_4BIT_M1_WIDE=1 and ORT_MATMUL_NBITS_8BIT_M1_WIDE=1, capture off, versus rt-base B0. INT4-only control: rt-it1 with only the INT4 switch. Both switches default off. The INT8 switch exists only with #33728's kernel; remeasure after rebasing on it, or report INT4-only numbers if it is not merged.

Published Qwen3.5-2B VLM package v4: 172 INT4 and 15 INT8 MatMulNBits projections, block128/uint8 zero points. FP16 KV, FPA1, context 8192, batch 1, eight CPU threads pinned to 32-39. A100 physical GPU 2; H200 physical GPU 1. llama.cpp uses Q4_K_M, CUDA graphs, flash attention and FP16 KV. Quantization/quality are not asserted equivalent.

Fresh unique-source workload: token prefixes 128/512/2048/4096/7680 of benchmark_src/16384-cache-capacity-diagnostics.txt (text SHA256 3496ea55dead1c0ca9cd72aa6b1ff4bfcfca4433262a228d193a33885d33c49d). Actual token counts/hashes come from the model tokenizer, not the manifest's historical API counts. Both engines consume identical input IDs. Exactly 128 generated tokens; fresh generator, no prefix reuse, tokenization and generator creation excluded. Two warmups + five measurements x three rotating rounds; medians of 15 measured requests per row. TTFT is synchronized native GenAI, not HTTP. Decode uses 127 first-to-last intervals. This is a new workload; do not combine it with historical repeated-seed rows.

Known Limitations

Historical combined w48 strict checks pass only 4/50 positions. The fresh H200 INT4-only prompt-512 outputs diverge after 113 matching generated tokens in all 15 measurements. Task quality of either switch set is unmeasured. Coordinate the INT4 extension with #33728 before any competing PR; the accumulated table includes #33728's INT8 contribution and does not isolate this kernel.

The staged runtime binaries are frozen experiments, not benchmarks of a newly split/rebased PR head. B0 is source-built ORT baseline 2bf400a, not the published wheel or Foundry service. Full revision/build pinning, corresponding llama.cpp per-benchmark quality, CPU performance, padded/multimodal coverage and agreed RAM/VRAM budgets remain outstanding. Shared-host median ratios are descriptive; no statistical-significance, cross-GPU scaling or service-level win is claimed.

Review required: retain draft status and default-off numerical optimizations. Do not promote from these timings alone; resolve the iteration-specific failures and every WORK.md gate below.

Quality Results

Benchmark	llama.cpp Q4_K_M	This exact candidate	Relative loss	Gate
MMLU	Not measured	Not measured	Unknown	BLOCKED
AI2D (multimodal package)	Not measured	Not measured	Unknown	BLOCKED

Use the same evaluation inputs/protocol and compute 100 * max(0, 1 - candidate_score / llama_score); require strictly less than 1 for each benchmark. This report does not transfer scores from other exports or models.

[github-actions]

You can commit the suggested changes from lintrunner.

[github-actions]

Suggested change

	#define M1_WIDE_DISPATCH_ZP(BS) \
	if (zero_points != nullptr) { \
	M1_WIDE_DISPATCH_COLS(BS, true) \
	} else { \
	M1_WIDE_DISPATCH_COLS(BS, false) \
	#define M1_WIDE_DISPATCH_ZP(BS) \
	if (zero_points != nullptr) { \
	M1_WIDE_DISPATCH_COLS(BS, true) \
	} else { \
	M1_WIDE_DISPATCH_COLS(BS, false) \

[github-actions]

Suggested change

	#define M1_WIDE_LAUNCH(BS, ZP, C) \
	#define M1_WIDE_LAUNCH(BS, ZP, C) \