[Performance][MRV2] Remove the per-step D2H metadata barrier from Ascend dense decode

[CubeLander]

背景

Ascend Model Runner V2（MRV2）已经继承了 core vLLM 的 device-resident request state、增量 block table 和设备侧 input preparation；但当前 Ascend 路径仍在每轮结束后把整张 num_computed_tokens 从 NPU 拷回 Host，并在下一轮 prepare_inputs 中无条件执行 event synchronize()，只是为了重建 attention backend 仍需要的 seq_lens_cpu。

当前热路径是：

1. postprocess[_sampled]() → _copy_num_computed_tokens_to_cpu()
2. 下一轮 prepare_inputs() → _update_seq_lens_cpu()
3. num_computed_tokens_event.synchronize()
4. attention builder 又对已有 device query_start_loc 做一次 H2D，并把部分 metadata .tolist()

对非 speculative 的普通自回归 decode，scheduler 交给 runner 的 CPU num_computed_tokens mirror 没有 rejection correction，因而可作为精确值。这条无条件 D2H + Host barrier 会抵消 MRV2 把 metadata 留在设备侧的主要工程收益。

这是一项 engineering/performance hardening，不是新的调度研究，也不改变“请求晚到就进入下一轮”的调度语义。

目标

把 Ascend MRV2 的普通 dense decode 做成真正无每轮 metadata D2H barrier 的路径，并用真实 NPU workload 证明正确性和收益边界；speculative decode、CP、dynamic EPLB 分阶段处理，不混在第一刀里。

当前状态（2026-07-31）

状态：工程实现和证据整理已完成；等待 tutor/maintainer review，不在 review 前扩大适用范围或提出 default-on。

• verified-core 兼容性前置项已按上游补丁做带归因的窄适配，见 draft PR #190；CI 已通过。
• metadata 实现位于 jingyuan/mrv2-device-metadata-188-20260731：ordinary single-token 路径绕过 num_computed_tokens D2H/event barrier，multi-token/spec/diffusion 保留 device-authoritative correction。
• exact core 1f486d96 下 combined worker suite 39 passed；Ruff、markdownlint、JSON 与 diff 检查通过。
• 一张卡 MRV2 graph smoke 已覆盖模型加载、KV cache、graph capture、prefill 和 decode；full-weight Qwen2.5-7B A/B 输出完全一致。
• 八步 profiler 证明目标 acl_memcpy_device_to_host 和 wait_event 各减少 8 次，即每个 decode step 各去掉一次。
• full-weight in-process 平均耗时下降 1.61%（batch 1）和 1.92%（batch 8）。
• 服务级三次独立启动/变体、共 576 个请求、0 失败：吞吐提升 1.51%–1.75%，Mean TPOT 下降 1.49%–1.68%；但 P99 ITL 上升 2.16%–2.32%，不声称 tail latency 改善。
• reviewer 入口和全部原始证据：evidence packet。

第一实现切片（本 issue 立即落地）

• 在普通 single-token 路径中，直接使用 scheduler-maintained CPU request state 生成 seq_lens_cpu
• 跳过每轮 num_computed_tokens D2H copy 与下一轮 event synchronize
• speculative decode 和其他 multi-token generation（包括 diffusion-style）保留现有 device-authoritative correction 路径，避免 rejection 后长度漂移
• 增加单测，分别证明：
  • 普通 single-token 路径不等待 NPU event，且长度由 scheduler mirror 正确生成
  • multi-token 路径仍等待 copy completion，并使用 rejection-corrected device result
  • postprocess 只在需要 device-authoritative correction 时安排 D2H
• 修复/补齐一张卡 dense MRV2 correctness smoke，覆盖已声明的 graph 支持边界

后续里程碑

M1：去掉重复 metadata 搬运

• 复用 model runner 已有的 device query_start_loc，取消 attention builder 的重复 H2D
• 在 FIA/MLA operator ABI 能直接消费 tensor 的位置，减少 .tolist() / scalar materialization
• 不在 Host 侧重新发明一套与 device state 并行漂移的状态机

M2：真实负载评测

按 #146 的测量契约固定 core/plugin/model/CANN/torch_npu 和 workload：

• MRV1 vs MRV2，同机 same-spec
• batch 1/8/32，短/长 context，graph/eager 分开
• 至少 3 次独立服务启动并交替运行
• 报告 TTFT、TPOT/TBT、throughput、P95/P99、错误率、峰值显存
• profiler/counter 单独报告 Host prepare time、D2H/H2D 次数和 device sync 次数

只有机制证据与端到端结果一致，才讨论默认启用 MRV2；smoke 或单次 probe 不算性能结论。

M3：能力边界

• speculative decode 与其他 multi-token generation 分别设计 accepted-token correction，不偷用普通 decode 的假设
• CP / dynamic EPLB / pooling 等能力分别补齐后，再扩大默认适用范围
• eager 基础用例恢复后才把 eager 纳入正式支持声明

验收标准

1. scoped ordinary single-token decode 每轮不再发生 num_computed_tokens D2H copy 或对应 event synchronize。
2. scheduler request churn（new / cached / finished / preempted）下 seq_lens_cpu 与 device seq_lens 保持一致。
3. speculative decode 与 diffusion-style multi-token 行为不变，rejection 后长度无漂移。
4. 至少一个真实一张卡 dense workload 给出可复现的正确性与性能 artifact；若端到端无收益，也如实记录并停止扩大投入。
5. unsupported feature fail closed，不静默切到错误 metadata 语义。

与现有工作的关系

• perf(attention): vectorize prefill/decode boundary computation #19 / [Research][Performance] Shape-adaptive prefill/decode boundary and attention dispatch #165 关注 attention boundary 与 dispatch；本 issue 聚焦 MRV2 request metadata residency 和跨轮 Host barrier，不重复其 boundary 向量化工作。
• 补齐 Ascend 性能相关 PR 的正式评测与排行榜记录 #146 是正式 NPU 评测台账，本 issue 的结果应按其 artifact 规范落账。
• core vLLM MRV2 已提供 device-resident state 和 staged writes；这里优先吸收其机制，而不是另造 scheduler/simulator。

负责人

• GitHub assignee：@CubeLander
• Fletcher × Lumi 共同推进；实现将标注 AI assistance，并由 Fletcher 做最终 human review。

[CubeLander]

第一实现切片已经在本地分支 jingyuan/mrv2-device-metadata-188-20260731 开始：

• ordinary single-token path 现在直接使用 scheduler-maintained num_computed_tokens_np，不再安排每轮 D2H，也不等待 num_computed_tokens_event；
• speculative / diffusion-style multi-token path 继续走 device-authoritative correction，fail closed；
• 新增了两条 authority path、D2H gating 和 diffusion-style guard 的单测；
• 设计文档和 ROADMAP.md 已落地，明确 engineering scope、真实负载门禁和负结果停止条件。

当前验证：

• targeted worker CPU tests: 37 passed
• new MRV2 unit tests: 4 passed
• Ruff 0.14.0 lint + format: passed
• git diff --check: passed

真实一张卡 MRV2 smoke / profiler 仍待做，因此目前不宣称端到端收益。任务专用 source-dev container 已验证 live source，但 NPU round-trip 因现有共享容器的 runtime 占用而未能初始化；没有进入、停止或修改该共享工作负载，测试容器已清理，设备 6 回到 idle baseline。

[CubeLander]

硬件复核更新：此前不是 NPU 容量不足。

• 当前 8 张 910B2 中，NPU 0/1/2/3/6/7 都是 idle baseline（无 device process、AICore 0%、约 3.4 GiB 驱动基线 HBM）；只有 4/5 有真实 Qwen3-30B-A3B TP2 负载。
• 独立新容器无法初始化，是因为现有 all-device source-dev container 已持有这组 runtime；同一容器内用 ASCEND_RT_VISIBLE_DEVICES=0 做了 tensor round-trip，device_count=1、结果正确。因此可用硬件已经确认。
• 随后在 NPU 0 上用 Qwen2.5-14B BF16 + VLLM_USE_V2_MODEL_RUNNER=1 做真实 smoke。模型成功加载到 27.54 GiB，并进入 MRV2 初始化；未到达请求执行，因为当前 main 的既有 core/plugin pairing 有两个启动门：
  1. HUST core e4ce33646：graph_manager_wrapper.factory() 不接受 core 新增的 lora_capture_cases；
  2. 仓库声明的 verified core a30addc754：downstream plugin 又已经依赖更晚出现的 _compute_global_logsumexp helper。

所以 hardware gate 已解除，当前 blocker 是 MRV2 main2main compatibility，而不是卡不够。没有把启动失败计作性能结果。临时 overlay、cache 和测试进程已清理，NPU 0 已回到 idle baseline；4/5 的既有负载没有被修改。

[CubeLander]

Upstream audit：MRV2 baseline compatibility prerequisite

真实 NPU smoke 暴露的问题并非本 issue 的 metadata 改动导致；失败发生在进入 _update_seq_lens_cpu() 热路径之前。上游已有可直接吸收的补丁，不需要重新设计：

1. Graph manager / LoRA capture API

   • core vLLM 在 vllm#35536 给 MRV2 graph manager 增加了 lora_capture_cases / lora_capture_hook。
   • 官方 vllm-ascend 已在 vllm-ascend#10935 完成 Ascend 适配：factory 接收并转发 lora_capture_cases，ModelAclGraphManager.capture() 按 keyword 转发 lora_capture_hook。
   • HUST 当前 main 尚未包含该补丁，对新 core 会报 factory() got an unexpected keyword argument 'lora_capture_cases'。

2. Rejection sampler helper rename

   • core vLLM 的 block-verification 变更 vllm#46781 引入 _compute_global_logsumexp / _compute_local_logits_stats_kernel。
   • 官方 vllm-ascend 在 vllm-ascend#10454 用 version-aware import 同时支持 v0.23.0 与 main。
   • HUST plugin 已采用新 helper 名称，但仓库 .github/vllm-main-verified.commit 仍指向缺少这些 symbol 的旧 core，因此 declared pin 组合在 import 阶段失败。

**决定：**保持 status: in progress。先最小采用上述 upstream compatibility hunks（保留作者/PR 归因），以一张卡 dense MRV2 smoke 验证 baseline；只有 baseline 恢复后，才继续 #188 的端到端性能测量。不会把兼容性修复包装成 #188 的性能收益。

[CubeLander]

Progress update (2026-07-31):

• Opened draft PR [BugFix][Worker] Align MRV2 with verified vLLM APIs #190 with the narrow verified-core compatibility stack. It adopts the relevant upstream vllm-ascend changes for the LoRA graph arguments, rejection-sampler helper rename plus AscendInputBatch fields, and the kv_cache_config reshape API. Original author attribution is preserved.
• Stacked the metadata experiment on that compatibility branch. The combined worker suite against exact vLLM 1f486d96 reports 39 passed; Ruff and git diff --check pass.
• On physical NPU 3, a Qwen2.5-7B-architecture BF16 dummy-weight graph smoke completed model load, KV-cache initialization, graph capture, prefill, and decode. This validates the MRV2 execution path, not output quality.
• Ran an early interleaved A/B probe with three independent engine starts per variant. Mean elapsed time was 1.50% lower at batch 1 and 0.49% lower at batch 8, but the observed ranges overlap substantially. We therefore do not claim a performance gain from this probe.

Next discriminating step: repeat with full weights and profiler counters for Host preparation, D2H copies, and event synchronization, then decide whether the end-to-end benefit is real.

[CubeLander]

Full-weight follow-up (2026-07-31):

• Ran Qwen2.5-7B-Instruct BF16 on physical Ascend 910B2 NPU 3 with exact vLLM 1f486d96, FULL_DECODE_ONLY graphs, and identical greedy outputs across every baseline/optimized run.
• Corrected an initial lazy graph-warmup artifact, then collected a balanced run with 3 independent engine starts per variant × 7 timed repeats per batch.
• Aggregate elapsed time was 1.61% lower at batch 1 and 1.92% lower at batch 8. The optimized variant was faster in all three paired engine-start means.
• In a separate batch-8, eight-step profile, acl_memcpy_device_to_host fell from 24 to 16 and wait_event from 16 to 8: exactly one D2H copy and one matching wait removed per decode step. Token outputs were byte-for-byte equivalent after canonicalization.

This is now positive mechanism evidence plus a small end-to-end signal, but still not a publication-grade claim: the host was shared, CPU binding was unavailable in the container, only three independent starts were collected, and the workload covered short decode.

Evidence ledger and reproducible runner: https://github.com/CubeLander/vllm-ascend-hust/tree/jingyuan/mrv2-device-metadata-188-20260731/benchmarks/evidence/issue188

Next gate: longer-context service-level TTFT/TPOT/TBT/throughput/tail-latency runs with CPU binding and an otherwise isolated card.

[CubeLander]

Service-level follow-up (2026-07-31):

• Ran the exact core/plugin A/B through the OpenAI-compatible vllm serve path with full Qwen2.5-7B-Instruct BF16 weights on physical NPU 3.
• Collected 3 independent server starts per variant in balanced order. Each start ran two fixed 512-input/128-output-token scenarios: 64 saturated requests at concurrency 8, and 32 target-QPS-1 requests at concurrency 1.
• All 576 measured requests completed; 0 failed.

Aggregate optimized deltas:

Scenario	Throughput	Mean TTFT	Mean TPOT	P99 TPOT	Mean E2E	P99 ITL
Saturated	+1.51%	-1.27%	-1.49%	-1.13%	-1.48%	+2.32%
Concurrency 1	+1.75%	-2.76%	-1.68%	-1.56%	-1.71%	+2.16%

Two of three paired starts were positive; one was slightly negative. So the small mean throughput/latency signal survives the service boundary, but it is not yet robust across starts, and P99 ITL regressed in both aggregates. We should preserve that tail-jitter result rather than turn the averages into a broader claim.

Raw benchmark JSON plus aggregate ledger: https://github.com/CubeLander/vllm-ascend-hust/tree/jingyuan/mrv2-device-metadata-188-20260731/benchmarks/evidence/issue188/service-2026-07-31

Next useful gate is a CPU-bound, otherwise isolated repetition with more independent starts, specifically testing whether the P99 ITL movement is workload noise or a real tradeoff.

[CubeLander]

Review handoff complete.

A small scope clarification for reviewers: this issue is not a new attention kernel. It removes the ordinary MRV2 decode path's per-step request-metadata D2H barrier; the attention backend itself is unchanged.

• Compatibility prerequisite: draft PR [BugFix][Worker] Align MRV2 with verified vLLM APIs #190 (CI green, upstream fixes adopted with attribution).
• Metadata/evidence branch head: 773ef3a24 on jingyuan/mrv2-device-metadata-188-20260731.
• Reviewer index: https://github.com/CubeLander/vllm-ascend-hust/blob/jingyuan/mrv2-device-metadata-188-20260731/benchmarks/evidence/issue188/README.md
• Raw in-process, profiler, and service artifacts are linked from that index.

The scoped engineering task and evidence packet are complete. The issue remains open without the status: in progress label for tutor/maintainer review. No wider enablement or default-on change is proposed before that review, and the P99 ITL regression remains explicitly recorded.

[ShuhaoZhangTony]

这份证据已经能支持“去掉普通单 token decode 每步一次 D2H copy 和一次 event wait”这一机制结论：profiler 计数与预期严格对应，full-weight 输出一致，服务层平均吞吐和延迟也有约 1.5%–1.9% 的同向信号。

现阶段还不适合默认开启。三次独立启动中只有两次为正，且两个场景的 P99 ITL 都回退约 2.2%。建议下一步只做一轮判别性实验：固定 CPU affinity、独占卡，baseline/optimized 交错运行至少 5 次独立 server start，保留逐请求 ITL 分布。如果平均收益仍稳定且 P99 ITL 的置信区间不再显示回退，可以提交独立优化 PR；否则保留为 opt-in 实验路径，并把 tail jitter 作为后续问题。

代码审查时重点确认 single-token authority 判定是 fail closed：任何 speculative、多 token 或无法证明 scheduler CPU state 完整的路径都必须回到 device-authoritative correction。#190 只作为兼容性前置，不计入本优化的性能收益。

[CubeLander]

CPU-bound discriminating follow-up complete (2026-08-01).

Conditions requested in the previous review were applied:

• full Qwen2.5-7B-Instruct BF16 weights, exact vLLM 1f486d96;
• physical Ascend 910B2 NPU 4, otherwise idle;
• server pinned to CPUs 96-119 and client pinned separately to CPUs 120-127;
• 5 complete independent server starts per variant in balanced order;
• detailed per-request TTFT and per-token ITL retained;
• 960 analyzed requests completed, 0 failed.

Optimized deltas:

Scenario	Throughput	Mean TPOT	Mean E2E	P99 ITL (95% paired-start cluster-bootstrap CI)	Positive throughput pairs
Saturated, concurrency 8	-0.41%	+0.40%	+0.41%	+1.65% (-0.64%, +2.98%)	2/5
Target QPS 1, concurrency 1	-0.17%	+0.17%	+0.17%	-0.65% (-2.59%, +0.80%)	3/5

The earlier 1.5%-1.9% mean service gain did not survive CPU binding and five independent starts. Every mean-metric confidence interval includes zero. P99 ITL is also inconclusive rather than establishing a regression or an improvement.

Decision: the D2H-copy/event-wait removal mechanism remains reproduced, but the service-level performance claim is NOT_REPRODUCED under the discriminating conditions. The preregistered gate rejects a standalone optimization PR and default enablement; the branch remains experimental evidence only. PR #190 remains a separate compatibility prerequisite.

Protocol deviation is preserved explicitly: the experiment was paused after an optimized-r5 saturated result but before QPS-1. That partial result is retained under excluded/; a fresh optimized-r5 server reran both scenarios, and only the complete replacement is analyzed.

Evidence branch head: 73c93ce0f

• Reviewer summary: https://github.com/CubeLander/vllm-ascend-hust/blob/jingyuan/mrv2-device-metadata-188-20260731/benchmarks/evidence/issue188/service-2026-08-01/README.md
• Aggregate ledger: https://github.com/CubeLander/vllm-ascend-hust/blob/jingyuan/mrv2-device-metadata-188-20260731/benchmarks/evidence/issue188/service-2026-08-01/summary.json
• Preregistered protocol: https://github.com/CubeLander/vllm-ascend-hust/blob/jingyuan/mrv2-device-metadata-188-20260731/benchmarks/evidence/issue188/service-tail-protocol-2026-08-01.json