[Xavier Dupré (xadupre)]

Description

• Move the MoE CPU offload roadmap and its images to docs/moe-cpu-offload/.
• Remove the obsolete docs/next_steps/ hierarchy.
• Restore the two exploratory routing graphs, their interpretation, and links to the merged foundation PRs.
• Split the remaining implementation into fixed kernel-order hybrid inference followed by adaptive swaps.
• Define nonblocking end-of-inference swaps, CPU-first transfer ordering, and a two-exchange pinned-buffer/staging-slot limit.
• Limit the deferred measurements to token throughput and peak CPU/CUDA memory for now.

Motivation and Context

The session-global expert counters landed in #32738. The roadmap now reflects the remaining work and separates hybrid inference correctness from dynamic placement changes.

Documentation-only change; no tests were run.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The roadmap leaves random placement reproducibility unspecified and describes unimplemented validation as already completed.

Review effort: Balanced
Findings: None

What changed in this PR

Updates the MoE CPU-offload roadmap to reflect completed counter infrastructure and separate hybrid inference from adaptive placement.

Changes:

• Removes the obsolete docs/next_steps indexes.
• Defines fixed random offloading followed by adaptive swaps.
• Leaves end-to-end measurement as the final phase.

File	Description
docs/​next_steps/​started/​index.md	Removes obsolete status index.
docs/​next_steps/​discuss/​index.md	Removes obsolete roadmap link.
docs/​next_steps/​completed/​index.md	Removes obsolete status index.
docs/​moe-cpu-offload/​README.md	Updates status, placement strategy, implementation steps, and validation plan.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟢 Approval recommended

The documentation-only restructuring is coherent, with only a minor numerical correction noted.

Review effort: Balanced
Findings: None

[Copilot]

Copilot review overview

🔵 Needs a closer look

The roadmap presents planned options as available and defines conflicting equal-counter placement ordering.

Review effort: Balanced
Findings: None

Previously missed (2)

In code that hasn't changed since last review

Align documented session options with implementation status

docs/​moe-cpu-offload/​README.md:57

This says all four entries are already exposed, but the session-option header currently defines only alpha and beta; the offload target and epsilon are still listed as future work in Steps 1 and 2. As written, readers may try unsupported keys. Please distinguish implemented options from planned ones and specify whether future offload automatically enables the existing opt-in expert-counting mode or requires that toggle separately.

Define deterministic tie-breaking for expert redistribution

docs/​moe-cpu-offload/​README.md:103

This tie order conflicts with the redistribution objective below. With equal counters, sorting by expert_id before node identity spreads CUDA slots across nodes, whereas lines 198–200 require maximizing fully CUDA-resident nodes and put node index before expert ID. Define separate per-node/global ranking rules or use one consistent global tie order so equal-counter placement has an unambiguous deterministic result.

[Ti-Tai Wang (titaiwangms)]

Approved as a roadmap: validating fixed-placement hybrid inference before adaptive swaps is the right sequence. I would treat these as go/no-go checkpoints for implementation rather than blockers for this documentation PR:

1. At the start of Step 1, demonstrate that a mixed QMoE node can initialize and run when available GPU memory is less than the complete expert-weight requirement. The current planner places whole initializers, and CUDA QMoE prepacking can allocate buffers for all experts. Include initialization-time peak memory, packed weights, scales, and scratch; decide early whether the proposed internal operator/input-memory fallback is needed.
2. Define the initially supported dtypes, quantization modes, and weight layouts. CPU and CUDA QMoE use different packed representations, and the CPU implementation does not support every CUDA configuration (for example BF16 inputs or mixed-width QMoE). Test the same quantized experts on both devices for prefill and decode, and specify what CPU-ready representation is retained or produced when an expert leaves CUDA; copying CUDA-packed bytes to host alone is not sufficient.
3. Move reproducible throughput and peak CPU/GPU memory measurements to the end of Step 1, separating prefill and decode. Routing concentration is promising but does not establish that the more complex asynchronous swap/redistribution policy will improve end-to-end performance. Compare adaptive placement with fixed placement at the same memory envelope before committing to the full Step 2 machinery.

These gates would strengthen an otherwise sensible progression without changing the proposed scope of this PR.