FastAPI-based working frontend by zhuohan123 · Pull Request #10 · vllm-project/vllm

zhuohan123 · 2023-03-27T06:19:00Z

Add a FastAPI-based frontend to cacheflow while keeping the old script working.

Remaining TODOs:

Add a README for the FastAPI frontend.
Rename the old script.
Add a gradio demo web frontend.

WoosukKwon

LGTM! Thanks for your effort.

* Add underlying functions * tests done

…sthrough Passthrough trust_remote_code

Wenxh/fp8 on a100 v1 pr

…dp-tcp-placement Port multi-node DP fixes from upstream PR vllm-project#38630

Prior versions of HANDOFF.md claimed "library complete" based on topology coverage (every DSL op has at least one Impl). That was a misread. The old ferrite library has ~1,400 kernel variants and a CSV-backed cost model against 55k rows of measured data; the new library has 7 Impls and an analytical flops/bandwidth proxy. This commit: - Adds a 16-row "Real gap inventory" table comparing old vs new across kernel variants, cost model, fusion patterns, attention variants, scheduled megakernel, codegen features, solver constraints, and test depth. - Replaces the stale "Path to vllm chat working" section (steps 5-9, which assumed the library was ready) with a priority- ordered "Re-scoped path" (Steps A-G) mapping each step to the gaps it closes. - Flags scheduled megakernel (gap vllm-project#10) as HIGH PRIORITY per user. - Relabels "Impl library (complete for Llama/Qwen2 topology)" to "topology coverage for Llama only" to prevent the same misread. No code changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…tion Pitfall-driven hardening collected over an end-to-end RTX PRO 6000 run. Each change is documented inline in reproduce/README.md "Reproduction pitfalls" section. Scripts: - common.sh: skip .venv activation when CONDA_PREFIX is set and vllm is already on PATH (avoids stale shadow venvs) - run_all_paper_artifacts.sh: env -u PHASES around child distshift / concshift scripts (they use PHASES for a different purpose); GPU_OFFSET env support; drop stale PAPER_VS_KVAWARE pointer - multiturn/export_dataset.py: defaults to paper-faithful 3000 conversations / min-turns 6 (was 500 / 8) - real_workloads/run_grid_search.sh: BS_VALUES/TB_VALUES env override; add pd_auto scheduler case - eb_plus/traffic/run_adaptive_selector_cfr.sh, synthetic_e2e/run_grid_search_cfr.sh: DATASET_NAME defaults to "random" (was hardcoded geometric_random) - disagg/run_4gpu_comparison.sh, long_context/run_long_context_*.sh, eb_plus/non_stationary/plot_distribution_shift.py, eb_plus/traffic/analyze_cfr_selector.py: misc path / arg fixes Docs: - reproduce/README.md: full rewrite of Quick Start (dataset prep steps, conda support, requirements/reproduce.txt); add "Reproduction pitfalls" section (vllm-project#1-vllm-project#10, all marked ✅ patched); rename THETA → EB(k̂*); link to PD_SCHEDULER_ENV_VARS.md - reproduce/REPRODUCE.md: minor sync New file: requirements/reproduce.txt — extra deps beyond base vllm (datasets, aiohttp, quart, matplotlib, pandas, scipy). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…code v1 beats FA4 at batched long-ctx - NSTG as fused-decode template param; GEMMA_DECODE_NSTG=1 single-stage ring FALSIFIED at +16-38% (cross-CTA buffering; CTAs phase-align on DMA stalls). Kept env-gated, default 2. - WS-decode v1 (non_causal + record640 at the op): GQA-as-M through the warp-specialized fmha engine, per-seq q_offset := seq_k, carried by the in-tree KV-split path (ws build-kernel flag, frontier fix). Correctness 2-3.5e-4 vs fp64. Untuned at 1.25KB/token: b8/64k 342.1us vs FA4 365.9 (-6.5%) vs fused 395.2 (-13.4%). Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>

…-agnostic wave wake - Remove in-core DP-rank routing from async_llm.py (drop the ROCm auto-route block, _pick_dp_rank_for_request / _ensure_disagg_transfer_id helpers, and the hashlib/current_platform imports). Rank pinning is delegated to the llm-d sidecar/router, per njhill/NickLucche. Delete the now-obsolete tests/v1/engine/test_dp_rank_routing.py. - Reframe the DP first-wave wake in core.py from a current_platform.is_rocm() gate to the external-LB DP mode (data_parallel_external_lb). In external LB the router addresses engines directly, so the coordinator is not in the per-request wake path; internal/hybrid LB stays bit-identical to upstream. Add a debug trace at the wake point (njhill vllm-project#5). - Revert VLLM_ENGINE_HANDSHAKE_TIMEOUT_MINUTES; restore the constant HANDSHAKE_TIMEOUT_MINS = 5 (njhill vllm-project#6). - Guard against division by zero on data_parallel_size_local in moriio_common.py port allocation (Doug vllm-project#10). - Fill the AITER root-cause/fix link (ROCm/aiter#3658) in the sampler warm-up skip TODO. - Pin the decode leg to the prefill DP rank via the X-data-parallel-rank header in the moriio toy proxy example. Signed-off-by: Shiksha Patel <shikpate@amd.com>

[R3] Simplify routed-experts sidecar plumbing

…-agnostic wave wake - Remove in-core DP-rank routing from async_llm.py (drop the ROCm auto-route block, _pick_dp_rank_for_request / _ensure_disagg_transfer_id helpers, and the hashlib/current_platform imports). Rank pinning is delegated to the llm-d sidecar/router, per njhill/NickLucche. Delete the now-obsolete tests/v1/engine/test_dp_rank_routing.py. - Reframe the DP first-wave wake in core.py from a current_platform.is_rocm() gate to the external-LB DP mode (data_parallel_external_lb). In external LB the router addresses engines directly, so the coordinator is not in the per-request wake path; internal/hybrid LB stays bit-identical to upstream. Add a debug trace at the wake point (njhill vllm-project#5). - Revert VLLM_ENGINE_HANDSHAKE_TIMEOUT_MINUTES; restore the constant HANDSHAKE_TIMEOUT_MINS = 5 (njhill vllm-project#6). - Guard against division by zero on data_parallel_size_local in moriio_common.py port allocation (Doug vllm-project#10). - Fill the AITER root-cause/fix link (ROCm/aiter#3658) in the sampler warm-up skip TODO. - Pin the decode leg to the prefill DP rank via the X-data-parallel-rank header in the moriio toy proxy example. Signed-off-by: Shiksha Patel <shikpate@amd.com>

) (vllm-project#6173) ### What this PR does / why we need it? **Scope of Changes**: | File Path | | :--- | |`vllm_ascend/ops/layer_shard_linear.py`| |`vllm_ascend/ops/linear.py`| |`vllm_ascend/ops/linear_op.py`| |`vllm_ascend/worker/worker.py`| | ` vllm_ascend/patch/worker/patch_bert.py` | | ` vllm_ascend/patch/worker/patch_deepseek.py` | | ` vllm_ascend/patch/worker/patch_distributed.py` | | ` vllm_ascend/patch/worker/patch_module.py` | | ` vllm_ascend/patch/worker/patch_multimodal_merge.py` | | ` vllm_ascend/patch/worker/patch_qwen3_next.py` | | ` vllm_ascend/patch/worker/patch_qwen3_next_mtp.py` | | ` vllm_ascend/patch/worker/patch_rejection_sampler.py` | | ` vllm_ascend/patch/worker/patch_rope.py` | | ` vllm_ascend/patch/worker/patch_triton.py` | | ` vllm_ascend/patch/worker/patch_unquantized_gemm.py` | | ` vllm_ascend/patch/worker/patch_v2_egale.py` | |` vllm_ascend/worker/npu_input_batch.py`| |` vllm_ascend/worker/v2/aclgraph_utils.py`| |` vllm_ascend/worker/v2/attn_utils.py`| |` vllm_ascend/worker/v2/model_runner.py`| |` vllm_ascend/worker/v2/sample/gumbel.py`| |` vllm_ascend/worker/v2/sample/penalties.py`| |` vllm_ascend/worker/v2/sample/sampler.py`| |` vllm_ascend/worker/v2/spec_decode/__init__.py`| |` vllm_ascend/worker/v2/spec_decode/eagle.py`| |` vllm_ascend/worker/v2/states.py`| ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.14.0 - vLLM main: vllm-project@d682094 Signed-off-by: MrZ20 <2609716663@qq.com> Signed-off-by: SILONG ZENG <2609716663@qq.com> Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>

…x faster flush The sinkhorn program keeps the whole [R, C] fp32 tile plus a working copy live; at 4 warps that is several KB of registers per thread, so the compiler spills to CUDA local memory and the driver permanently reserves local_bytes x max_threads x num_SMs of device memory for the context (~2 GiB for the [256, 128] head_dim-256 tile on a 188-SM part). This was silently charged to vLLM's memory profile and shrank the derived KV-cache capacity (issue vllm-project#10). 8 warps: ~70% less reserved local memory AND ~4x faster flush (2.50 -> 0.61 ms for a 256-tile batch — the spills were also the kernel's bottleneck; 16 warps saves a bit more memory but is 2x slower than 8). Balanced-tile output unchanged within fp32 reduction noise (~5e-7 rel); Qwen3-4B kvarn-vs-fp16 greedy decode remains 100% token-identical.

Without an explicit --block-size, the generic get_preferred_block_size fallback returns the MINIMUM supported kernel block size (64) whenever the framework default (16) is unsupported. A g128 preset then builds its cache with 64-token kernel blocks and dies on the 'KVarN requires block_size (64) == group (128)' assert — e.g. any hybrid-model deployment without spec decode (with MTP the hybrid block-size reconciliation happened to land on 128, which is why issue vllm-project#10 reporters never hit it; explicit --block-size 128 also avoids it). Override get_preferred_block_size on the KVarN backend to return the active preset's group. Validated: Qwen3.6-27B TP2 without spec decode now starts ('Setting kv cache block size to 128 for KVARN backend') and serves; with-MTP behaviour unchanged.

…ffers Follow-up to the issue vllm-project#10 capacity fixes — recovers another ~1.6 GiB of KV budget by making the startup memory accounting honest: 1. Materialize ALL of KVarN's lazy state (per-layer fp16 tail pools, shared decode scratch, kernel JIT + autotune) inside the worker's memory-profiling window. The dummy profile run skips attention (attn_metadata=None), so this state previously first ran inside the CUDA-graph memory estimation warmup and was charged to the 'graph memory' estimate (1.7 GiB mislabeled; the real graph pool is ~54 MiB), with the pools additionally double-counted against the explicit worker reservation. The arithmetic pool reservation is dropped — the allocation is now real and measured. Accounting stays correct with VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. 2. Size the split-K partial buffers (mid_o / mid_lse) by what the split-K decode path can actually index — N = max_num_seqs x Hq rows — instead of the max_num_batched_tokens-driven q_rows (~85x oversized: 256 MiB -> ~3 MiB at typical configs). The decode driver additionally falls back to the single-stage kernel if a (padded dummy) batch ever exceeds the buffer rows. Qwen3.6-27B AWQ TP2 (issue vllm-project#10 configs), per rank: - CUDA graph estimate 1.76 -> 0.08 GiB (= fp8's), available KV 5.2 -> 6.77 GiB, capacity 540,503 -> 699,951 tokens (1.87x fp8's 373,445; the original report was ~0.1x). - without spec decode: 774,516 tokens, decode 70-72 tok/s coherent (exercises the resized split-K path); with MTP: 49 tok/s coherent, 8-way concurrent 0 errors. - Qwen3-4B kvarn-vs-fp16 greedy regression: 100% token-identical.

…project#10 loops) Prefix caching shares/recycles blocks across requests; the dense builder kept per-request flush state and discarded finished requests' pool data, so cache hits read stale tiles and an empty first block — surfacing as repetition loops. Reworked to a per-block, sharing-safe lifecycle (same design as the MLA fix 2e8f5cad7): reclaim flushes complete blocks, sinks stay fp16-resident and are evicted lazily. Cache-hit replay now byte-identical on the reporter's Qwen3.6-27B-AWQ setup, with and without MTP; repro + stress scripts in scripts_kvarn_dense/.

…lm-project#10) Fixes the two regressions reported after the MTP fix (d7bbff123): decode collapsing to <5 tok/s, and KV-cache capacity ~10x below fp8. 1. _cached_multiquery_path: a spec-decode verify step (or chunked-prefill continuation) now builds the batch's rotated fp16 K/V with the ONE block_table-driven Triton kernel and runs a single flash_attn_varlen call. FA's varlen causal mask is bottom-right aligned for seqlen_q < seqlen_k — token t attends keys <= cached_len + t — exactly the verify semantics, so no explicit mask is needed. Replaces _decode_path_slow on this route (per-request Python gather with per-block .item() syncs + Python dequant + fp32 SDPA, per layer, per step). The slow path remains the fallback for head_dim > 256 or a batch whose total KV exceeds the materialize scratch. 2. _warm_decode_kernels: compile + autotune the fused decode kernel, split-K stage1/2, and the packed-KV build kernel at profile time on tiny synthetic state. They never ran during vLLM's prefill-shaped profiling, so their one-time JIT/autotune cost (and, before fix 1, the slow path's fp32 materializations) landed in the CUDA-graph memory estimation warmup and was charged as graph memory — GiBs off the derived KV capacity. Qwen3.6-27B AWQ TP2 + MTP(3) + prefix caching + chunked prefill (the issue vllm-project#10 configs): - KV capacity 92,800 -> 540,503 tokens (fp8 baseline: 373,445; KVarN is now 1.45x fp8 instead of 0.25x); --max-model-len -1 fits the full 262K context again. - decode 43-47 tok/s single stream, 205 tok/s at 8 concurrent, coherent output, 0 errors (was <5 tok/s). - Qwen3-4B no-spec regression: kvarn-vs-fp16 greedy decode 100% token-identical.

Under MTP every decode step is a verify step, and the previous route materialized the request's ENTIRE context into fp16 scratch (then ran flash attention over it) on every step, per layer — an O(context) per-step cost that dominated long-context decode (issue vllm-project#10 follow-up: 5-12 tok/s on long agent sessions). Both fused dual-source decode kernels (single-stage and split-K stage 1) gain a VQ_INDIRECT mode: one program per (verify token, KV head), with a block-table row indirection and a per-token bottom-right causal length, so the verify step reads int4 tiles and the fp16 tail pool directly — no materialization. New driver kvarn_verify_attention; the backend routes by context depth (KVARN_FUSED_VERIFY_MIN_BLOCKS, default 64 blocks): the materialize route stays for short contexts, where its single round-trip is cheap and measured faster, and for chunked-prefill continuations, where one materialization amortizes over thousands of query tokens. KVARN_FUSED_VERIFY=0 forces the previous behavior. The kernel warmup covers the new specializations so the first verify step does not JIT. Measured (Qwen3.6-27B AWQ, single GPU, single stream, greedy, MTP k=3): decode 45.2 -> 51.6 tok/s at 32K context (+14%), parity at shorter contexts; the advantage grows with depth since the kernel reads ~4x fewer bytes than the fp16 round-trip. MTP long generation stays coherent, and the non-speculative path is byte-identical (Qwen3-4B greedy battery) since existing launch sites pass VQ_INDIRECT=False.

…x faster flush The sinkhorn program keeps the whole [R, C] fp32 tile plus a working copy live; at 4 warps that is several KB of registers per thread, so the compiler spills to CUDA local memory and the driver permanently reserves local_bytes x max_threads x num_SMs of device memory for the context (~2 GiB for the [256, 128] head_dim-256 tile on a 188-SM part). This was silently charged to vLLM's memory profile and shrank the derived KV-cache capacity (issue vllm-project#10). 8 warps: ~70% less reserved local memory AND ~4x faster flush (2.50 -> 0.61 ms for a 256-tile batch — the spills were also the kernel's bottleneck; 16 warps saves a bit more memory but is 2x slower than 8). Balanced-tile output unchanged within fp32 reduction noise (~5e-7 rel); Qwen3-4B kvarn-vs-fp16 greedy decode remains 100% token-identical.

Without an explicit --block-size, the generic get_preferred_block_size fallback returns the MINIMUM supported kernel block size (64) whenever the framework default (16) is unsupported. A g128 preset then builds its cache with 64-token kernel blocks and dies on the 'KVarN requires block_size (64) == group (128)' assert — e.g. any hybrid-model deployment without spec decode (with MTP the hybrid block-size reconciliation happened to land on 128, which is why issue vllm-project#10 reporters never hit it; explicit --block-size 128 also avoids it). Override get_preferred_block_size on the KVarN backend to return the active preset's group. Validated: Qwen3.6-27B TP2 without spec decode now starts ('Setting kv cache block size to 128 for KVARN backend') and serves; with-MTP behaviour unchanged.

…ffers Follow-up to the issue vllm-project#10 capacity fixes — recovers another ~1.6 GiB of KV budget by making the startup memory accounting honest: 1. Materialize ALL of KVarN's lazy state (per-layer fp16 tail pools, shared decode scratch, kernel JIT + autotune) inside the worker's memory-profiling window. The dummy profile run skips attention (attn_metadata=None), so this state previously first ran inside the CUDA-graph memory estimation warmup and was charged to the 'graph memory' estimate (1.7 GiB mislabeled; the real graph pool is ~54 MiB), with the pools additionally double-counted against the explicit worker reservation. The arithmetic pool reservation is dropped — the allocation is now real and measured. Accounting stays correct with VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. 2. Size the split-K partial buffers (mid_o / mid_lse) by what the split-K decode path can actually index — N = max_num_seqs x Hq rows — instead of the max_num_batched_tokens-driven q_rows (~85x oversized: 256 MiB -> ~3 MiB at typical configs). The decode driver additionally falls back to the single-stage kernel if a (padded dummy) batch ever exceeds the buffer rows. Qwen3.6-27B AWQ TP2 (issue vllm-project#10 configs), per rank: - CUDA graph estimate 1.76 -> 0.08 GiB (= fp8's), available KV 5.2 -> 6.77 GiB, capacity 540,503 -> 699,951 tokens (1.87x fp8's 373,445; the original report was ~0.1x). - without spec decode: 774,516 tokens, decode 70-72 tok/s coherent (exercises the resized split-K path); with MTP: 49 tok/s coherent, 8-way concurrent 0 errors. - Qwen3-4B kvarn-vs-fp16 greedy regression: 100% token-identical.

…project#10 loops) Prefix caching shares/recycles blocks across requests; the dense builder kept per-request flush state and discarded finished requests' pool data, so cache hits read stale tiles and an empty first block — surfacing as repetition loops. Reworked to a per-block, sharing-safe lifecycle (same design as the MLA fix 2e8f5cad7): reclaim flushes complete blocks, sinks stay fp16-resident and are evicted lazily. Cache-hit replay now byte-identical on the reporter's Qwen3.6-27B-AWQ setup, with and without MTP; repro + stress scripts in scripts_kvarn_dense/.

…lm-project#10) Fixes the two regressions reported after the MTP fix (d7bbff123): decode collapsing to <5 tok/s, and KV-cache capacity ~10x below fp8. 1. _cached_multiquery_path: a spec-decode verify step (or chunked-prefill continuation) now builds the batch's rotated fp16 K/V with the ONE block_table-driven Triton kernel and runs a single flash_attn_varlen call. FA's varlen causal mask is bottom-right aligned for seqlen_q < seqlen_k — token t attends keys <= cached_len + t — exactly the verify semantics, so no explicit mask is needed. Replaces _decode_path_slow on this route (per-request Python gather with per-block .item() syncs + Python dequant + fp32 SDPA, per layer, per step). The slow path remains the fallback for head_dim > 256 or a batch whose total KV exceeds the materialize scratch. 2. _warm_decode_kernels: compile + autotune the fused decode kernel, split-K stage1/2, and the packed-KV build kernel at profile time on tiny synthetic state. They never ran during vLLM's prefill-shaped profiling, so their one-time JIT/autotune cost (and, before fix 1, the slow path's fp32 materializations) landed in the CUDA-graph memory estimation warmup and was charged as graph memory — GiBs off the derived KV capacity. Qwen3.6-27B AWQ TP2 + MTP(3) + prefix caching + chunked prefill (the issue vllm-project#10 configs): - KV capacity 92,800 -> 540,503 tokens (fp8 baseline: 373,445; KVarN is now 1.45x fp8 instead of 0.25x); --max-model-len -1 fits the full 262K context again. - decode 43-47 tok/s single stream, 205 tok/s at 8 concurrent, coherent output, 0 errors (was <5 tok/s). - Qwen3-4B no-spec regression: kvarn-vs-fp16 greedy decode 100% token-identical.

Under MTP every decode step is a verify step, and the previous route materialized the request's ENTIRE context into fp16 scratch (then ran flash attention over it) on every step, per layer — an O(context) per-step cost that dominated long-context decode (issue vllm-project#10 follow-up: 5-12 tok/s on long agent sessions). Both fused dual-source decode kernels (single-stage and split-K stage 1) gain a VQ_INDIRECT mode: one program per (verify token, KV head), with a block-table row indirection and a per-token bottom-right causal length, so the verify step reads int4 tiles and the fp16 tail pool directly — no materialization. New driver kvarn_verify_attention; the backend routes by context depth (KVARN_FUSED_VERIFY_MIN_BLOCKS, default 64 blocks): the materialize route stays for short contexts, where its single round-trip is cheap and measured faster, and for chunked-prefill continuations, where one materialization amortizes over thousands of query tokens. KVARN_FUSED_VERIFY=0 forces the previous behavior. The kernel warmup covers the new specializations so the first verify step does not JIT. Measured (Qwen3.6-27B AWQ, single GPU, single stream, greedy, MTP k=3): decode 45.2 -> 51.6 tok/s at 32K context (+14%), parity at shorter contexts; the advantage grows with depth since the kernel reads ~4x fewer bytes than the fp16 round-trip. MTP long generation stays coherent, and the non-speculative path is byte-identical (Qwen3-4B greedy battery) since existing launch sites pass VQ_INDIRECT=False.

zhuohan123 added 6 commits March 26, 2023 12:41

modify current frontend & server to prepare for real frontend

521dcfa

add server_utils

3e8c132

fix small bugs

8873415

add a new server class

71c2b93

initial implementation of fastapi frontend

d262ac9

fix memory bugs and add test client

ea06325

zhuohan123 requested a review from WoosukKwon March 27, 2023 06:19

zhuohan123 added 7 commits March 27, 2023 09:09

Rename and small fixes

03f09f0

Merge branch 'main' into real-frontend

16e9674

Rename and update readme

f045128

fix

6f735fa

fix api and add gradio webserver

f1666e8

Modify readme

9b2972f

Update README

9d23237

WoosukKwon approved these changes Mar 28, 2023

View reviewed changes

Comment thread cacheflow/sampling_params.py Outdated

Comment thread cacheflow/master/server.py Outdated

Address review comments.

739f599

zhuohan123 merged commit 721fa3d into main Mar 29, 2023

zhuohan123 deleted the real-frontend branch March 29, 2023 06:49

TheBloke mentioned this pull request Jul 20, 2023

Can't launch OpenAI API server on newly installed vLLM in Docker - fastchat not found #537

Closed

xiangyuT pushed a commit to xiangyuT/vllm that referenced this pull request Oct 25, 2023

Clean unused KVCache after usage (vllm-project#10)

fbcfee9

* Add underlying functions * tests done

shanshanpt mentioned this pull request Nov 17, 2023

Run long conetxt error : CUDA error: an illegal memory access was encountered #1700

Closed

junior-zsy mentioned this pull request Nov 20, 2023

Error with 32k Long Text in chatglm2-6b-32k Model #1725

Closed

hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024

FastAPI-based working frontend (vllm-project#10)

e1f8fcc

slyalin pushed a commit to slyalin/vllm that referenced this pull request Mar 22, 2024

Merge pull request vllm-project#10 from slyalin/trust_remote_code_pas…

a398824

…sthrough Passthrough trust_remote_code

yuhuixu1993 mentioned this pull request Jun 2, 2024

[Bug]: loading squeezellm model #5190

Closed

ykim362 pushed a commit to ykim362/vllm that referenced this pull request Jun 17, 2024

Merge pull request vllm-project#10 from wenxcs/wenxh/fp8-on-a100-v1-pr

94d4614

Wenxh/fp8 on a100 v1 pr

alixiaodi mentioned this pull request Aug 2, 2024

[Bug]: #7072

Closed

danisereb pushed a commit to de-inf/vllm that referenced this pull request Apr 5, 2026

Merge pull request vllm-project#10 from TomerBN-Nvidia/fix/multinode-…

c6eb917

…dp-tcp-placement Port multi-node DP fixes from upstream PR vllm-project#38630

myia-ai-01 mentioned this pull request Apr 19, 2026

[Bug]: V1 engine workers die after idle period (SystemError: PyCFunction / EngineDeadError) — TP=2, multiprocessing #35104

Open

2 tasks

yuyue0225sc mentioned this pull request Jun 23, 2026

[Bugfix][Structured Output][Spec Decode] Constrain bitmask and trim grammar advance at the reasoning boundary #44297

Merged

lcheng321 mentioned this pull request Jul 1, 2026

[Bugfix] Preserve FP8 indexer WK pairs across incremental load_weights #46168

Merged

aoshen02 referenced this pull request in aoshen02/vllm Jul 31, 2026

Merge pull request #10 from aoshen02/codex/pr45635-r3-simplify

3b1d20a

[R3] Simplify routed-experts sidecar plumbing

Navigation Menu

Uh oh!

Uh oh!

FastAPI-based working frontend - #10

FastAPI-based working frontend#10
zhuohan123 merged 14 commits into
mainfrom
real-frontend

zhuohan123 commented Mar 27, 2023 •

edited

Loading

Uh oh!

WoosukKwon left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

zhuohan123 commented Mar 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WoosukKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhuohan123 commented Mar 27, 2023 •

edited

Loading