@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 118ff92111
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 07:15 UTC

[Doc] Update V1 user guide for embedding and enc-dec models (#19060)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Jun 3, 2025 at 17:29 UTC 118ff921118cc81061a2af865a1e13840ceb6792
1 file changed +10 -5
docs/usage/v1_guide.md
+10 -5
@@ -1,5 +1,7 @@
1 # vLLM V1
2
3 +**We have started the process of deprecating V0. Please read [RFC #18571](https://github.com/vllm-project/vllm/issues/18571) for more details.**
4 +
5 V1 is now enabled by default for all supported use cases, and we will gradually enable it for every use case we plan to support. Please share any feedback on [GitHub](https://github.com/vllm-project/vllm) or in the [vLLM Slack](https://inviter.co/vllm-slack).
6
7 To disable V1, please set the environment variable as: `VLLM_USE_V1=0`, and send us a GitHub issue sharing the reason!
@@ -51,9 +53,9 @@ This living user guide outlines a few known **important changes and limitations*
53 | **Spec Decode** | <nobr>🚧 WIP ([PR #13933](https://github.com/vllm-project/vllm/pull/13933))</nobr>|
54 | **Prompt Logprobs with Prefix Caching** | <nobr>🟡 Planned ([RFC #13414](https://github.com/vllm-project/vllm/issues/13414))</nobr>|
55 | **Structured Output Alternative Backends** | <nobr>🟡 Planned</nobr> |
54 -| **Embedding Models** | <nobr>🟡 Planned ([RFC #12249](https://github.com/vllm-project/vllm/issues/12249))</nobr> |
56 +| **Embedding Models** | <nobr>🚧 WIP ([PR #18015](https://github.com/vllm-project/vllm/pull/18015))</nobr> |
57 | **Mamba Models** | <nobr>🟡 Planned</nobr> |
56 -| **Encoder-Decoder Models** | <nobr>🟡 Planned</nobr> |
58 +| **Encoder-Decoder Models** | <nobr>🟠 Delayed</nobr> |
59 | **Request-level Structured Output Backend** | <nobr>🔴 Deprecated</nobr> |
60 | **best_of** | <nobr>🔴 Deprecated ([RFC #13361](https://github.com/vllm-project/vllm/issues/13361))</nobr>|
61 | **Per-Request Logits Processors** | <nobr>🔴 Deprecated ([RFC #13360](https://github.com/vllm-project/vllm/pull/13360))</nobr> |
@@ -63,10 +65,11 @@ This living user guide outlines a few known **important changes and limitations*
65 - **🟢 Functional**: Fully operational, with ongoing optimizations.
66 - **🚧 WIP**: Under active development.
67 - **🟡 Planned**: Scheduled for future implementation (some may have open PRs/RFCs).
66 -- **🔴 Deprecated**: Not planned for v1 unless there is strong demand.
68 +- **🟠 Delayed**: Temporarily dropped in V1 but planned to be re-introduced later.
69 +- **🔴 Deprecated**: Not planned for V1 unless there is strong demand.
70
71 **Note**: vLLM V1’s unified scheduler treats both prompt and output tokens the same
69 -way by using a simple dictionary (e.g., {request_id: num_tokens}) to dynamically
72 +way by using a simple dictionary (e.g., `{request_id: num_tokens}`) to dynamically
73 allocate a fixed token budget per request, enabling features like chunked prefills,
74 prefix caching, and speculative decoding without a strict separation between prefill
75 and decode phases.
@@ -140,7 +143,9 @@ vLLM V1 currently excludes model architectures with the `SupportsV0Only` protoco
143 and the majority fall into the following categories. V1 support for these models will be added eventually.
144
145 **Embedding Models**
143 -Instead of having a separate model runner, hidden states processor [RFC #12249](https://github.com/vllm-project/vllm/issues/12249), which is based on global logits processor [RFC #13360](https://github.com/vllm-project/vllm/pull/13360), has been proposed to enable simultaneous generation and embedding using the same engine instance in V1. It is still in the planning stage.
146 +Initially, we will create a [separate model runner](https://github.com/vllm-project/vllm/pull/18015) to provide V1 support without conflicting with other ongoing work.
147 +
148 +Later, we will consider using [hidden states processor](https://github.com/vllm-project/vllm/issues/12249), which is based on [global logits processor](https://github.com/vllm-project/vllm/pull/13360) to enable simultaneous generation and embedding using the same engine instance in V1. [PR #16188](https://github.com/vllm-project/vllm/pull/16188) is the first step towards enabling this.
149
150 **Mamba Models**
151 Models using selective state-space mechanisms (instead of standard transformer attention)