@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 133765760b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 9, 07:30 UTC

[Docs] Adding links and intro to Speculators and LLM Compressor (#32849)

Signed-off-by: Aidan Reilly <aireilly@redhat.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Aidan Reilly committed Jan 29, 2026 at 22:12 UTC 133765760b21fde228bf1ca19fa4b35b5c206359
5 files changed +73 -7
docs/features/README.md
+3 -3
@@ -36,12 +36,12 @@ th:not(:first-child) {
36 }
37 </style>
38
39 -| Feature | [CP](../configuration/optimization.md#chunked-prefill) | [APC](automatic_prefix_caching.md) | [LoRA](lora.md) | [SD](spec_decode.md) | CUDA graph | [pooling](../models/pooling_models.md) | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | <abbr title="Logprobs">logP</abbr> | <abbr title="Prompt Logprobs">prmpt logP</abbr> | <abbr title="Async Output Processing">async output</abbr> | multi-step | <abbr title="Multimodal Inputs">mm</abbr> | best-of | beam-search | [prompt-embeds](prompt_embeds.md) |
39 +| Feature | [CP](../configuration/optimization.md#chunked-prefill) | [APC](automatic_prefix_caching.md) | [LoRA](lora.md) | [SD](spec_decode/README.md) | CUDA graph | [pooling](../models/pooling_models.md) | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | <abbr title="Logprobs">logP</abbr> | <abbr title="Prompt Logprobs">prmpt logP</abbr> | <abbr title="Async Output Processing">async output</abbr> | multi-step | <abbr title="Multimodal Inputs">mm</abbr> | best-of | beam-search | [prompt-embeds](prompt_embeds.md) |
40 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
41 | [CP](../configuration/optimization.md#chunked-prefill) | ✅ | | | | | | | | | | | | | | |
42 | [APC](automatic_prefix_caching.md) | ✅ | ✅ | | | | | | | | | | | | | |
43 | [LoRA](lora.md) | ✅ | ✅ | ✅ | | | | | | | | | | | | |
44 -| [SD](spec_decode.md) | ✅ | ✅ | ❌ | ✅ | | | | | | | | | | | |
44 +| [SD](spec_decode/README.md) | ✅ | ✅ | ❌ | ✅ | | | | | | | | | | | |
45 | CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | |
46 | [pooling](../models/pooling_models.md) | 🟠\* | 🟠\* | ✅ | ❌ | ✅ | ✅ | | | | | | | | | |
47 | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ❌ | [❌](https://github.com/vllm-project/vllm/issues/7366) | ❌ | [❌](https://github.com/vllm-project/vllm/issues/7366) | ✅ | ✅ | ✅ | | | | | | | | |
@@ -64,7 +64,7 @@ th:not(:first-child) {
64 | [CP](../configuration/optimization.md#chunked-prefill) | [❌](https://github.com/vllm-project/vllm/issues/2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
65 | [APC](automatic_prefix_caching.md) | [❌](https://github.com/vllm-project/vllm/issues/3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
66 | [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
67 -| [SD](spec_decode.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
67 +| [SD](spec_decode/README.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
68 | CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/26970) |
69 | [pooling](../models/pooling_models.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
70 | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
docs/features/quantization/README.md
+4 -1
@@ -2,7 +2,10 @@
2
3 Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.
4
5 -Contents:
5 +!!! tip
6 + To get started with quantization, see [LLM Compressor](llm_compressor.md), a library for optimizing models for deployment with vLLM that supports FP8, INT8, INT4, and other quantization formats.
7 +
8 +The following are the supported quantization formats for vLLM:
9
10 - [AutoAWQ](auto_awq.md)
11 - [BitsAndBytes](bnb.md)
docs/features/quantization/llm_compressor.md new
+31
@@ -0,0 +1,31 @@
1 +# LLM Compressor
2 +
3 +[LLM Compressor](https://docs.vllm.ai/projects/llm-compressor/en/latest/) is a library for optimizing models for deployment with vLLM.
4 +It provides a comprehensive set of quantization algorithms, including support for techniques such as FP4, FP8, INT8, and INT4 quantization.
5 +
6 +## Why use LLM Compressor?
7 +
8 +Modern LLMs often contain billions of parameters stored in 16-bit or 32-bit floating point, requiring substantial GPU memory and limiting deployment options.
9 +Quantization lowers memory requirements while maintaining inference output quality by reducing the precision of model weights and activations to smaller data types.
10 +
11 +LLM Compressor provides the following benefits:
12 +
13 +- **Reduced memory footprint**: Run larger models on smaller GPUs.
14 +- **Lower inference costs**: Serve more concurrent users per GPU, directly reducing the cost per query in production deployments.
15 +- **Faster inference**: Smaller data types mean less memory bandwidth consumed, which often translates to higher throughput, especially for memory-bound workloads.
16 +
17 +LLM Compressor handles the complexity of quantization, calibration, and format conversion, producing models ready for immediate use with vLLM.
18 +
19 +## Key features
20 +
21 +- **Multiple Quantization Algorithms**: Support for AWQ, GPTQ, AutoRound, and Round-to-Nearest.
22 +Also includes support for QuIP and SpinQuant-style transforms as well as KV cache and attention quantization.
23 +- **Multiple Quantization Methods**: Support for FP8, INT8, INT4, NVFP4, MXFP4, and mixed-precision quantization
24 +- **One-Shot Quantization**: Quantize models quickly with minimal calibration data
25 +- **vLLM Integration**: Seamlessly deploy quantized models with vLLM using the compressed-tensors format
26 +- **Hugging Face Compatibility**: Works with models from the Hugging Face Hub
27 +
28 +## Resources
29 +
30 +- [LLM Compressor examples](https://github.com/vllm-project/llm-compressor/tree/main/examples)
31 +- [GitHub Repository](https://github.com/vllm-project/llm-compressor)
docs/features/spec_decode/README.md renamed
+6 -3
@@ -11,6 +11,9 @@
11 This document shows how to use [Speculative Decoding](https://x.com/karpathy/status/1697318534555336961) with vLLM.
12 Speculative decoding is a technique which improves inter-token latency in memory-bound LLM inference.
13
14 +!!! tip
15 + To train your own draft models for speculative decoding, see [Speculators](speculators.md), a library for training draft models that integrates seamlessly with vLLM.
16 +
17 ## Speculating with a draft model
18
19 The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.
@@ -223,7 +226,7 @@ A variety of speculative models of this type are available on HF hub:
226 ## Speculating using EAGLE based draft models
227
228 The following code configures vLLM to use speculative decoding where proposals are generated by
226 -an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](../../examples/offline_inference/spec_decode.py).
229 +an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found in [examples/offline_inference/spec_decode.py](../../../examples/offline_inference/spec_decode.py)
230
231 ??? code
232
@@ -313,7 +316,7 @@ speculative decoding, breaking down the guarantees into three key areas:
316 3. **vLLM Logprob Stability**
317 \- vLLM does not currently guarantee stable token log probabilities (logprobs). This can result in different outputs for the
318 same request across runs. For more details, see the FAQ section
316 - titled *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](../usage/faq.md).
319 + titled *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](../../usage/faq.md).
320
321 While vLLM strives to ensure losslessness in speculative decoding, variations in generated outputs with and without speculative decoding
322 can occur due to following factors:
@@ -322,7 +325,7 @@ can occur due to following factors:
325 - **Batch Size and Numerical Stability**: Changes in batch size may cause variations in logprobs and output probabilities, potentially
326 due to non-deterministic behavior in batched operations or numerical instability.
327
325 -For mitigation strategies, please refer to the FAQ entry *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](../usage/faq.md).
328 +For mitigation strategies, please refer to the FAQ entry *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](../../usage/faq.md).
329
330 ## Resources for vLLM contributors
331
docs/features/spec_decode/speculators.md new
+29
@@ -0,0 +1,29 @@
1 +# Speculators
2 +
3 +[Speculators](https://docs.vllm.ai/projects/speculators/en/latest/) is a library for accelerating LLM inference through speculative decoding, providing efficient draft model training that integrates seamlessly with vLLM to reduce latency and improve throughput.
4 +
5 +Speculators provides the following key features:
6 +
7 +- **Offline training data generation using vLLM**: Enable the generation of hidden states using vLLM. Data samples are saved to disk and can be used for draft model training.
8 +- **Draft model training support**: E2E training support of single and multi-layer draft models. Training is supported for both non-MoE and MoE models.
9 +- **Standardized, extensible format**: Provides a Hugging Face-compatible format for defining speculative models, with tools to convert from external research repositories into a standard speculators format for easy adoption.
10 +- **Seamless vLLM Integration**: Built for direct deployment into vLLM, enabling low-latency, production-grade inference with minimal overhead.
11 +
12 +## Why use Speculators?
13 +
14 +Large language models generate text one token at a time, which creates a fundamental bottleneck: each token requires a full forward pass through the model, leaving GPU compute underutilized while waiting for memory-bound operations.
15 +Speculative decoding addresses this by using a smaller, faster "draft" model (often times, just a single transformer layer) to predict multiple tokens ahead, and then verifying tokens in parallel with the primary model.
16 +
17 +Speculative decoding provides the following benefits:
18 +
19 +- **Reduced latency**: Generates tokens 2-3 times faster for interactive applications such as chatbots and code assistants, where response time directly impacts user experience
20 +- **Better GPU utilization**: Converts latency and memory-bound decoding in the large model into compute-bound parallel token verification, improving hardware utilization.
21 +- **No quality loss**: Speculative decoding does not approximate the target model. Accepted tokens are exactly those the target model would have produced under the same sampling configuration; rejected draft tokens are discarded and regenerated by the target model.
22 +- **Cost efficiency**: Serve more requests per GPU by reducing the time each request occupies the hardware
23 +
24 +Speculators is particularly valuable for latency-sensitive applications where users are waiting for responses in real-time, such as conversational AI, interactive coding assistants, and streaming text generation.
25 +
26 +## Resources
27 +
28 +- [Speculators examples](https://github.com/vllm-project/speculators/tree/main/examples)
29 +- [GitHub Repository](https://github.com/vllm-project/speculators)