| 1 | # vLLM-Project/Speculators |
| 2 | |
| 3 |  |
| 4 |  |
| 5 | |
| 6 | [Speculators](https://docs.vllm.ai/projects/speculators/en/latest/) is a library for accelerating LLM inference through speculative decoding, providing efficient draft model training that integrates seamlessly with vLLM to reduce latency and improve throughput. |
| 7 | |
| 8 | Speculators provides the following key features: |
| 9 | |
| 10 | - **Offline training data generation using vLLM**: Enable the generation of hidden states using vLLM. Data samples are saved to disk and can be used for draft model training. |
| 11 | - **Draft model training support**: E2E training support of single and multi-layer draft models. Training is supported for both non-MoE and MoE models. |
| 12 | - **Standardized, extensible format**: Provides a Hugging Face-compatible format for defining speculative models, with tools to convert from external research repositories into a standard speculators format for easy adoption. |
| 13 | - **Seamless vLLM Integration**: Built for direct deployment into vLLM, enabling low-latency, production-grade inference with minimal overhead. |
| 14 | |
| 15 | ## Why use Speculators? |
| 16 | |
| 17 | Large language models generate text one token at a time, which creates a fundamental bottleneck: each token requires a full forward pass through the model, leaving GPU compute underutilized while waiting for memory-bound operations. |
| 18 | Speculative decoding addresses this by using a smaller, faster "draft" model (often times, just a single transformer layer) to predict multiple tokens ahead, and then verifying tokens in parallel with the primary model. |
| 19 | |
| 20 | Speculative decoding provides the following benefits: |
| 21 | |
| 22 | - **Reduced latency**: Generates tokens 2-3 times faster for interactive applications such as chatbots and code assistants, where response time directly impacts user experience |
| 23 | - **Better GPU utilization**: Converts latency and memory-bound decoding in the large model into compute-bound parallel token verification, improving hardware utilization. |
| 24 | - **No quality loss**: Speculative decoding does not approximate the target model. Accepted tokens are exactly those the target model would have produced under the same sampling configuration; rejected draft tokens are discarded and regenerated by the target model. |
| 25 | - **Cost efficiency**: Serve more requests per GPU by reducing the time each request occupies the hardware |
| 26 | |
| 27 | Speculators is particularly valuable for latency-sensitive applications where users are waiting for responses in real-time, such as conversational AI, interactive coding assistants, and streaming text generation. |
| 28 | |
| 29 | ## Resources |
| 30 | |
| 31 | - [Speculators examples](https://github.com/vllm-project/speculators/tree/main/examples) |
| 32 | - [GitHub Repository](https://github.com/vllm-project/speculators) |