| 1 | # Dynamic Speculative Decoding |
| 2 | |
| 3 | ## Why is Dynamic SD needed? |
| 4 | |
| 5 | SD methods need to verify K tokens for each sequence during decoding. As BS increases, the effective BS becomes BS\*K which increases the compute requirement during verification. When this BS\*K goes beyond a critical BS then SD negatively impacts the decode speed (TPOT). DSD helps by tuning the K to an optimal value such that we continue to reap the benefits from SD. |
| 6 | |
| 7 | ## Use cases |
| 8 | |
| 9 | * Variable concurrency workload using same deployment. K would decrease as concurrency increases. |
| 10 | * During RL rollout where we start off with high BS but then end up with small BS due to very few long tail request which end up generating a lot of tokens stalling the progress of the current rollout. Here K would go up during the end of rollout. |
| 11 | |
| 12 | ## `--speculative-config` schema |
| 13 | |
| 14 | To use Dynamic SD, add `num_speculative_tokens_per_batch_size` to the config of an SD method which is a list of list. Here, an entry is `[start_bs, end_bs, optimal_K]` which means when the concurrency is within range `[start_bs, end_bs]` then `optimal_K` number of draft tokens are used. For e.g., |
| 15 | |
| 16 | ```bash |
| 17 | --speculative-config '{ |
| 18 | "method": "eagle", |
| 19 | "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B", |
| 20 | "num_speculative_tokens": 3, |
| 21 | "num_speculative_tokens_per_batch_size": [ |
| 22 | [1, 64, 3], |
| 23 | [65, 128, 1], |
| 24 | [129, 512, 0] |
| 25 | ] |
| 26 | }' |
| 27 | ``` |
| 28 | |
| 29 | implies that: |
| 30 | |
| 31 | * K=3 will be used when the concurrency is in range [1, 64] |
| 32 | * K=1 will be used when the concurrency is in range [65, 128] |
| 33 | * K=0 will be used when the concurrency is in range [129, 512], i.e., no draft tokens will be produced. |
| 34 | |
| 35 | ## Online Examples |
| 36 | |
| 37 | ### Dynamic SD Eagle Drafter |
| 38 | |
| 39 | ```bash |
| 40 | VLLM_USE_V2_MODEL_RUNNER=0 vllm serve meta-llama/Llama-3.1-8B-Instruct \ |
| 41 | --speculative-config '{ |
| 42 | "method": "eagle", |
| 43 | "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B", |
| 44 | "num_speculative_tokens": 3, |
| 45 | "num_speculative_tokens_per_batch_size": [ |
| 46 | [1, 64, 3], |
| 47 | [65, 128, 1], |
| 48 | [129, 512, 0] |
| 49 | ] |
| 50 | }' |
| 51 | ``` |
| 52 | |
| 53 | ### Dynamic SD Eagle3 Drafter |
| 54 | |
| 55 | ```bash |
| 56 | VLLM_USE_V2_MODEL_RUNNER=0 vllm serve meta-llama/Llama-3.1-8B-Instruct \ |
| 57 | --speculative-config '{ |
| 58 | "method": "eagle3", |
| 59 | "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B", |
| 60 | "num_speculative_tokens": 3, |
| 61 | "num_speculative_tokens_per_batch_size": [ |
| 62 | [1, 16, 5], |
| 63 | [17, 32, 4], |
| 64 | [33, 64, 3], |
| 65 | [65, 128, 1], |
| 66 | [129, 512, 0] |
| 67 | ] |
| 68 | }' |
| 69 | |
| 70 | ``` |
| 71 | |
| 72 | ## Limitations |
| 73 | |
| 74 | * Tested with Eagle, Eagle-3, and DFlash. Other SD methods may or may not work out of the box |
| 75 | * Full Cudagraph only works with Model Runner V2. MRv1 only supports piece-wise cuda graph with this feature |
| 76 | * Not compatible with data parallelism (`--data-parallel-size > 1`). Each DP rank schedules independently, so ranks can pick different K values, causing DP collective divergence and deadlocks. When DP is enabled, vLLM automatically disables `num_speculative_tokens_per_batch_size` and falls back to the static `num_speculative_tokens` value. |