| 1 | # Parallel Draft Models |
| 2 | |
| 3 | The following code configures vLLM to use speculative decoding where proposals are generated by [PARD](https://arxiv.org/pdf/2504.18583) (Parallel Draft Models). |
| 4 | |
| 5 | ## PARD Offline Mode Example |
| 6 | |
| 7 | ```python |
| 8 | from vllm import LLM, SamplingParams |
| 9 | |
| 10 | prompts = ["The future of AI is"] |
| 11 | sampling_params = SamplingParams(temperature=0.8, top_p=0.95) |
| 12 | |
| 13 | llm = LLM( |
| 14 | model="Qwen/Qwen3-8B", |
| 15 | tensor_parallel_size=1, |
| 16 | speculative_config={ |
| 17 | "model": "amd/PARD-Qwen3-0.6B", |
| 18 | "num_speculative_tokens": 12, |
| 19 | "method": "draft_model", |
| 20 | "parallel_drafting": True, |
| 21 | }, |
| 22 | ) |
| 23 | outputs = llm.generate(prompts, sampling_params) |
| 24 | |
| 25 | for output in outputs: |
| 26 | prompt = output.prompt |
| 27 | generated_text = output.outputs[0].text |
| 28 | print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") |
| 29 | ``` |
| 30 | |
| 31 | ## PARD Online Mode Example |
| 32 | |
| 33 | ```bash |
| 34 | vllm serve Qwen/Qwen3-4B \ |
| 35 | --host 0.0.0.0 \ |
| 36 | --port 8000 \ |
| 37 | --seed 42 \ |
| 38 | -tp 1 \ |
| 39 | --max-model-len 2048 \ |
| 40 | --gpu-memory-utilization 0.8 \ |
| 41 | --speculative-config '{"model": "amd/PARD-Qwen3-0.6B", "num_speculative_tokens": 12, "method": "draft_model", "parallel_drafting": true}' |
| 42 | ``` |
| 43 | |
| 44 | ## Pre-trained PARD weights |
| 45 | |
| 46 | - [amd/pard](https://huggingface.co/collections/amd/pard) |