A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 08:30 UTC
[Doc] cleanup deprecated flag for doc (#18715)
Signed-off-by: calvin chen <120380290@qq.com>
Calvin Chen committed
May 27, 2025 at 15:12 UTC
4693a3438cebd6c2f764ad67af5a85f85d03db13
1 file changed
+2
-4
benchmarks/README.md
+2
-4
@@ -146,10 +146,9 @@ python3 vllm/benchmarks/benchmark_serving.py \
146
147
``` bash
148
VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
149
- --speculative-model "[ngram]" \
149
--ngram_prompt_lookup_min 2 \
150
--ngram-prompt-lookup-max 5 \
152
- --num_speculative_tokens 5
151
+ --speculative_config '{"model": "[ngram]", "num_speculative_tokens": 5}
152
```
153
154
``` bash
@@ -274,10 +273,9 @@ python3 vllm/benchmarks/benchmark_throughput.py \
273
--output-len=100 \
274
--num-prompts=2048 \
275
--async-engine \
277
- --speculative-model="[ngram]" \
276
--ngram_prompt_lookup_min=2 \
277
--ngram-prompt-lookup-max=5 \
280
- --num_speculative_tokens=5
278
+ --speculative_config '{"model": "[ngram]", "num_speculative_tokens": 5}
279
```
280
281
```