A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 10:45 UTC
[Doc] Fix code formatting in spec_decode.rst (#9348)
Michael Goin committed
Oct 15, 2024 at 00:29 UTC
8e836d982ab19afbfce2bc28074a64fa7dca104c
1 file changed
+3
-3
docs/source/models/spec_decode.rst
+3
-3
@@ -44,10 +44,10 @@ To perform the same with an online mode launch the server:
44
.. code-block:: bash
45
46
python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8000 --model facebook/opt-6.7b \
47
- --seed 42 -tp 1 --speculative_model facebook/opt-125m --use-v2-block-manager \
48
- --num_speculative_tokens 5 --gpu_memory_utilization 0.8
47
+ --seed 42 -tp 1 --speculative_model facebook/opt-125m --use-v2-block-manager \
48
+ --num_speculative_tokens 5 --gpu_memory_utilization 0.8
49
50
- Then use a client:
50
+Then use a client:
51
52
.. code-block:: python
53