@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 8e836d982a
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 10:45 UTC

[Doc] Fix code formatting in spec_decode.rst (#9348)

Michael Goin committed Oct 15, 2024 at 00:29 UTC 8e836d982ab19afbfce2bc28074a64fa7dca104c
1 file changed +3 -3
docs/source/models/spec_decode.rst
+3 -3
@@ -44,10 +44,10 @@ To perform the same with an online mode launch the server:
44 .. code-block:: bash
45
46 python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8000 --model facebook/opt-6.7b \
47 - --seed 42 -tp 1 --speculative_model facebook/opt-125m --use-v2-block-manager \
48 - --num_speculative_tokens 5 --gpu_memory_utilization 0.8
47 + --seed 42 -tp 1 --speculative_model facebook/opt-125m --use-v2-block-manager \
48 + --num_speculative_tokens 5 --gpu_memory_utilization 0.8
49
50 - Then use a client:
50 +Then use a client:
51
52 .. code-block:: python
53