@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 398521ad19
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 18:15 UTC

[OpenVINO] Updated documentation (#7687)

Ilya Lavrenov committed Aug 20, 2024 at 17:33 UTC 398521ad199d0ca8f822a98a004fef0e5914753a
1 file changed +1 -3
docs/source/getting_started/openvino-installation.rst
+1 -3
@@ -70,7 +70,7 @@ vLLM OpenVINO backend uses the following environment variables to control behavi
70
71 - ``VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8`` to control KV cache precision. By default, FP16 / BF16 is used depending on platform.
72
73 -- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off.
73 +- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off. You can also export model with different compression techniques using `optimum-cli` and pass exported folder as `<model_id>`
74
75 To enable better TPOT / TTFT latency, you can use vLLM's chunked prefill feature (``--enable-chunked-prefill``). Based on the experiments, the recommended batch size is ``256`` (``--max-num-batched-tokens``)
76
@@ -91,5 +91,3 @@ Limitations
91 - Only LLM models are currently supported. LLaVa and encoder-decoder models are not currently enabled in vLLM OpenVINO integration.
92
93 - Tensor and pipeline parallelism are not currently enabled in vLLM integration.
94 -
95 -- Speculative sampling is not tested within vLLM integration.