A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 18:15 UTC
[OpenVINO] Updated documentation (#7687)
Ilya Lavrenov committed
Aug 20, 2024 at 17:33 UTC
398521ad199d0ca8f822a98a004fef0e5914753a
1 file changed
+1
-3
docs/source/getting_started/openvino-installation.rst
+1
-3
@@ -70,7 +70,7 @@ vLLM OpenVINO backend uses the following environment variables to control behavi
70
71
- ``VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8`` to control KV cache precision. By default, FP16 / BF16 is used depending on platform.
72
73
-- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off.
73
+- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off. You can also export model with different compression techniques using `optimum-cli` and pass exported folder as `<model_id>`
74
75
To enable better TPOT / TTFT latency, you can use vLLM's chunked prefill feature (``--enable-chunked-prefill``). Based on the experiments, the recommended batch size is ``256`` (``--max-num-batched-tokens``)
76
@@ -91,5 +91,3 @@ Limitations
91
- Only LLM models are currently supported. LLaVa and encoder-decoder models are not currently enabled in vLLM OpenVINO integration.
92
93
- Tensor and pipeline parallelism are not currently enabled in vLLM integration.
94
-
95
-- Speculative sampling is not tested within vLLM integration.