A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
[Docs] Update documentation for gpu-memory-utilization option (#2162)
Suhong Moon committed
Dec 17, 2023 at 13:51 UTC
3ec8c25cd07c4a3d747b846ece8e305a7fb44349
1 file changed
+4
-2
docs/source/models/engine_args.rst
+4
-2
@@ -89,9 +89,11 @@ Below, you can find an explanation of every engine argument for vLLM:
89
90
CPU swap space size (GiB) per GPU.
91
92
-.. option:: --gpu-memory-utilization <percentage>
92
+.. option:: --gpu-memory-utilization <fraction>
93
94
- The percentage of GPU memory to be used for the model executor.
94
+ The fraction of GPU memory to be used for the model executor, which can range from 0 to 1.
95
+ For example, a value of 0.5 would imply 50% GPU memory utilization.
96
+ If unspecified, will use the default value of 0.9.
97
98
.. option:: --max-num-batched-tokens <tokens>
99