@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 3ec8c25cd0
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 05:30 UTC

[Docs] Update documentation for gpu-memory-utilization option (#2162)

Suhong Moon committed Dec 17, 2023 at 13:51 UTC 3ec8c25cd07c4a3d747b846ece8e305a7fb44349
1 file changed +4 -2
docs/source/models/engine_args.rst
+4 -2
@@ -89,9 +89,11 @@ Below, you can find an explanation of every engine argument for vLLM:
89
90 CPU swap space size (GiB) per GPU.
91
92 -.. option:: --gpu-memory-utilization <percentage>
92 +.. option:: --gpu-memory-utilization <fraction>
93
94 - The percentage of GPU memory to be used for the model executor.
94 + The fraction of GPU memory to be used for the model executor, which can range from 0 to 1.
95 + For example, a value of 0.5 would imply 50% GPU memory utilization.
96 + If unspecified, will use the default value of 0.9.
97
98 .. option:: --max-num-batched-tokens <tokens>
99