@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 78107fa091
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 03:00 UTC

[Doc]Add asynchronous engine arguments to documentation. (#3810)

Co-authored-by: Simon Mo <simon.mo@hey.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

Sean Gallen committed Apr 4, 2024 at 23:52 UTC 78107fa0911567f131cbad810872ae25594a4506
1 file changed +16
docs/source/models/engine_args.rst
+16
@@ -118,3 +118,19 @@ Below, you can find an explanation of every engine argument for vLLM:
118 .. option:: --quantization (-q) {awq,squeezellm,None}
119
120 Method used to quantize the weights.
121 +
122 +Async Engine Arguments
123 +----------------------
124 +Below are the additional arguments related to the asynchronous engine:
125 +
126 +.. option:: --engine-use-ray
127 +
128 + Use Ray to start the LLM engine in a separate process as the server process.
129 +
130 +.. option:: --disable-log-requests
131 +
132 + Disable logging requests.
133 +
134 +.. option:: --max-log-len
135 +
136 + Max number of prompt characters or prompt ID numbers being printed in log. Defaults to unlimited.
\ No newline at end of file