A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 03:00 UTC
[Doc]Add asynchronous engine arguments to documentation. (#3810)
Co-authored-by: Simon Mo <simon.mo@hey.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Sean Gallen committed
Apr 4, 2024 at 23:52 UTC
78107fa0911567f131cbad810872ae25594a4506
1 file changed
+16
docs/source/models/engine_args.rst
+16
@@ -118,3 +118,19 @@ Below, you can find an explanation of every engine argument for vLLM:
118
.. option:: --quantization (-q) {awq,squeezellm,None}
119
120
Method used to quantize the weights.
121
+
122
+Async Engine Arguments
123
+----------------------
124
+Below are the additional arguments related to the asynchronous engine:
125
+
126
+.. option:: --engine-use-ray
127
+
128
+ Use Ray to start the LLM engine in a separate process as the server process.
129
+
130
+.. option:: --disable-log-requests
131
+
132
+ Disable logging requests.
133
+
134
+.. option:: --max-log-len
135
+
136
+ Max number of prompt characters or prompt ID numbers being printed in log. Defaults to unlimited.
\ No newline at end of file