A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 09:45 UTC
[doc] update env variable export (#18391)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
May 20, 2025 at 16:53 UTC
1b1e8e05ff3c26b98e4161bd3c8671e86fb145f4
1 file changed
+5
docs/source/getting_started/quickstart.md
+5
@@ -82,6 +82,11 @@ llm = LLM(model="facebook/opt-125m")
82
83
:::{note}
84
By default, vLLM downloads models from [Hugging Face](https://huggingface.co/). If you would like to use models from [ModelScope](https://www.modelscope.cn), set the environment variable `VLLM_USE_MODELSCOPE` before initializing the engine.
85
+
86
+```shell
87
+export VLLM_USE_MODELSCOPE=True
88
+```
89
+
90
:::
91
92
Now, the fun part! The outputs are generated using `llm.generate`. It adds the input prompts to the vLLM engine's waiting queue and executes the vLLM engine to generate the outputs with high throughput. The outputs are returned as a list of `RequestOutput` objects, which include all of the output tokens.