A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 08:15 UTC
[Doc] Update VLM doc about loading from local files (#9999)
Signed-off-by: Roger Wang <ywang@roblox.com>
Roger Wang committed
Nov 4, 2024 at 11:47 UTC
6e056bcf0414dfaee4db646f8f36ec961f0c9a33
1 file changed
+4
docs/source/models/vlm.rst
+4
@@ -242,6 +242,10 @@ To consume the server, you can use the OpenAI client like in the example below:
242
243
A full code example can be found in `examples/openai_chat_completion_client_for_multimodal.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client_for_multimodal.py>`_.
244
245
+.. tip::
246
+ Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via ``--allowed-local-media-path`` when launching the API server/engine,
247
+ and pass the file path as ``url`` in the API request.
248
+
249
.. tip::
250
There is no need to place image placeholders in the text content of the API request - they are already represented by the image content.
251
In fact, you can place image placeholders in the middle of the text by interleaving text and image content.