@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 6e056bcf04
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 08:15 UTC

[Doc] Update VLM doc about loading from local files (#9999)

Signed-off-by: Roger Wang <ywang@roblox.com>

Roger Wang committed Nov 4, 2024 at 11:47 UTC 6e056bcf0414dfaee4db646f8f36ec961f0c9a33
1 file changed +4
docs/source/models/vlm.rst
+4
@@ -242,6 +242,10 @@ To consume the server, you can use the OpenAI client like in the example below:
242
243 A full code example can be found in `examples/openai_chat_completion_client_for_multimodal.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client_for_multimodal.py>`_.
244
245 +.. tip::
246 + Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via ``--allowed-local-media-path`` when launching the API server/engine,
247 + and pass the file path as ``url`` in the API request.
248 +
249 .. tip::
250 There is no need to place image placeholders in the text content of the API request - they are already represented by the image content.
251 In fact, you can place image placeholders in the middle of the text by interleaving text and image content.