A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 11:00 UTC
[Misc] remove --model from vllm serve usage (#17944)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
May 10, 2025 at 21:23 UTC
4c31218f80e35c4d94097a792a15b7817381daf0
1 file changed
+2
-2
examples/online_serving/openai_chat_completion_client_with_tools.py
+2
-2
@@ -7,12 +7,12 @@ IMPORTANT: for mistral, you must use one of the provided mistral tool call
7
templates, or your own - the model default doesn't work for tool calls with vLLM
8
See the vLLM docs on OpenAI server & tool calling for more details.
9
10
-vllm serve --model mistralai/Mistral-7B-Instruct-v0.3 \
10
+vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
11
--chat-template examples/tool_chat_template_mistral.jinja \
12
--enable-auto-tool-choice --tool-call-parser mistral
13
14
OR
15
-vllm serve --model NousResearch/Hermes-2-Pro-Llama-3-8B \
15
+vllm serve NousResearch/Hermes-2-Pro-Llama-3-8B \
16
--chat-template examples/tool_chat_template_hermes.jinja \
17
--enable-auto-tool-choice --tool-call-parser hermes
18
"""