A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 11:00 UTC
[Chore][Doc] uses model id determined from OpenAI client (#17815)
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Aaron Pham committed
May 7, 2025 at 21:48 UTC
a8238bbdb086d2e25a6c1a16b3438e0ffeb0de89
3 files changed
+3
-3
examples/online_serving/openai_chat_completion_structured_outputs.py
+1
-1
@@ -138,7 +138,7 @@ def main():
138
api_key="-",
139
)
140
141
- model = "Qwen/Qwen2.5-3B-Instruct"
141
+ model = client.models.list().data[0].id
142
143
print("Guided Choice Completion:")
144
print(guided_choice_completion(client, model))
examples/online_serving/openai_chat_completion_structured_outputs_structural_tag.py
+1
-1
@@ -59,7 +59,7 @@ and San Francisco?
59
}]
60
61
response = client.chat.completions.create(
62
- model="meta-llama/Llama-3.1-8B-Instruct",
62
+ model=client.models.list().data[0].id,
63
messages=messages,
64
response_format={
65
"type":
examples/online_serving/openai_chat_completion_structured_outputs_with_reasoning.py
+1
-1
@@ -4,7 +4,7 @@ An example shows how to generate structured outputs from reasoning models
4
like DeepSeekR1. The thinking process will not be guided by the JSON
5
schema provided by the user. Only the final output will be structured.
6
7
-To run this example, you need to start the vLLM server with the reasoning
7
+To run this example, you need to start the vLLM server with the reasoning
8
parser:
9
10
```bash