A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
Add chat doc in quick start (#21213)
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
TankNee committed
Aug 3, 2025 at 22:47 UTC
83f7bbb3180fe7503cfbc4fb49b06200fb64cdf0
1 file changed
+37
docs/getting_started/quickstart.md
+37
@@ -98,6 +98,43 @@ for output in outputs:
98
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
99
```
100
101
+!!! note
102
+ The `llm.generate` method does not automatically apply the model's chat template to the input prompt. Therefore, if you are using an Instruct model or Chat model, you should manually apply the corresponding chat template to ensure the expected behavior. Alternatively, you can use the `llm.chat` method and pass a list of messages which have the same format as those passed to OpenAI's `client.chat.completions`:
103
+
104
+ ??? code
105
+
106
+ ```python
107
+ # Using tokenizer to apply chat template
108
+ from transformers import AutoTokenizer
109
+
110
+ tokenizer = AutoTokenizer.from_pretrained("/path/to/chat_model")
111
+ messages_list = [
112
+ [{"role": "user", "content": prompt}]
113
+ for prompt in prompts
114
+ ]
115
+ texts = tokenizer.apply_chat_template(
116
+ messages_list,
117
+ tokenize=False,
118
+ add_generation_prompt=True,
119
+ )
120
+
121
+ # Generate outputs
122
+ outputs = llm.generate(texts, sampling_params)
123
+
124
+ # Print the outputs.
125
+ for output in outputs:
126
+ prompt = output.prompt
127
+ generated_text = output.outputs[0].text
128
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
129
+
130
+ # Using chat interface.
131
+ outputs = llm.chat(messages_list, sampling_params)
132
+ for idx, output in enumerate(outputs):
133
+ prompt = prompts[idx]
134
+ generated_text = output.outputs[0].text
135
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
136
+ ```
137
+
138
[](){ #quickstart-online }
139
140
## OpenAI-Compatible Server