A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 17:15 UTC
[doc] add missing imports (#15699)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
Mar 28, 2025 at 23:56 UTC
2914006fe09875ebfa33626d945e34173c7441c6
5 files changed
+30
docs/source/models/generative_models.md
+6
@@ -23,6 +23,8 @@ It is similar to [its counterpart in HF Transformers](https://huggingface.co/doc
23
except that tokenization and detokenization are also performed automatically.
24
25
```python
26
+from vllm import LLM
27
+
28
llm = LLM(model="facebook/opt-125m")
29
outputs = llm.generate("Hello, my name is")
30
@@ -36,6 +38,8 @@ You can optionally control the language generation by passing {class}`~vllm.Samp
38
For example, you can use greedy sampling by setting `temperature=0`:
39
40
```python
41
+from vllm import LLM, SamplingParams
42
+
43
llm = LLM(model="facebook/opt-125m")
44
params = SamplingParams(temperature=0)
45
outputs = llm.generate("Hello, my name is", params)
@@ -83,6 +87,8 @@ Base models may perform poorly as they are not trained to respond to the chat co
87
:::
88
89
```python
90
+from vllm import LLM
91
+
92
llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
93
conversation = [
94
{
docs/source/models/pooling_models.md
+8
@@ -68,6 +68,8 @@ The {class}`~vllm.LLM.encode` method is available to all pooling models in vLLM.
68
It returns the extracted hidden states directly, which is useful for reward models.
69
70
```python
71
+from vllm import LLM
72
+
73
llm = LLM(model="Qwen/Qwen2.5-Math-RM-72B", task="reward")
74
(output,) = llm.encode("Hello, my name is")
75
@@ -81,6 +83,8 @@ The {class}`~vllm.LLM.embed` method outputs an embedding vector for each prompt.
83
It is primarily designed for embedding models.
84
85
```python
86
+from vllm import LLM
87
+
88
llm = LLM(model="intfloat/e5-mistral-7b-instruct", task="embed")
89
(output,) = llm.embed("Hello, my name is")
90
@@ -96,6 +100,8 @@ The {class}`~vllm.LLM.classify` method outputs a probability vector for each pro
100
It is primarily designed for classification models.
101
102
```python
103
+from vllm import LLM
104
+
105
llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", task="classify")
106
(output,) = llm.classify("Hello, my name is")
107
@@ -116,6 +122,8 @@ To handle RAG at a higher level, you should use integration frameworks such as [
122
:::
123
124
```python
125
+from vllm import LLM
126
+
127
llm = LLM(model="BAAI/bge-reranker-v2-m3", task="score")
128
(output,) = llm.score("What is the capital of France?",
129
"The capital of Brazil is Brasilia.")
docs/source/performance/optimization.md
+2
@@ -31,6 +31,8 @@ vLLM supports an experimental feature chunked prefill. Chunked prefill allows to
31
You can enable the feature by specifying `--enable-chunked-prefill` in the command line or setting `enable_chunked_prefill=True` in the LLM constructor.
32
33
```python
34
+from vllm import LLM
35
+
36
llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_chunked_prefill=True)
37
# Set max_num_batched_tokens to tune performance.
38
# NOTE: 2048 is the default max_num_batched_tokens for chunked prefill.
docs/source/serving/multimodal_inputs.md
+8
@@ -21,6 +21,8 @@ To input multi-modal data, follow this schema in {class}`vllm.inputs.PromptType`
21
You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
22
23
```python
24
+from vllm import LLM
25
+
26
llm = LLM(model="llava-hf/llava-1.5-7b-hf")
27
28
# Refer to the HuggingFace repo for the correct format to use
@@ -65,6 +67,8 @@ Full example: <gh-file:examples/offline_inference/vision_language.py>
67
To substitute multiple images inside the same text prompt, you can pass in a list of images instead:
68
69
```python
70
+from vllm import LLM
71
+
72
llm = LLM(
73
model="microsoft/Phi-3.5-vision-instruct",
74
trust_remote_code=True, # Required to load Phi-3.5-vision
@@ -96,6 +100,8 @@ Full example: <gh-file:examples/offline_inference/vision_language_multi_image.py
100
Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:
101
102
```python
103
+from vllm import LLM
104
+
105
# Specify the maximum number of frames per video to be 4. This can be changed.
106
llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
107
@@ -139,6 +145,8 @@ To input pre-computed embeddings belonging to a data type (i.e. image, video, or
145
pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
146
147
```python
148
+from vllm import LLM
149
+
150
# Inference with image embeddings as input
151
llm = LLM(model="llava-hf/llava-1.5-7b-hf")
152
docs/source/serving/offline_inference.md
+6
@@ -11,6 +11,8 @@ For example, the following code downloads the [`facebook/opt-125m`](https://hugg
11
and runs it in vLLM using the default configuration.
12
13
```python
14
+from vllm import LLM
15
+
16
llm = LLM(model="facebook/opt-125m")
17
```
18
@@ -47,6 +49,8 @@ To fix this, explicitly specify the model architecture by passing `config.json`
49
For example:
50
51
```python
52
+from vllm import LLM
53
+
54
model = LLM(
55
model="cerebras/Cerebras-GPT-1.3B",
56
hf_overrides={"architectures": ["GPT2LMHeadModel"]}, # GPT-2
@@ -92,6 +96,8 @@ You can further reduce memory usage by limiting the context length of the model
96
and the maximum batch size (`max_num_seqs` option).
97
98
```python
99
+from vllm import LLM
100
+
101
llm = LLM(model="adept/fuyu-8b",
102
max_model_len=2048,
103
max_num_seqs=2)