A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 4, 17:30 UTC
[Deprecate] Deprecate LLM.reward offline api, use LLM.encode instead. (#40688)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
wang.yuqi committed
Apr 24, 2026 at 13:37 UTC
9744b699bafed423909ed10da96b80eb0542424b
11 files changed
+203
-38
docs/models/pooling_models/README.md
+27
-21
@@ -78,7 +78,7 @@ The scoring models is designed to compute similarity scores between two input pr
78
|-----------------------|---------------|----------------------------------------------|--------------------|--------------------------|
79
| `classify` (see note) | Sequence-wise | reranker score for each sequence | `cross-encoder` | linear classifier |
80
| `embed` | Sequence-wise | vector representations for each sequence | `bi-encoder` | cosine similarity |
81
-| `token_classify` | Token-wise | probability vector of classes for each token | nan | nan |
81
+| `token_classify` | Token-wise | probability vector of classes for each token | N/A | N/A |
82
| `token_embed` | Token-wise | vector representations for each token | `late-interaction` | late interaction(MaxSim) |
83
84
!!! note
@@ -86,14 +86,15 @@ The scoring models is designed to compute similarity scores between two input pr
86
87
### Pooling Usages
88
89
-| Pooling Usages | Description |
90
-|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
91
-| Classification Usages | Predicting which predefined category, class, or label best corresponds to a given input. |
92
-| Embedding Usages | Converts unstructured data (text, images, audio, etc.) into structured numerical vectors (embeddings). |
93
-| Token Classification Usages | Token-wise classification |
94
-| Token Embedding Usages | Token-wise embedding |
95
-| Scoring Usages | Computes similarity scores between two inputs. It supports three model types (aka `score_type`): `cross-encoder`, `late-interaction`, and `bi-encoder`. |
96
-| Reward Usages | Evaluates the quality of outputs generated by a language model, acting as a proxy for human preferences. |
89
+| Pooling Usages | Description |
90
+|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
91
+| Classification Usages | Predicting which predefined category, class, or label best corresponds to a given input. |
92
+| Embedding Usages | Converts unstructured data (text, images, audio, etc.) into structured numerical vectors (embeddings). |
93
+| Token Classification Usages | Token-wise classification |
94
+| Token Embedding Usages | Token-wise embedding |
95
+| Reward Usages | Evaluates the quality of outputs generated by a language model, acting as a proxy for human preferences. |
96
+| Scoring Usages | Computes similarity scores between two inputs. It supports three model types (aka `score_type`): `cross-encoder`, `late-interaction`, and `bi-encoder`. |
97
+| Plugins Usages | Allow users to customize input and output processors. For more information, please refer to [IO Processor Plugins](../../design/io_processor_plugins.md). |
98
99
We also have some special models that support multiple pooling tasks, or have specific usage scenarios, or support special inputs and outputs.
100
@@ -101,9 +102,9 @@ For more detailed information, please refer to the link below.
102
103
- [Classification Usages](classify.md)
104
- [Embedding Usages](embed.md)
104
-- [Reward Usages](reward.md)
105
- [Token Classification Usages](token_classify.md)
106
- [Token Embedding Usages](token_embed.md)
107
+- [Reward Usages](reward.md)
108
- [Scoring Usages](scoring.md)
109
- [Specific Model Examples](specific_models.md)
110
@@ -113,15 +114,17 @@ Each pooling model in vLLM supports one or more of these tasks according to
114
[Pooler.get_supported_tasks][vllm.model_executor.layers.pooler.Pooler.get_supported_tasks],
115
enabling the corresponding APIs.
116
116
-### Offline APIs corresponding to pooling tasks
117
+### Offline APIs corresponding to pooling usages
118
118
-| Task | APIs |
119
-|------------------|---------------------------------------------------------------------------------------|
120
-| `embed` | `LLM.embed(...)`, `LLM.encode(..., pooling_task="embed")`, `LLM.score(...)`(see note) |
121
-| `classify` | `LLM.classify(...)`, `LLM.encode(..., pooling_task="classify")`, `LLM.score(...)` |
122
-| `token_classify` | `LLM.reward(...)`, `LLM.encode(..., pooling_task="token_classify")` |
123
-| `token_embed` | `LLM.encode(..., pooling_task="token_embed")`, `LLM.score(...)` |
124
-| `plugin` | `LLM.encode(..., pooling_task="plugin")` |
119
+| Pooling Usages | Dedicated API | Pooling task for `LLM.encode` API | Score Types | scoring function |
120
+|-----------------------------|---------------------|-----------------------------------|----------------------------|--------------------------|
121
+| Classification Usages | `LLM.classify(...)` | `classify` | `cross-encoder` (see note) | linear classifier |
122
+| Embedding Usages | `LLM.embed(...)` | `embed` | `bi-encoder` | cosine similarity |
123
+| Token Classification Usages | N/A | `token_classify` | N/A | N/A |
124
+| Token Embedding Usages | N/A | `token_embed` | `late-interaction` | late interaction(MaxSim) |
125
+| Reward Usages | N/A | `classify` & `token_classify` | N/A | N/A |
126
+| Scoring Usages | `LLM.score(...)` | N/A | N/A | N/A |
127
+| Plugins Usages | N/A | `plugin` | N/A | N/A |
128
129
!!! note
130
Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled.
@@ -147,7 +150,7 @@ It is primarily designed for [score models](scoring.md).
150
151
The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.
152
150
-Please use one of the more specific methods or set the task directly when using `LLM.encode`, refer to the [table above](#offline-apis-corresponding-to-pooling-tasks).
153
+Please use one of the more specific methods or set the task directly when using `LLM.encode`, refer to the [table above](#offline-apis-corresponding-to-pooling-usages).
154
155
### Examples
156
@@ -183,9 +186,12 @@ Our Pooling API (`/pooling`) is similar to `LLM.encode`, being applicable to all
186
187
The input format is the same as [Embeddings API](embed.md#openai-compatible-embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
188
186
-Please use one of the more specific APIs or set the task directly when using the Pooling API, refer to the [table above](#offline-apis-corresponding-to-pooling-tasks).
189
+Please use one of the more specific APIs or set the task directly when using the Pooling API, refer to the [table above](#offline-apis-corresponding-to-pooling-usages).
190
+
191
+Code examples:
192
188
-Code example: [examples/pooling/pooling/pooling_online.py](../../../examples/pooling/pooling/pooling_online.py)
193
+- [Online example](../../../examples/pooling/reward/token_reward_online.py)
194
+- [Offline example](../../../examples/pooling/reward/token_reward_offline.py)
195
196
### Examples
197
docs/models/pooling_models/reward.md
+10
@@ -134,3 +134,13 @@ print(f"Data: {data!r}")
134
## Online Serving
135
136
Please refer to the [pooling API](README.md#pooling-api). Pooling task corresponding to reward model types refer to the [table above](#summary).
137
+
138
+## More examples
139
+
140
+More examples can be found here: [examples/pooling/reward](../../../examples/pooling/reward)
141
+
142
+## Deprecated Features
143
+
144
+### `LLM.reward`
145
+
146
+`llm.reward` api is deprecated and will be removed in v0.23. Please use `LLM.encode` with `pooling_task="classify"` or `pooling_task="token_classify"` instead.
examples/pooling/reward/sequence_reward_offline.py
new
+62
@@ -0,0 +1,62 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+
4
+"""
5
+Example offline usage of sequence reward models.
6
+
7
+The key distinction between sequence classification and token classification
8
+lies in their output granularity: sequence classification produces a single
9
+result for an entire input sequence, whereas token classification yields a
10
+result for each individual token within the sequence.
11
+"""
12
+
13
+from argparse import Namespace
14
+
15
+from vllm import LLM, EngineArgs
16
+from vllm.utils.argparse_utils import FlexibleArgumentParser
17
+from vllm.utils.print_utils import print_embeddings
18
+
19
+
20
+def parse_args():
21
+ parser = FlexibleArgumentParser()
22
+ parser = EngineArgs.add_cli_args(parser)
23
+ # Set example specific arguments
24
+ parser.set_defaults(
25
+ model="Skywork/Skywork-Reward-V2-Qwen3-0.6B",
26
+ runner="pooling",
27
+ enforce_eager=True,
28
+ max_model_len=1024,
29
+ trust_remote_code=True,
30
+ )
31
+ return parser.parse_args()
32
+
33
+
34
+def main(args: Namespace):
35
+ # Sample prompts.
36
+ prompts = [
37
+ "Hello, my name is",
38
+ "The president of the United States is",
39
+ "The capital of France is",
40
+ "The future of AI is",
41
+ ]
42
+
43
+ # Create an LLM.
44
+ # You should pass runner="pooling" for reward models
45
+ llm = LLM(**vars(args))
46
+
47
+ # Generate rewards. The output is a list of PoolingRequestOutput.
48
+ # Use pooling_task="classify" for sequence reward models.
49
+ outputs = llm.encode(prompts, pooling_task="classify")
50
+
51
+ # Print the outputs.
52
+ print("\nGenerated Outputs:\n" + "-" * 60)
53
+ for prompt, output in zip(prompts, outputs):
54
+ rewards = output.outputs.data
55
+ print(f"Prompt: {prompt!r}")
56
+ print_embeddings(rewards.tolist(), prefix="Reward")
57
+ print("-" * 60)
58
+
59
+
60
+if __name__ == "__main__":
61
+ args = parse_args()
62
+ main(args)
examples/pooling/reward/sequence_reward_online.py
new
+71
@@ -0,0 +1,71 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+"""
4
+Example online usage of sequence reward models.
5
+
6
+Run `vllm serve <model> --runner pooling`
7
+to start up the server in vLLM. e.g.
8
+
9
+vllm serve Skywork/Skywork-Reward-V2-Qwen3-0.6B
10
+
11
+The key distinction between sequence classification and token classification
12
+lies in their output granularity: sequence classification produces a single
13
+result for an entire input sequence, whereas token classification yields a
14
+result for each individual token within the sequence.
15
+"""
16
+
17
+import argparse
18
+import pprint
19
+
20
+import requests
21
+
22
+
23
+def post_http_request(prompt: dict, api_url: str) -> requests.Response:
24
+ headers = {"User-Agent": "Test Client"}
25
+ response = requests.post(api_url, headers=headers, json=prompt)
26
+ return response
27
+
28
+
29
+def parse_args():
30
+ parser = argparse.ArgumentParser()
31
+ parser.add_argument("--host", type=str, default="localhost")
32
+ parser.add_argument("--port", type=int, default=8000)
33
+
34
+ return parser.parse_args()
35
+
36
+
37
+def main(args):
38
+ base_url = f"http://{args.host}:{args.port}"
39
+ models_url = base_url + "/v1/models"
40
+ pooing_url = base_url + "/pooling"
41
+
42
+ response = requests.get(models_url)
43
+ model = response.json()["data"][0]["id"]
44
+
45
+ # Input like Completions API
46
+ prompt = {"model": model, "input": "vLLM is great!"}
47
+ pooling_response = post_http_request(prompt=prompt, api_url=pooing_url)
48
+ print("-" * 50)
49
+ print("Pooling Response:")
50
+ pprint.pprint(pooling_response.json())
51
+ print("-" * 50)
52
+
53
+ # Input like Chat API
54
+ prompt = {
55
+ "model": model,
56
+ "messages": [
57
+ {
58
+ "role": "user",
59
+ "content": [{"type": "text", "text": "vLLM is great!"}],
60
+ }
61
+ ],
62
+ }
63
+ pooling_response = post_http_request(prompt=prompt, api_url=pooing_url)
64
+ print("Pooling Response:")
65
+ pprint.pprint(pooling_response.json())
66
+ print("-" * 50)
67
+
68
+
69
+if __name__ == "__main__":
70
+ args = parse_args()
71
+ main(args)
examples/pooling/reward/token_reward_offline.py
renamed
+11
-2
@@ -1,6 +1,15 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4
+"""
5
+Example offline usage of token reward models.
6
+
7
+The key distinction between sequence classification and token classification
8
+lies in their output granularity: sequence classification produces a single
9
+result for an entire input sequence, whereas token classification yields a
10
+result for each individual token within the sequence.
11
+"""
12
+
13
from argparse import Namespace
14
15
from vllm import LLM, EngineArgs
@@ -36,14 +45,14 @@ def main(args: Namespace):
45
llm = LLM(**vars(args))
46
47
# Generate rewards. The output is a list of PoolingRequestOutput.
39
- outputs = llm.reward(prompts)
48
+ outputs = llm.encode(prompts, pooling_task="token_classify")
49
50
# Print the outputs.
51
print("\nGenerated Outputs:\n" + "-" * 60)
52
for prompt, output in zip(prompts, outputs):
53
rewards = output.outputs.data
54
print(f"Prompt: {prompt!r}")
46
- print_embeddings(rewards, prefix="Reward")
55
+ print_embeddings(rewards.tolist(), prefix="Reward")
56
print("-" * 60)
57
58
examples/pooling/reward/token_reward_online.py
renamed
+6
-1
@@ -1,12 +1,17 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
"""
4
-Example online usage of Pooling API.
4
+Example online usage of token reward models.
5
6
Run `vllm serve <model> --runner pooling`
7
to start up the server in vLLM. e.g.
8
9
vllm serve internlm/internlm2-1_8b-reward --trust-remote-code
10
+
11
+The key distinction between sequence classification and token classification
12
+lies in their output granularity: sequence classification produces a single
13
+result for an entire input sequence, whereas token classification yields a
14
+result for each individual token within the sequence.
15
"""
16
17
import argparse
tests/conftest.py
+1
-1
@@ -1183,7 +1183,7 @@ class VllmRunner:
1183
return [req_output.outputs.data for req_output in req_outputs]
1184
1185
def reward(self, prompts: list[str]) -> list[list[float]]:
1186
- req_outputs = self.llm.reward(prompts)
1186
+ req_outputs = self.llm.encode(prompts, pooling_task="token_classify")
1187
return [req_output.outputs.data for req_output in req_outputs]
1188
1189
def score(
tests/entrypoints/pooling/pooling/__init__.py
tests/entrypoints/pooling/reward/test_token_reward_offline.py
renamed
tests/entrypoints/pooling/reward/test_token_reward_online.py
renamed
vllm/entrypoints/llm.py
+15
-13
@@ -1166,19 +1166,16 @@ class LLM:
1166
1167
if pooling_task is None:
1168
raise ValueError(
1169
- "pooling_task required for `LLM.encode`\n"
1170
- "Please use one of the more specific methods or set the "
1171
- "pooling_task when using `LLM.encode`:\n"
1172
- " - For embeddings, use `LLM.embed(...)` "
1173
- 'or `pooling_task="embed"`.\n'
1174
- " - For classification logits, use `LLM.classify(...)` "
1175
- 'or `pooling_task="classify"`.\n'
1176
- " - For similarity scores, use `LLM.score(...)`.\n"
1177
- " - For rewards, use `LLM.reward(...)` "
1178
- 'or `pooling_task="token_classify"`\n'
1179
- " - For token classification, "
1180
- 'use `pooling_task="token_classify"`\n'
1181
- ' - For multi-vector retrieval, use `pooling_task="token_embed"`'
1169
+ """
1170
+ pooling_task required for `LLM.encode`.
1171
+ Please use one of the more specific methods or set the pooling_task when using `LLM.encode`:
1172
+ - For embeddings, use `LLM.embed(...)` or `pooling_task="embed"`.
1173
+ - For classification logits, use `LLM.classify(...)` or `pooling_task="classify"`.
1174
+ - For similarity scores, use `LLM.score(...)`.
1175
+ - For rewards, `pooling_task="classify"` or `pooling_task="token_classify"`.
1176
+ - For token classification, use `pooling_task="token_classify"`.
1177
+ - For multi-vector retrieval, use `pooling_task="token_embed"`.
1178
+ """ # noqa: E501
1179
)
1180
1181
if (
@@ -1340,6 +1337,11 @@ class LLM:
1337
A list of `PoolingRequestOutput` objects containing the
1338
pooled hidden states in the same order as the input prompts.
1339
"""
1340
+ logger.warning_once(
1341
+ "`llm.reward` api is deprecated and will be removed in v0.23. "
1342
+ 'Please use `LLM.encode` with `pooling_task="classify"` or '
1343
+ '`pooling_task="token_classify"` instead.'
1344
+ )
1345
return self.encode(
1346
prompts,
1347
use_tqdm=use_tqdm,