@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 462 lines 18.4 KB
Rendered Raw
1 # Scoring Usages
2
3 The score models is designed to compute similarity scores between two input prompts. It supports three model types (aka `score_type`): `cross-encoder`, `late-interaction`, and `bi-encoder`.
4
5 !!! note
6 vLLM handles only the model inference component of RAG pipelines (such as embedding generation and reranking). For higher-level RAG orchestration, you should leverage integration frameworks like [LangChain](https://github.com/langchain-ai/langchain).
7
8 ## Summary
9
10 - Model Usage: Scoring
11 - Pooling Task:
12
13 | Score Types | Pooling Tasks | scoring function |
14 |--------------------|-----------------------|--------------------------|
15 | `cross-encoder` | `classify` (see note) | linear classifier |
16 | `late-interaction` | `token_embed` | late interaction(MaxSim) |
17 | `bi-encoder` | `embed` | cosine similarity |
18
19 - Offline APIs:
20 - `LLM.score`
21 - Online APIs:
22 - [Score API](scoring.md#score-api) (`/score`, `/v1/score`)
23 - [Cohere Rerank API](scoring.md#cohere-rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
24
25 !!! note
26 Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled.
27
28 ### Score Types
29
30 The three supported scoring functions are as illustrated in the figure below.
31
32 ![Score Types](../../assets/models/pooling_models/score_types.svg)
33
34 ## Supported Models
35
36 ### Cross-encoder models
37
38 [Cross-encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) (aka reranker) models are a subset of classification models that accept two prompts as input and output num_labels equal to 1.
39
40 --8<-- [start:supported-cross-encoder-models]
41
42 #### Text-only Models
43
44 | Architecture | Models | Example HF Models | Score template (see note) | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
45 | ------------ | ------ | ----------------- | ------------------------- | --------------------------- | --------------------------------------- |
46 | `BertForSequenceClassification` | BERT-based | `cross-encoder/ms-marco-MiniLM-L-6-v2`, etc. | N/A | | |
47 | `GemmaForSequenceClassification` | Gemma-based | `BAAI/bge-reranker-v2-gemma`(see note), etc. | [bge-reranker-v2-gemma.jinja](../../../examples/pooling/score/template/bge-reranker-v2-gemma.jinja) | ✅︎ | ✅︎ |
48 | `GteNewForSequenceClassification` | mGTE-TRM (see note) | `Alibaba-NLP/gte-multilingual-reranker-base`, etc. | N/A | | |
49 | `LlamaBidirectionalForSequenceClassification`<sup>C</sup> | Llama-based with bidirectional attention | `nvidia/llama-nemotron-rerank-1b-v2`, etc. | [nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja) | ✅︎ | ✅︎ |
50 | `ModernBertForSequenceClassification` | ModernBERT-based | `Alibaba-NLP/gte-reranker-modernbert-base`, etc. | N/A | | |
51 | `Qwen2ForSequenceClassification`<sup>C</sup> | Qwen2-based | `mixedbread-ai/mxbai-rerank-base-v2`(see note), etc. | [mxbai_rerank_v2.jinja](../../../examples/pooling/score/template/mxbai_rerank_v2.jinja) | ✅︎ | ✅︎ |
52 | `Qwen3ForSequenceClassification`<sup>C</sup> | Qwen3-based | `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `Qwen/Qwen3-Reranker-0.6B`(see note), etc. | [qwen3_reranker.jinja](../../../examples/pooling/score/template/qwen3_reranker.jinja) | ✅︎ | ✅︎ |
53 | `RobertaForSequenceClassification` | RoBERTa-based | `cross-encoder/quora-roberta-base`, etc. | N/A | ✅︎ | |
54 | `XLMRobertaForSequenceClassification` | XLM-RoBERTa-based | `BAAI/bge-reranker-v2-m3`, etc. | N/A | ✅︎ | |
55 | `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | N/A | \* | \* |
56
57 <sup>C</sup> Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))
58 \* Feature support is the same as that of the original model.
59
60 !!! note
61 Some models require a specific prompt format to work correctly.
62
63 You can find Example HF Models's corresponding score template in [examples/pooling/score/template/](../../../examples/pooling/score/template)
64
65 Examples : [examples/pooling/score/using_template_offline.py](../../../examples/pooling/score/using_template_offline.py) [examples/pooling/score/using_template_online.py](../../../examples/pooling/score/using_template_online.py)
66
67 !!! note
68 Load the official original `BAAI/bge-reranker-v2-gemma` by using the following command.
69
70 ```bash
71 vllm serve BAAI/bge-reranker-v2-gemma --hf_overrides '{"architectures": ["GemmaForSequenceClassification"],"classifier_from_token": ["Yes"],"method": "no_post_processing"}'
72 ```
73
74 !!! note
75 The second-generation GTE model (mGTE-TRM) is named `NewForSequenceClassification`. The name `NewForSequenceClassification` is too generic, you should set `--hf-overrides '{"architectures": ["GteNewForSequenceClassification"]}'` to specify the use of the `GteNewForSequenceClassification` architecture.
76
77 !!! note
78 Load the official original `mxbai-rerank-v2` by using the following command.
79
80 ```bash
81 vllm serve mixedbread-ai/mxbai-rerank-base-v2 --hf_overrides '{"architectures": ["Qwen2ForSequenceClassification"],"classifier_from_token": ["0", "1"], "method": "from_2_way_softmax"}'
82 ```
83
84 !!! note
85 Load the official original `Qwen3 Reranker` by using the following command. More information can be found at: [examples/pooling/score/qwen3_reranker_offline.py](../../../examples/pooling/score/qwen3_reranker_offline.py) [examples/pooling/score/qwen3_reranker_online.py](../../../examples/pooling/score/qwen3_reranker_online.py).
86
87 ```bash
88 vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
89 ```
90
91 #### Multimodal Models
92
93 !!! note
94 For more information about multimodal models inputs, see [this page](../supported_models.md#list-of-multimodal-language-models).
95
96 | Architecture | Models | Inputs | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
97 | ------------ | ------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |
98 | `JinaVLForSequenceClassification` | JinaVL-based | T + I<sup>E+</sup> | `jinaai/jina-reranker-m0`, etc. | ✅︎ | ✅︎ |
99 | `LlamaNemotronVLForSequenceClassification` | Llama Nemotron Reranker + SigLIP | T + I<sup>E+</sup> | `nvidia/llama-nemotron-rerank-vl-1b-v2` | | |
100 | `Qwen3VLForSequenceClassification` | Qwen3-VL-Reranker | T + I<sup>E+</sup> + V<sup>E+</sup> | `Qwen/Qwen3-VL-Reranker-2B`(see note), etc. | ✅︎ | ✅︎ |
101
102 <sup>C</sup> Automatically converted into a classification model via `--convert classify`. ([details](README.md#model-conversion))
103 \* Feature support is the same as that of the original model.
104
105 !!! note
106 Similar to Qwen3-Reranker, you need to use the following `--hf_overrides` to load the official original `Qwen3-VL-Reranker`. `Qwen3-VL` officially uses `qwen_vl_utils` for image preprocessing, while vLLM uses `transformers`' `video_processing_qwen3_vl`, which leads to slightly different results compared to the official Hugging Face repository examples.
107
108 ```bash
109 vllm serve Qwen/Qwen3-VL-Reranker-2B --hf_overrides '{"architectures": ["Qwen3VLForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
110 ```
111
112 --8<-- [end:supported-cross-encoder-models]
113
114 ### Late-interaction models
115
116 All models that support token embedding task also support using the score API to compute similarity scores by calculating the late interaction of two input prompts. See [this page](token_embed.md) for more information about token embedding models.
117
118 --8<-- "docs/models/pooling_models/token_embed.md:supported-token-embed-models"
119
120 ### Bi-encoder
121
122 All models that support embedding task also support using the score API to compute similarity scores by calculating the cosine similarity of two input prompt's embeddings. See [this page](embed.md) for more information about embedding models.
123
124 --8<-- "docs/models/pooling_models/embed.md:supported-embed-models"
125
126 ## Offline Inference
127
128 ### Pooling Parameters
129
130 The following [pooling parameters][vllm.PoolingParams] are only supported by cross-encoder models and do not work for late-interaction and bi-encoder models.
131
132 ```python
133 --8<-- "vllm/pooling_params.py:common-pooling-params"
134 --8<-- "vllm/pooling_params.py:classify-pooling-params"
135 ```
136
137 ### `LLM.score`
138
139 The [score][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.score] method outputs similarity scores between sentence pairs.
140
141 ```python
142 from vllm import LLM
143
144 llm = LLM(model="BAAI/bge-reranker-v2-m3", runner="pooling")
145 (output,) = llm.score(
146 "What is the capital of France?",
147 "The capital of Brazil is Brasilia.",
148 )
149
150 score = output.outputs.score
151 print(f"Score: {score}")
152 ```
153
154 A code example can be found here: [examples/basic/offline_inference/score.py](../../../examples/basic/offline_inference/score.py)
155
156 ## Online Serving
157
158 ### Score API
159
160 Our Score API (`/score`, `/v1/score`) is similar to `LLM.score`, compute similarity scores between two input prompts.
161
162 #### Parameters
163
164 The following Score API parameters are supported:
165
166 ```python
167 --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"
168 --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"
169 --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"
170 --8<-- "vllm/entrypoints/pooling/scoring/protocol.py:scoring-common-params"
171 --8<-- "vllm/entrypoints/pooling/scoring/protocol.py:score-request-params"
172 ```
173
174 #### Examples
175
176 ##### Single inference
177
178 You can pass a string to both `queries` and `documents`, forming a single sentence pair.
179
180 ```bash
181 curl -X 'POST' \
182 'http://127.0.0.1:8000/score' \
183 -H 'accept: application/json' \
184 -H 'Content-Type: application/json' \
185 -d '{
186 "model": "BAAI/bge-reranker-v2-m3",
187 "encoding_format": "float",
188 "queries": "What is the capital of France?",
189 "documents": "The capital of France is Paris."
190 }'
191 ```
192
193 ??? console "Response"
194
195 ```json
196 {
197 "id": "score-request-id",
198 "object": "list",
199 "created": 693447,
200 "model": "BAAI/bge-reranker-v2-m3",
201 "data": [
202 {
203 "index": 0,
204 "object": "score",
205 "score": 1
206 }
207 ],
208 "usage": {}
209 }
210 ```
211
212 ##### Batch inference
213
214 You can pass a string to `queries` and a list to `documents`, forming multiple sentence pairs
215 where each pair is built from `queries` and a string in `documents`.
216 The total number of pairs is `len(documents)`.
217
218 ??? console "Request"
219
220 ```bash
221 curl -X 'POST' \
222 'http://127.0.0.1:8000/score' \
223 -H 'accept: application/json' \
224 -H 'Content-Type: application/json' \
225 -d '{
226 "model": "BAAI/bge-reranker-v2-m3",
227 "queries": "What is the capital of France?",
228 "documents": [
229 "The capital of Brazil is Brasilia.",
230 "The capital of France is Paris."
231 ]
232 }'
233 ```
234
235 ??? console "Response"
236
237 ```json
238 {
239 "id": "score-request-id",
240 "object": "list",
241 "created": 693570,
242 "model": "BAAI/bge-reranker-v2-m3",
243 "data": [
244 {
245 "index": 0,
246 "object": "score",
247 "score": 0.001094818115234375
248 },
249 {
250 "index": 1,
251 "object": "score",
252 "score": 1
253 }
254 ],
255 "usage": {}
256 }
257 ```
258
259 You can pass a list to both `queries` and `documents`, forming multiple sentence pairs
260 where each pair is built from a string in `queries` and the corresponding string in `documents` (similar to `zip()`).
261 The total number of pairs is `len(documents)`.
262
263 ??? console "Request"
264
265 ```bash
266 curl -X 'POST' \
267 'http://127.0.0.1:8000/score' \
268 -H 'accept: application/json' \
269 -H 'Content-Type: application/json' \
270 -d '{
271 "model": "BAAI/bge-reranker-v2-m3",
272 "encoding_format": "float",
273 "queries": [
274 "What is the capital of Brazil?",
275 "What is the capital of France?"
276 ],
277 "documents": [
278 "The capital of Brazil is Brasilia.",
279 "The capital of France is Paris."
280 ]
281 }'
282 ```
283
284 ??? console "Response"
285
286 ```json
287 {
288 "id": "score-request-id",
289 "object": "list",
290 "created": 693447,
291 "model": "BAAI/bge-reranker-v2-m3",
292 "data": [
293 {
294 "index": 0,
295 "object": "score",
296 "score": 1
297 },
298 {
299 "index": 1,
300 "object": "score",
301 "score": 1
302 }
303 ],
304 "usage": {}
305 }
306 ```
307
308 ##### Multi-modal inputs
309
310 You can pass multi-modal inputs to scoring models by passing `content` including a list of multi-modal input (image, etc.) in the request. Refer to the examples below for illustration.
311
312 === "JinaVL-Reranker"
313
314 To serve the model:
315
316 ```bash
317 vllm serve jinaai/jina-reranker-m0
318 ```
319
320 Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
321
322 ??? Code
323
324 ```python
325 import requests
326
327 response = requests.post(
328 "http://localhost:8000/v1/score",
329 json={
330 "model": "jinaai/jina-reranker-m0",
331 "queries": "slm markdown",
332 "documents": [
333 {
334 "content": [
335 {
336 "type": "image_url",
337 "image_url": {
338 "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
339 },
340 }
341 ],
342 },
343 {
344 "content": [
345 {
346 "type": "image_url",
347 "image_url": {
348 "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
349 },
350 }
351 ]
352 },
353 ],
354 },
355 )
356 response.raise_for_status()
357 response_json = response.json()
358 print("Scoring output:", response_json["data"][0]["score"])
359 print("Scoring output:", response_json["data"][1]["score"])
360 ```
361 Full example:
362
363 - [examples/pooling/score/vision_score_api_online.py](../../../examples/pooling/score/vision_score_api_online.py)
364 - [examples/pooling/score/vision_rerank_api_online.py](../../../examples/pooling/score/vision_rerank_api_online.py)
365
366 ### Cohere Rerank API
367
368 `/rerank`, `/v1/rerank`, and `/v2/rerank` APIs are compatible with both [Jina AI's rerank API interface](https://jina.ai/reranker/) and
369 [Cohere's rerank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with
370 popular open-source tools.
371
372 Code example: [examples/pooling/score/rerank_api_online.py](../../../examples/pooling/score/rerank_api_online.py)
373
374 #### Parameters
375
376 The following rerank api parameters are supported:
377
378 ```python
379 --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"
380 --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"
381 --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"
382 --8<-- "vllm/entrypoints/pooling/scoring/protocol.py:scoring-common-params"
383 --8<-- "vllm/entrypoints/pooling/scoring/protocol.py:rerank-request-params"
384 ```
385
386 #### Examples
387
388 Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.
389 Result documents will be sorted by relevance, and the `index` property can be used to determine original order.
390
391 ??? console "Request"
392
393 ```bash
394 curl -X 'POST' \
395 'http://127.0.0.1:8000/v1/rerank' \
396 -H 'accept: application/json' \
397 -H 'Content-Type: application/json' \
398 -d '{
399 "model": "BAAI/bge-reranker-base",
400 "query": "What is the capital of France?",
401 "documents": [
402 "The capital of Brazil is Brasilia.",
403 "The capital of France is Paris.",
404 "Horses and cows are both animals"
405 ]
406 }'
407 ```
408
409 ??? console "Response"
410
411 ```json
412 {
413 "id": "rerank-fae51b2b664d4ed38f5969b612edff77",
414 "model": "BAAI/bge-reranker-base",
415 "usage": {
416 "total_tokens": 56
417 },
418 "results": [
419 {
420 "index": 1,
421 "document": {
422 "text": "The capital of France is Paris."
423 },
424 "relevance_score": 0.99853515625
425 },
426 {
427 "index": 0,
428 "document": {
429 "text": "The capital of Brazil is Brasilia."
430 },
431 "relevance_score": 0.0005860328674316406
432 }
433 ]
434 }
435 ```
436
437 ## More examples
438
439 More examples can be found here: [examples/pooling/score](../../../examples/pooling/score)
440
441 ## Supported Features
442
443 As cross-encoder models are a subset of classification models that accept two prompts as input and output num_labels equal to 1, cross-encoder features should be consistent with (sequence) classification. For more information, see [this page](classify.md#supported-features).
444
445 ### Score Template
446
447 Score templates are supported for **cross-encoder** models only. If you are using an **embedding** model for scoring, vLLM does not apply a score template.
448
449 Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/online_serving/README.md#chat-template)).
450
451 Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter:
452
453 - **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}`
454 - **Document**: `{{ (messages | selectattr("role", "eq", "document") | first).content }}`
455
456 This approach is more robust than index-based access (`messages[0]`, `messages[1]`) because it selects messages by their semantic role. It also avoids assumptions about message ordering if additional message types are added to `messages` in the future.
457
458 Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)
459
460 ### Enable/disable activation
461
462 You can enable or disable activation via `use_activation` only works for cross-encoder models.