A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 6, 09:30 UTC
[Model] Deprecate the score task (this will not affect users). (#37537)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
wang.yuqi committed
Mar 20, 2026 at 16:07 UTC
ed359c497a728f08b5b41456c07a688ccd510fbc
22 files changed
+184
-163
docs/models/pooling_models/README.md
+35
-28
@@ -31,28 +31,29 @@ Of course, we also have "plugin" tasks that allow users to customize input and o
31
32
### Pooling Tasks
33
34
-| Pooling Tasks | Granularity | Outputs |
35
-|--------------------|---------------|-------------------------------------------------|
36
-| `classify` | Sequence-wise | probability vector of classes for each sequence |
37
-| `score` (see note) | Sequence-wise | reranker score for each sequence |
38
-| `embed` | Sequence-wise | vector representations for each sequence |
39
-| `token_classify` | Token-wise | probability vector of classes for each token |
40
-| `token_embed` | Token-wise | vector representations for each token |
34
+| Pooling Tasks | Granularity | Outputs |
35
+|-----------------------|---------------|-------------------------------------------------|
36
+| `classify` (see note) | Sequence-wise | probability vector of classes for each sequence |
37
+| `embed` | Sequence-wise | vector representations for each sequence |
38
+| `token_classify` | Token-wise | probability vector of classes for each token |
39
+| `token_embed` | Token-wise | vector representations for each token |
40
41
!!! note
42
Within classification tasks, there is a specialized subcategory: Cross-encoder (aka reranker) models. These models are a subset of classification models that accept two prompts as input and output num_labels equal to 1.
43
44
### Score Types
45
47
-| Pooling Tasks | Granularity | Outputs | Score Types | scoring function |
48
-|--------------------|---------------|-------------------------------------------------|--------------------|--------------------------|
49
-| `classify` | Sequence-wise | probability vector of classes for each sequence | nan | nan |
50
-| `score` (see note) | Sequence-wise | reranker score for each sequence | `cross-encoder` | linear classifier |
51
-| `embed` | Sequence-wise | vector representations for each sequence | `bi-encoder` | cosine similarity |
52
-| `token_classify` | Token-wise | probability vector of classes for each token | nan | nan |
53
-| `token_embed` | Token-wise | vector representations for each token | `late-interaction` | late interaction(MaxSim) |
46
+The scoring models is designed to compute similarity scores between two input prompts. It supports three model types (aka `score_type`): `cross-encoder`, `late-interaction`, and `bi-encoder`.
47
55
-The score models is designed to compute similarity scores between two input prompts. It supports three model types (aka `score_type`): `cross-encoder`, `late-interaction`, and `bi-encoder`.
48
+| Pooling Tasks | Granularity | Outputs | Score Types | scoring function |
49
+|-----------------------|---------------|----------------------------------------------|--------------------|--------------------------|
50
+| `classify` (see note) | Sequence-wise | reranker score for each sequence | `cross-encoder` | linear classifier |
51
+| `embed` | Sequence-wise | vector representations for each sequence | `bi-encoder` | cosine similarity |
52
+| `token_classify` | Token-wise | probability vector of classes for each token | nan | nan |
53
+| `token_embed` | Token-wise | vector representations for each token | `late-interaction` | late interaction(MaxSim) |
54
+
55
+!!! note
56
+ Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled.
57
58
### Pooling Usages
59
@@ -85,14 +86,16 @@ enabling the corresponding APIs.
86
87
### Offline APIs corresponding to pooling tasks
88
88
-| Task | APIs |
89
-|------------------|----------------------------------------------------------------------------|
90
-| `embed` | `LLM.embed(...)`,`LLM.encode(..., pooling_task="embed")`, `LLM.score(...)` |
91
-| `classify` | `LLM.classify(...)`, `LLM.encode(..., pooling_task="classify")` |
92
-| `score` | `LLM.score(...)` |
93
-| `token_classify` | `LLM.reward(...)`, `LLM.encode(..., pooling_task="token_classify")` |
94
-| `token_embed` | `LLM.encode(..., pooling_task="token_embed")`, `LLM.score(...)` |
95
-| `plugin` | `LLM.encode(..., pooling_task="plugin")` |
89
+| Task | APIs |
90
+|------------------|---------------------------------------------------------------------------------------|
91
+| `embed` | `LLM.embed(...)`, `LLM.encode(..., pooling_task="embed")`, `LLM.score(...)`(see note) |
92
+| `classify` | `LLM.classify(...)`, `LLM.encode(..., pooling_task="classify")`, `LLM.score(...)` |
93
+| `token_classify` | `LLM.reward(...)`, `LLM.encode(..., pooling_task="token_classify")` |
94
+| `token_embed` | `LLM.encode(..., pooling_task="token_embed")`, `LLM.score(...)` |
95
+| `plugin` | `LLM.encode(..., pooling_task="plugin")` |
96
+
97
+!!! note
98
+ Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled.
99
100
### `LLM.classify`
101
@@ -206,11 +209,11 @@ If `--runner pooling` has been set (manually or automatically) but the model doe
209
vLLM will attempt to automatically convert the model according to the architecture names
210
shown in the table below.
211
209
-| Architecture | `--convert` | Supported pooling tasks |
210
-| ----------------------------------------------- | ----------- | ------------------------------------- |
211
-| `*ForTextEncoding`, `*EmbeddingModel`, `*Model` | `embed` | `token_embed`, `embed` |
212
-| `*ForRewardModeling`, `*RewardModel` | `embed` | `token_embed`, `embed` |
213
-| `*For*Classification`, `*ClassificationModel` | `classify` | `token_classify`, `classify`, `score` |
212
+| Architecture | `--convert` | Supported pooling tasks |
213
+|-------------------------------------------------|-------------|------------------------------|
214
+| `*ForTextEncoding`, `*EmbeddingModel`, `*Model` | `embed` | `token_embed`, `embed` |
215
+| `*ForRewardModeling`, `*RewardModel` | `embed` | `token_embed`, `embed` |
216
+| `*For*Classification`, `*ClassificationModel` | `classify` | `token_classify`, `classify` |
217
218
!!! tip
219
You can explicitly set `--convert <type>` to specify how to convert the model.
@@ -251,3 +254,7 @@ Pooling models now default support all pooling, you can use it without any setti
254
255
- Extracting hidden states prefers using `token_embed` task.
256
- Named Entity Recognition (NER) and reward models prefers using `token_classify` task.
257
+
258
+### Score task
259
+
260
+`score` task is deprecated and will be removed in v0.20. Please use `classify` instead. Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled.
docs/models/pooling_models/classify.md
+3
-1
@@ -17,6 +17,8 @@ The key distinction between (sequence) classification and token classification l
17
18
Many classification models support both (sequence) classification and token classification. For further details on token classification, please refer to [this page](token_classify.md).
19
20
+Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled, please refer to [this page](scoring.md).
21
+
22
## Typical Use Cases
23
24
### Classification
@@ -54,7 +56,7 @@ If your model is not in the above list, we will try to automatically convert the
56
57
Cross-encoder (aka reranker) models are a subset of classification models that accept two prompts as input and output num_labels equal to 1. Most classification models can also be used as [cross-encoder models](scoring.md#cross-encoder-models). For more information on cross-encoder models, please refer to [this page](scoring.md).
58
57
---8<-- "docs/models/pooling_models/scoring.md:supported-score-models"
59
+--8<-- "docs/models/pooling_models/scoring.md:supported-cross-encoder-models"
60
61
### Reward Models
62
docs/models/pooling_models/scoring.md
+10
-7
@@ -10,11 +10,11 @@ The score models is designed to compute similarity scores between two input prom
10
- Model Usage: Scoring
11
- Pooling Task:
12
13
-| Score Types | Pooling Tasks | scoring function |
14
-|--------------------|---------------|--------------------------|
15
-| `cross-encoder` | `score` | linear classifier |
16
-| `late-interaction` | `token_embed` | late interaction(MaxSim) |
17
-| `bi-encoder` | `embed` | cosine similarity |
13
+| Score Types | Pooling Tasks | scoring function |
14
+|--------------------|-----------------------|--------------------------|
15
+| `cross-encoder` | `classify` (see note) | linear classifier |
16
+| `late-interaction` | `token_embed` | late interaction(MaxSim) |
17
+| `bi-encoder` | `embed` | cosine similarity |
18
19
- Offline APIs:
20
- `LLM.score`
@@ -22,13 +22,16 @@ The score models is designed to compute similarity scores between two input prom
22
- [Score API](scoring.md#score-api) (`/score`)
23
- [Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
24
25
+!!! note
26
+ Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled.
27
+
28
## Supported Models
29
30
### Cross-encoder models
31
32
[Cross-encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) (aka reranker) models are a subset of classification models that accept two prompts as input and output num_labels equal to 1.
33
31
---8<-- [start:supported-score-models]
34
+--8<-- [start:supported-cross-encoder-models]
35
36
#### Text-only Models
37
@@ -99,7 +102,7 @@ The score models is designed to compute similarity scores between two input prom
102
vllm serve Qwen/Qwen3-VL-Reranker-2B --hf_overrides '{"architectures": ["Qwen3VLForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
103
```
104
102
---8<-- [end:supported-score-models]
105
+--8<-- [end:supported-cross-encoder-models]
106
107
### Late-interaction models
108
tests/test_pooling_params.py
+1
-1
@@ -74,7 +74,7 @@ def test_embed_dimensions(model_info: EmbedModelInfo):
74
pooling_params.verify(model_config)
75
76
77
-@pytest.mark.parametrize("task", ["score", "classify"])
77
+@pytest.mark.parametrize("task", ["classify"])
78
def test_classify(task):
79
model_config = MockModelConfig(pooler_config=PoolerConfig(seq_pooling_type="CLS"))
80
vllm/config/model.py
+4
-4
@@ -1435,10 +1435,10 @@ class ModelConfig:
1435
@property
1436
def score_type(self) -> ScoreType:
1437
"""
1438
- Score API handles score/rerank for:
1439
- - "score" task (score_type: cross-encoder models)
1440
- - "embed" task (score_type: bi-encoder models)
1441
- - "token_embed" task (score_type: late interaction models)
1438
+ Scoring API handles score/rerank for:\n
1439
+ - "classify" task (score_type: cross-encoder models)\n
1440
+ - "embed" task (score_type: bi-encoder models)\n
1441
+ - "token_embed" task (score_type: late interaction models)\n
1442
"""
1443
# fixme: self._model_info.score_type is the score type before
1444
# as_seq_cls_model, which is "bi-encoder", rather than the
vllm/entrypoints/llm.py
+2
-2
@@ -1477,9 +1477,9 @@ class LLM:
1477
data_1 = data_1 * len(data_2)
1478
1479
if pooling_params is None:
1480
- pooling_params = PoolingParams(task="score")
1480
+ pooling_params = PoolingParams(task="classify")
1481
elif pooling_params.task is None:
1482
- pooling_params.task = "score"
1482
+ pooling_params.task = "classify"
1483
1484
pooling_params_list = list[PoolingParams]()
1485
vllm/entrypoints/openai/api_server.py
+9
-5
@@ -22,7 +22,7 @@ from fastapi.middleware.cors import CORSMiddleware
22
from starlette.datastructures import State
23
24
import vllm.envs as envs
25
-from vllm.config import VllmConfig
25
+from vllm.config import ModelConfig, VllmConfig
26
from vllm.engine.arg_utils import AsyncEngineArgs
27
from vllm.engine.protocol import EngineClient
28
from vllm.entrypoints.chat_utils import load_chat_template
@@ -155,7 +155,9 @@ async def build_async_engine_client_from_engine_args(
155
156
157
def build_app(
158
- args: Namespace, supported_tasks: tuple["SupportedTask", ...] | None = None
158
+ args: Namespace,
159
+ supported_tasks: tuple["SupportedTask", ...] | None = None,
160
+ model_config: ModelConfig | None = None,
161
) -> FastAPI:
162
if supported_tasks is None:
163
warnings.warn(
@@ -191,7 +193,7 @@ def build_app(
193
attach_router as register_sagemaker_api_router,
194
)
195
194
- register_sagemaker_api_router(app, supported_tasks)
196
+ register_sagemaker_api_router(app, supported_tasks, model_config)
197
198
if "generate" in supported_tasks:
199
from vllm.entrypoints.openai.generate.api_router import (
@@ -242,7 +244,7 @@ def build_app(
244
if any(task in POOLING_TASKS for task in supported_tasks):
245
from vllm.entrypoints.pooling import register_pooling_api_routers
246
245
- register_pooling_api_routers(app, supported_tasks)
247
+ register_pooling_api_routers(app, supported_tasks, model_config)
248
249
app.root_path = args.root_path
250
app.add_middleware(
@@ -583,8 +585,10 @@ async def build_and_serve(
585
uvicorn_kwargs["log_config"] = log_config
586
587
supported_tasks = await engine_client.get_supported_tasks()
588
+ model_config = engine_client.model_config
589
+
590
logger.info("Supported tasks: %s", supported_tasks)
587
- app = build_app(args, supported_tasks)
591
+ app = build_app(args, supported_tasks, model_config)
592
await init_app_state(engine_client, app.state, args, supported_tasks)
593
594
logger.info("Starting vLLM server on %s", listen_address)
vllm/entrypoints/pooling/__init__.py
+29
-11
@@ -5,6 +5,9 @@ from typing import TYPE_CHECKING
5
6
from fastapi import FastAPI
7
8
+from vllm.config import ModelConfig
9
+from vllm.logger import init_logger
10
+
11
if TYPE_CHECKING:
12
from argparse import Namespace
13
@@ -17,9 +20,30 @@ else:
20
RequestLogger = object
21
SupportedTask = object
22
23
+logger = init_logger(__name__)
24
+
25
+
26
+def enable_scoring_api(
27
+ supported_tasks: tuple["SupportedTask", ...],
28
+ model_config: ModelConfig | None = None,
29
+) -> bool:
30
+ if any(t in supported_tasks for t in ("embed", "token_embed")):
31
+ return True
32
+
33
+ if model_config is not None and "classify" in supported_tasks:
34
+ num_labels = getattr(model_config.hf_config, "num_labels", 0)
35
+ if num_labels != 1:
36
+ logger.debug_once("Score API is only enabled for num_labels == 1.")
37
+ return False
38
+ return True
39
+
40
+ return False
41
+
42
43
def register_pooling_api_routers(
22
- app: FastAPI, supported_tasks: tuple["SupportedTask", ...]
44
+ app: FastAPI,
45
+ supported_tasks: tuple["SupportedTask", ...],
46
+ model_config: ModelConfig | None = None,
47
):
48
from vllm.entrypoints.pooling.pooling.api_router import router as pooling_router
49
@@ -37,11 +61,7 @@ def register_pooling_api_routers(
61
62
app.include_router(embed_router)
63
40
- # Score API handles score/rerank for:
41
- # - "score" task (score_type: cross-encoder models)
42
- # - "embed" task (score_type: bi-encoder models)
43
- # - "token_embed" task (score_type: late interaction models)
44
- if any(t in supported_tasks for t in ("score", "embed", "token_embed")):
64
+ if enable_scoring_api(supported_tasks, model_config):
65
from vllm.entrypoints.pooling.score.api_router import router as score_router
66
67
app.include_router(score_router)
@@ -61,6 +81,8 @@ def init_pooling_state(
81
from vllm.entrypoints.pooling.score.serving import ServingScores
82
from vllm.tasks import POOLING_TASKS
83
84
+ model_config = engine_client.model_config
85
+
86
resolved_chat_template = load_chat_template(args.chat_template)
87
88
state.serving_pooling = (
@@ -102,10 +124,6 @@ def init_pooling_state(
124
if "classify" in supported_tasks
125
else None
126
)
105
- # Score API handles score/rerank for:
106
- # - "score" task (score_type: cross-encoder models)
107
- # - "embed" task (score_type: bi-encoder models)
108
- # - "token_embed" task (score_type: late interaction models)
127
state.serving_scores = (
128
ServingScores(
129
engine_client,
@@ -114,6 +132,6 @@ def init_pooling_state(
132
score_template=resolved_chat_template,
133
log_error_stack=args.log_error_stack,
134
)
117
- if any(t in supported_tasks for t in ("embed", "score", "token_embed"))
135
+ if enable_scoring_api(supported_tasks, model_config)
136
else None
137
)
vllm/entrypoints/pooling/score/protocol.py
+2
-2
@@ -35,7 +35,7 @@ class ScoreRequestMixin(PoolingBasicRequestMixin, ClassifyRequestMixin):
35
max_total_tokens_param="max_model_len",
36
)
37
38
- def to_pooling_params(self, task: PoolingTask = "score"):
38
+ def to_pooling_params(self, task: PoolingTask = "classify"):
39
return PoolingParams(
40
task=task,
41
use_activation=self.use_activation,
@@ -111,7 +111,7 @@ class RerankRequest(PoolingBasicRequestMixin, ClassifyRequestMixin):
111
max_total_tokens_param="max_model_len",
112
)
113
114
- def to_pooling_params(self, task: PoolingTask = "score"):
114
+ def to_pooling_params(self, task: PoolingTask = "classify"):
115
return PoolingParams(
116
task=task,
117
use_activation=self.use_activation,
vllm/entrypoints/pooling/score/serving.py
+1
-1
@@ -413,7 +413,7 @@ class ServingScores(OpenAIServing):
413
# Schedule the request and get the result generator.
414
generators: list[AsyncGenerator[PoolingRequestOutput, None]] = []
415
416
- default_pooling_params = request.to_pooling_params("score")
416
+ default_pooling_params = request.to_pooling_params("classify")
417
418
for i, engine_prompt in enumerate(engine_prompts):
419
request_id_item = f"{request_id}-{i}"
vllm/entrypoints/sagemaker/api_router.py
+13
-5
@@ -10,9 +10,11 @@ import pydantic
10
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Request
11
from fastapi.responses import JSONResponse, Response
12
13
+from vllm.config import ModelConfig
14
from vllm.entrypoints.openai.engine.protocol import ErrorResponse
15
from vllm.entrypoints.openai.engine.serving import OpenAIServing
16
from vllm.entrypoints.openai.utils import validate_json_request
17
+from vllm.entrypoints.pooling import enable_scoring_api
18
from vllm.entrypoints.pooling.base.serving import PoolingServing
19
from vllm.entrypoints.serve.instrumentator.basic import base
20
from vllm.entrypoints.serve.instrumentator.health import health
@@ -25,7 +27,10 @@ GetHandlerFn = Callable[[Request], OpenAIServing | PoolingServing | None]
27
EndpointFn = Callable[[RequestType, Request], Awaitable[Any]]
28
29
28
-def get_invocation_types(supported_tasks: tuple["SupportedTask", ...]):
30
+def get_invocation_types(
31
+ supported_tasks: tuple["SupportedTask", ...],
32
+ model_config: ModelConfig | None = None,
33
+):
34
# NOTE: Items defined earlier take higher priority
35
INVOCATION_TYPES: list[tuple[RequestType, tuple[GetHandlerFn, EndpointFn]]] = []
36
@@ -70,7 +75,7 @@ def get_invocation_types(supported_tasks: tuple["SupportedTask", ...]):
75
(ClassificationRequest, (classify, create_classify)),
76
]
77
73
- if "score" in supported_tasks:
78
+ if enable_scoring_api(supported_tasks, model_config):
79
from vllm.entrypoints.pooling.score.api_router import do_rerank, rerank
80
from vllm.entrypoints.pooling.score.protocol import RerankRequest
81
@@ -78,7 +83,6 @@ def get_invocation_types(supported_tasks: tuple["SupportedTask", ...]):
83
(RerankRequest, (rerank, do_rerank)),
84
]
85
81
- if "score" in supported_tasks or "embed" in supported_tasks:
86
from vllm.entrypoints.pooling.score.api_router import create_score, score
87
from vllm.entrypoints.pooling.score.protocol import ScoreRequest
88
@@ -97,11 +101,15 @@ def get_invocation_types(supported_tasks: tuple["SupportedTask", ...]):
101
return INVOCATION_TYPES
102
103
100
-def attach_router(app: FastAPI, supported_tasks: tuple["SupportedTask", ...]):
104
+def attach_router(
105
+ app: FastAPI,
106
+ supported_tasks: tuple["SupportedTask", ...],
107
+ model_config: ModelConfig | None = None,
108
+):
109
router = APIRouter()
110
111
# NOTE: Construct the TypeAdapters only once
104
- INVOCATION_TYPES = get_invocation_types(supported_tasks)
112
+ INVOCATION_TYPES = get_invocation_types(supported_tasks, model_config)
113
INVOCATION_VALIDATORS = [
114
(pydantic.TypeAdapter(request_type), (get_handler, endpoint))
115
for request_type, (get_handler, endpoint) in INVOCATION_TYPES
vllm/model_executor/layers/pooler/activations.py
+10
-22
@@ -16,25 +16,22 @@ from vllm.utils.import_utils import resolve_obj_by_qualname
16
logger = init_logger(__name__)
17
18
19
-def get_classification_act_fn(
19
+def get_act_fn(
20
config: PretrainedConfig,
21
+ static_num_labels: bool = True,
22
) -> "PoolerActivation":
23
+ # get classification act_fn
24
# Implement alignment with transformers ForSequenceClassificationLoss
25
# https://github.com/huggingface/transformers/blob/57bb6db6ee4cfaccc45b8d474dfad5a17811ca60/src/transformers/loss/loss_utils.py#L92
26
problem_type = getattr(config, "problem_type", "")
27
if problem_type == "regression":
28
return PoolerIdentity()
29
if problem_type == "single_label_classification":
28
- return PoolerClassify()
30
+ return PoolerClassify(static_num_labels=static_num_labels)
31
if problem_type == "multi_label_classification":
32
return PoolerMultiLabelClassify()
33
32
- return PoolerClassify()
33
-
34
-
35
-def get_cross_encoder_act_fn(
36
- config: PretrainedConfig,
37
-) -> "PoolerActivation":
34
+ # get cross_encoder act_fn
35
function_name: str | None = None
36
if (
37
hasattr(config, "sentence_transformers")
@@ -55,24 +52,16 @@ def get_cross_encoder_act_fn(
52
fn = resolve_obj_by_qualname(function_name)()
53
return PoolerActivation.wraps(fn)
54
58
- return PoolerClassify()
55
+ return PoolerClassify(static_num_labels=static_num_labels)
56
57
58
def resolve_classifier_act_fn(
59
model_config: ModelConfig,
60
static_num_labels: bool = True,
64
- act_fn: "PoolerActivation | str | None" = None,
61
+ act_fn: "PoolerActivation | None" = None,
62
):
66
- if isinstance(act_fn, str):
67
- if act_fn == "classify":
68
- return get_classification_act_fn(model_config.hf_config)
69
- if act_fn == "score":
70
- return get_cross_encoder_act_fn(model_config.hf_config)
71
-
72
- raise ValueError(f"act_fn [{act_fn=}] not supported.")
73
-
63
if act_fn is None:
75
- return PoolerClassify(static_num_labels=static_num_labels)
64
+ return get_act_fn(model_config.hf_config, static_num_labels)
65
66
assert callable(act_fn)
67
return act_fn
@@ -97,9 +86,8 @@ class PoolerActivation(nn.Module, ABC):
86
87
def forward(self, pooled_data: _T) -> _T:
88
# shape:
100
- # classify (& score) -> (batch_size, num_classes)
101
- # embed -> (batch_size, embedding_dim) or list(embedding_dim)
102
- # (batch_size, dimensions) or list(dimensions) if using MRL
89
+ # classify -> (batch_size, num_classes)
90
+ # embed -> (batch_size, embedding_size) or list(embedding_size)
91
if isinstance(pooled_data, list):
92
return [self.forward_chunk(data) for data in pooled_data]
93
vllm/model_executor/layers/pooler/seqwise/heads.py
+26
-22
@@ -56,29 +56,31 @@ class EmbeddingPoolerHead(SequencePoolerHead):
56
57
if isinstance(pooled_data, list):
58
pooled_data = torch.stack(pooled_data)
59
- # pooled_data shape: [batchsize, hidden_dimension]
59
+ # pooled_data shape: [batchsize, hidden_size]
60
61
if self.head_dtype is not None:
62
pooled_data = pooled_data.to(self.head_dtype)
63
64
# Apply ST projector
65
if self.projector is not None:
66
- pooled_data = self.projector(pooled_data)
67
- # pooled_data shape: [batchsize, embedding_dimension]
66
+ embeddings = self.projector(pooled_data)
67
+ else:
68
+ embeddings = pooled_data
69
+ # embeddings shape: [batchsize, embedding_size]
70
71
# for matryoshka representation
72
dimensions_list = [pooling_param.dimensions for pooling_param in pooling_params]
73
if any(d is not None for d in dimensions_list):
74
# change the output dimension
73
- assert len(pooled_data) == len(dimensions_list)
74
- if len(set(dimensions_list)) == 1 and not isinstance(pooled_data, list):
75
+ assert len(embeddings) == len(dimensions_list)
76
+ if len(set(dimensions_list)) == 1 and not isinstance(embeddings, list):
77
# if all dimensions are the same
78
d = dimensions_list[0]
77
- pooled_data = pooled_data[..., :d]
79
+ embeddings = embeddings[..., :d]
80
else:
79
- pooled_data = [
81
+ embeddings = [
82
vecs if d is None else vecs[..., :d]
81
- for vecs, d in zip(pooled_data, dimensions_list)
83
+ for vecs, d in zip(embeddings, dimensions_list)
84
]
85
86
# for normalize
@@ -86,15 +88,15 @@ class EmbeddingPoolerHead(SequencePoolerHead):
88
flags = [p.use_activation for p in pooling_params]
89
if len(set(flags)) == 1:
90
if flags[0]:
89
- pooled_data = self.activation(pooled_data)
91
+ embeddings = self.activation(embeddings)
92
else:
91
- pooled_data = [
93
+ embeddings = [
94
self.activation(vecs) if f else vecs
93
- for vecs, f in zip(pooled_data, flags)
95
+ for vecs, f in zip(embeddings, flags)
96
]
97
96
- # pooled_data shape: [batchsize, embedding_dimension]
97
- return pooled_data
98
+ # embeddings shape: [batchsize, embedding_size]
99
+ return embeddings
100
101
102
class ClassifierPoolerHead(SequencePoolerHead):
@@ -113,7 +115,7 @@ class ClassifierPoolerHead(SequencePoolerHead):
115
self.activation = activation
116
117
def get_supported_tasks(self) -> Set[PoolingTask]:
116
- return {"classify", "score"}
118
+ return {"classify"}
119
120
def forward(
121
self,
@@ -131,21 +133,23 @@ class ClassifierPoolerHead(SequencePoolerHead):
133
pooled_data = pooled_data.to(self.head_dtype)
134
135
if self.classifier is not None:
134
- pooled_data = self.classifier(pooled_data)
135
- # pooled_data shape: [batchsize, num_labels]
136
+ logits = self.classifier(pooled_data)
137
+ else:
138
+ logits = pooled_data
139
140
+ # logits shape: [batchsize, num_labels]
141
if self.logit_bias is not None:
138
- pooled_data -= self.logit_bias
142
+ logits -= self.logit_bias
143
144
if self.activation is not None:
145
flags = [p.use_activation for p in pooling_params]
146
if len(set(flags)) == 1:
143
- pooled_data = self.activation(pooled_data) if flags[0] else pooled_data
147
+ logits = self.activation(logits) if flags[0] else logits
148
else:
145
- pooled_data = [
149
+ logits = [
150
self.activation(vecs) if f else vecs
147
- for vecs, f in zip(pooled_data, flags)
151
+ for vecs, f in zip(logits, flags)
152
]
153
150
- # pooled_data shape: [batchsize, num_labels]
151
- return pooled_data
154
+ # logits shape: [batchsize, num_labels]
155
+ return logits
vllm/model_executor/layers/pooler/seqwise/methods.py
+1
-1
@@ -17,7 +17,7 @@ SequencePoolingMethodOutput: TypeAlias = torch.Tensor | list[torch.Tensor]
17
18
class SequencePoolingMethod(nn.Module, ABC):
19
def get_supported_tasks(self) -> Set[PoolingTask]:
20
- return {"token_embed", "token_classify", "embed", "classify", "score"}
20
+ return {"token_embed", "token_classify", "embed", "classify"}
21
22
def get_pooling_updates(self, task: PoolingTask) -> PoolingParamsUpdate:
23
return PoolingParamsUpdate()
vllm/model_executor/layers/pooler/seqwise/poolers.py
+1
-1
@@ -108,7 +108,7 @@ def pooler_for_classify(
108
*,
109
pooling: SequencePoolingMethod | SequencePoolingFn | None = None,
110
classifier: ClassifierFn | None = None,
111
- act_fn: PoolerActivation | str | None = None,
111
+ act_fn: PoolerActivation | None = None,
112
):
113
if pooling is None:
114
pooling = get_seq_pooling_method(pooler_config.get_seq_pooling_type())
vllm/model_executor/layers/pooler/special.py
+1
-8
@@ -52,13 +52,6 @@ class DispatchPooler(Pooler):
52
pooler_config,
53
pooling=pooling,
54
classifier=classifier,
55
- act_fn="classify",
56
- ),
57
- "score": pooler_for_classify(
58
- pooler_config,
59
- pooling=pooling,
60
- classifier=classifier,
61
- act_fn="score",
55
),
56
}
57
)
@@ -115,7 +108,7 @@ class DispatchPooler(Pooler):
108
109
class IdentityPooler(Pooler):
110
def get_supported_tasks(self) -> Set[PoolingTask]:
118
- return {"plugin", "score"}
111
+ return {"plugin"}
112
113
def forward(
114
self,
vllm/model_executor/layers/pooler/tokwise/heads.py
+16
-14
@@ -68,22 +68,24 @@ class TokenEmbeddingPoolerHead(TokenPoolerHead):
68
69
if self.head_dtype is not None:
70
pooled_data = pooled_data.to(self.head_dtype)
71
- # pooled_data shape: [n_tokens, hidden_dimension]
71
+ # pooled_data shape: [n_tokens, hidden_size]
72
73
# Apply ST projector
74
if self.projector is not None:
75
- pooled_data = self.projector(pooled_data)
76
- # pooled_data shape: [n_tokens, embedding_dimension]
75
+ embeddings = self.projector(pooled_data)
76
+ else:
77
+ embeddings = pooled_data
78
+ # embeddings shape: [n_tokens, embedding_size]
79
80
# for matryoshka representation
79
- pooled_data = pooled_data[..., : pooling_param.dimensions]
81
+ embeddings = embeddings[..., : pooling_param.dimensions]
82
83
# for normalize
84
if self.activation is not None and pooling_param.use_activation:
83
- pooled_data = self.activation(pooled_data)
85
+ embeddings = self.activation(embeddings)
86
85
- # pooled_data shape: [n_tokens, embedding_dimension]
86
- return pooled_data
87
+ # embeddings shape: [n_tokens, embedding_size]
88
+ return embeddings
89
90
91
class TokenClassifierPoolerHead(TokenPoolerHead):
@@ -118,16 +120,16 @@ class TokenClassifierPoolerHead(TokenPoolerHead):
120
# hidden_states shape: [n_token, hidden_size]
121
122
if self.classifier is not None:
121
- scores = self.classifier(pooled_data)
123
+ logits = self.classifier(pooled_data)
124
else:
123
- scores = pooled_data
124
- # scores shape: [n_token, num_labels]
125
+ logits = pooled_data
126
+ # logits shape: [n_token, num_labels]
127
128
if self.logit_bias is not None:
127
- scores -= self.logit_bias
129
+ logits -= self.logit_bias
130
131
if self.activation is not None and pooling_param.use_activation:
130
- scores = self.activation(scores)
132
+ logits = self.activation(logits)
133
132
- # scores shape: [n_token, num_labels]
133
- return scores
134
+ # logits shape: [n_token, num_labels]
135
+ return logits
vllm/model_executor/layers/pooler/tokwise/poolers.py
+1
-1
@@ -116,7 +116,7 @@ def pooler_for_token_classify(
116
*,
117
pooling: TokenPoolingMethod | TokenPoolingFn | None = None,
118
classifier: ClassifierFn | None = None,
119
- act_fn: PoolerActivation | str | None = None,
119
+ act_fn: PoolerActivation | None = None,
120
):
121
if pooling is None:
122
pooling = get_tok_pooling_method(pooler_config.get_tok_pooling_type())
vllm/model_executor/models/interfaces_base.py
+7
-7
@@ -194,18 +194,18 @@ class VllmModelForPooling(VllmModel[T_co], Protocol[T_co]):
194
[vllm.config.model.ModelConfig.score_type][]
195
to use by default.
196
197
- Score API handles score/rerank for:
198
- - "score" task (score_type: cross-encoder models)
199
- - "embed" task (score_type: bi-encoder models)
200
- - "token_embed" task (score_type: late interaction models)
197
+ Scoring API handles score/rerank for:\n
198
+ - "classify" task (score_type: cross-encoder models)\n
199
+ - "embed" task (score_type: bi-encoder models)\n
200
+ - "token_embed" task (score_type: late interaction models)\n
201
202
- score_type defaults to bi-encoder, then the Score API uses the "embed" task.
202
+ score_type defaults to bi-encoder, then the Score API uses the "embed" task.\n
203
If you set score_type to cross-encoder via
204
[vllm.model_executor.models.interfaces.SupportsCrossEncoding][],
205
- then the Score API uses the "score" task.
205
+ then the Score API uses the "score" task.\n
206
If you set score_type to late-interaction via
207
[vllm.model_executor.models.interfaces.SupportsLateInteraction][],
208
- then the Score API uses the "token_embed" task.
208
+ then the Score API uses the "token_embed" task.\n
209
"""
210
211
pooler: Pooler
vllm/pooling_params.py
+11
-6
@@ -7,9 +7,12 @@ from typing import Any
7
import msgspec
8
9
from vllm.config import ModelConfig, PoolerConfig
10
+from vllm.logger import init_logger
11
from vllm.sampling_params import RequestOutputKind
12
from vllm.tasks import PoolingTask
13
14
+logger = init_logger(__name__)
15
+
16
17
class LateInteractionParams(
18
msgspec.Struct,
@@ -54,10 +57,6 @@ class PoolingParams(
57
dimensions: int | None = None
58
# --8<-- [end:embed-pooling-params]
59
57
- ## for classification, scoring and rerank
58
- # --8<-- [start:classify-pooling-params]
59
- # --8<-- [end:classify-pooling-params]
60
-
60
## for step pooling models
61
step_tag_id: int | None = None
62
returned_token_ids: list[int] | None = None
@@ -79,7 +78,6 @@ class PoolingParams(
78
return {
79
"embed": ["dimensions", "use_activation"],
80
"classify": ["use_activation"],
82
- "score": ["use_activation"],
81
"token_embed": ["dimensions", "use_activation"],
82
"token_classify": ["use_activation"],
83
}
@@ -89,6 +87,13 @@ class PoolingParams(
87
return deepcopy(self)
88
89
def verify(self, model_config: ModelConfig) -> None:
90
+ if self.task == "score":
91
+ logger.warning_once(
92
+ "`score` task is deprecated and will be removed in v0.20. "
93
+ "Please use `classify` instead."
94
+ )
95
+ self.task = "classify"
96
+
97
# plugin task uses io_processor.parse_request to verify inputs,
98
# skipping PoolingParams verify
99
if self.task == "plugin":
@@ -184,7 +189,7 @@ class PoolingParams(
189
elif self.dimensions < 1:
190
raise ValueError("Dimensions must be greater than 0")
191
187
- elif self.task in ["classify", "score", "token_classify"]:
192
+ elif self.task in ["classify", "token_classify"]:
193
if self.use_activation is None:
194
self.use_activation = True
195
else:
vllm/tasks.py
-5
@@ -8,7 +8,6 @@ GENERATION_TASKS: tuple[GenerationTask, ...] = get_args(GenerationTask)
8
PoolingTask = Literal[
9
"embed",
10
"classify",
11
- "score",
11
"token_embed",
12
"token_classify",
13
"plugin",
@@ -16,10 +15,6 @@ PoolingTask = Literal[
15
]
16
POOLING_TASKS: tuple[PoolingTask, ...] = get_args(PoolingTask)
17
19
-# Score API handles score/rerank for:
20
-# - "score" task (score_type: cross-encoder models)
21
-# - "embed" task (score_type: bi-encoder models)
22
-# - "token_embed" task (score_type: late interaction models)
18
ScoreType = Literal["bi-encoder", "cross-encoder", "late-interaction"]
19
20
FrontendTask = Literal["render"]
vllm/v1/worker/gpu_model_runner.py
+1
-9
@@ -2834,15 +2834,7 @@ class GPUModelRunner(
2834
if not is_pooling_model(model):
2835
return []
2836
2837
- supported_tasks = list(model.pooler.get_supported_tasks())
2838
-
2839
- if "score" in supported_tasks:
2840
- num_labels = getattr(self.model_config.hf_config, "num_labels", 0)
2841
- if num_labels != 1:
2842
- supported_tasks.remove("score")
2843
- logger.debug_once("Score API is only enabled for num_labels == 1.")
2844
-
2845
- return supported_tasks
2837
+ return list(model.pooler.get_supported_tasks())
2838
2839
def get_supported_tasks(self) -> tuple[SupportedTask, ...]:
2840
tasks = list[SupportedTask]()