@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 300 lines 10 KB
Rendered Raw
1 # Classification Usages
2
3 Classification involves predicting which predefined category, class, or label best corresponds to a given input.
4
5 ## Summary
6
7 - Model Usage: (sequence) classification
8 - Pooling Task: `classify`
9 - Offline APIs:
10 - `LLM.classify(...)`
11 - `LLM.encode(..., pooling_task="classify")`
12 - Online APIs:
13 - [Classification API](classify.md#online-serving) (`/classify`)
14 - Pooling API (`/pooling`)
15
16 The key distinction between (sequence) classification and token classification lies in their output granularity: (sequence) classification produces a single result for an entire input sequence, whereas token classification yields a result for each individual token within the sequence.
17
18 Many classification models support both (sequence) classification and token classification. For further details on token classification, please refer to [this page](token_classify.md).
19
20 Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled, please refer to [this page](scoring.md).
21
22 ## Typical Use Cases
23
24 ### Classification
25
26 The most fundamental application of classification models is to categorize input data into predefined classes.
27
28 ## Supported Models
29
30 ### Text-only Models
31
32 | Architecture | Models | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
33 | ------------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |
34 | `GPT2ForSequenceClassification` | GPT2 | `nie3e/sentiment-polish-gpt2-small` | | |
35 | `Qwen2ForSequenceClassification`<sup>C</sup> | Qwen2-based | `jason9693/Qwen2.5-1.5B-apeach` | | |
36 | `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* |
37
38 ### Multimodal Models
39
40 !!! note
41 For more information about multimodal models inputs, see [this page](../supported_models.md#list-of-multimodal-language-models).
42
43 | Architecture | Models | Inputs | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
44 | ------------ | ------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |
45 | `Qwen2_5_VLForSequenceClassification`<sup>C</sup> | Qwen2_5_VL-based | T + I<sup>E+</sup> + V<sup>E+</sup> | `muziyongshixin/Qwen2.5-VL-7B-for-VideoCls` | | |
46 | `*ForConditionalGeneration`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | \* | N/A | \* | \* |
47
48 <sup>C</sup> Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))
49 \* Feature support is the same as that of the original model.
50
51 If your model is not in the above list, we will try to automatically convert the model using
52 [as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.
53
54 ### Cross-encoder Models
55
56 Cross-encoder (aka reranker) models are a subset of classification models that accept two prompts as input and output num_labels equal to 1. Most classification models can also be used as [cross-encoder models](scoring.md#cross-encoder-models). For more information on cross-encoder models, please refer to [this page](scoring.md).
57
58 --8<-- "docs/models/pooling_models/scoring.md:supported-cross-encoder-models"
59
60 ### Reward Models
61
62 Using (sequence) classification models as reward models. For more information, see [Reward Models](reward.md).
63
64 --8<-- "docs/models/pooling_models/reward.md:supported-sequence-reward-models"
65
66 ## Offline Inference
67
68 ### Pooling Parameters
69
70 The following [pooling parameters][vllm.PoolingParams] are supported.
71
72 ```python
73 --8<-- "vllm/pooling_params.py:common-pooling-params"
74 --8<-- "vllm/pooling_params.py:classify-pooling-params"
75 ```
76
77 ### `LLM.classify`
78
79 The [classify][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.classify] method outputs a probability vector for each prompt.
80
81 ```python
82 from vllm import LLM
83
84 llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", runner="pooling")
85 (output,) = llm.classify("Hello, my name is")
86
87 probs = output.outputs.probs
88 print(f"Class Probabilities: {probs!r} (size={len(probs)})")
89 ```
90
91 A code example can be found here: [examples/basic/offline_inference/classify.py](../../../examples/basic/offline_inference/classify.py)
92
93 ### `LLM.encode`
94
95 The [encode][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.encode] method is available to all pooling models in vLLM.
96
97 Set `pooling_task="classify"` when using `LLM.encode` for classification Models:
98
99 ```python
100 from vllm import LLM
101
102 llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", runner="pooling")
103 (output,) = llm.encode("Hello, my name is", pooling_task="classify")
104
105 data = output.outputs.data
106 print(f"Data: {data!r}")
107 ```
108
109 ## Online Serving
110
111 ### Classification API
112
113 Online `/classify` API is similar to `LLM.classify`.
114
115 #### Completion Parameters
116
117 The following Classification API parameters are supported:
118
119 ??? code
120
121 ```python
122 --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"
123 --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-params"
124 --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-params"
125 ```
126
127 The following extra parameters are supported:
128
129 ??? code
130
131 ```python
132 --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"
133 --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-extra-params"
134 --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"
135 ```
136
137 #### Chat Parameters
138
139 For chat-like input (i.e. if `messages` is passed), the following parameters are supported:
140
141 ??? code
142
143 ```python
144 --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"
145 --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-params"
146 --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-params"
147 ```
148
149 these extra parameters are supported instead:
150
151 ??? code
152
153 ```python
154 --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"
155 --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-extra-params"
156 --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"
157 ```
158
159 #### Example Requests
160
161 Code example: [examples/pooling/classify/classification_online.py](../../../examples/pooling/classify/classification_online.py)
162
163 You can classify multiple texts by passing an array of strings:
164
165 ```bash
166 curl -v "http://127.0.0.1:8000/classify" \
167 -H "Content-Type: application/json" \
168 -d '{
169 "model": "jason9693/Qwen2.5-1.5B-apeach",
170 "input": [
171 "Loved the new café—coffee was great.",
172 "This update broke everything. Frustrating."
173 ]
174 }'
175 ```
176
177 ??? console "Response"
178
179 ```json
180 {
181 "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",
182 "object": "list",
183 "created": 1745383065,
184 "model": "jason9693/Qwen2.5-1.5B-apeach",
185 "data": [
186 {
187 "index": 0,
188 "label": "Default",
189 "probs": [
190 0.565970778465271,
191 0.4340292513370514
192 ],
193 "num_classes": 2
194 },
195 {
196 "index": 1,
197 "label": "Spoiled",
198 "probs": [
199 0.26448777318000793,
200 0.7355121970176697
201 ],
202 "num_classes": 2
203 }
204 ],
205 "usage": {
206 "prompt_tokens": 20,
207 "total_tokens": 20,
208 "completion_tokens": 0,
209 "prompt_tokens_details": null
210 }
211 }
212 ```
213
214 You can also pass a string directly to the `input` field:
215
216 ```bash
217 curl -v "http://127.0.0.1:8000/classify" \
218 -H "Content-Type: application/json" \
219 -d '{
220 "model": "jason9693/Qwen2.5-1.5B-apeach",
221 "input": "Loved the new café—coffee was great."
222 }'
223 ```
224
225 ??? console "Response"
226
227 ```json
228 {
229 "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
230 "object": "list",
231 "created": 1745383213,
232 "model": "jason9693/Qwen2.5-1.5B-apeach",
233 "data": [
234 {
235 "index": 0,
236 "label": "Default",
237 "probs": [
238 0.565970778465271,
239 0.4340292513370514
240 ],
241 "num_classes": 2
242 }
243 ],
244 "usage": {
245 "prompt_tokens": 10,
246 "total_tokens": 10,
247 "completion_tokens": 0,
248 "prompt_tokens_details": null
249 }
250 }
251 ```
252
253 ## More examples
254
255 More examples can be found here: [examples/pooling/classify](../../../examples/pooling/classify)
256
257 ## Supported Features
258
259 ### Enable/disable activation
260
261 You can enable or disable activation via `use_activation`.
262
263 ### Problem type (e.g. `multi_label_classification`)
264
265 You can modify the `problem_type` via problem_type in the Hugging Face config. The supported problem types are: `single_label_classification`, `multi_label_classification`, and `regression`.
266
267 Implement alignment with transformers [ForSequenceClassificationLoss](https://github.com/huggingface/transformers/blob/57bb6db6ee4cfaccc45b8d474dfad5a17811ca60/src/transformers/loss/loss_utils.py#L92).
268
269 ### Affine Score Calibration
270
271 Affine Score Calibration, also known as [Platt Scaling](https://en.wikipedia.org/wiki/Platt_scaling) (Platt, 1999), is the most widely used method for calibrating classifier outputs into well-calibrated probabilities.
272
273 The calibration follows the transformation:
274
275 `activation((logit - logit_mean) / logit_sigma)`
276
277 | Parameter | Default | Description |
278 | --------- | ------- | ----------- |
279 | `logit_mean` | `None` | Mean subtracted from logits (centers scores) |
280 | `logit_sigma` | `None` | Standard deviation used to scale logits after mean subtraction |
281
282 The computation order is as follows:
283
284 ```python
285 logits -= logit_mean # subtract mean (center scores)
286 logits /= logit_sigma # divide by sigma (scale)
287 logits = activation(logits) # e.g. sigmoid
288 ```
289
290 Example configuration:
291
292 ```bash
293 --pooler-config '{"use_activation": true, "logit_mean": 4.5, "logit_sigma": 1.0}'
294 ```
295
296 ## Removed Features
297
298 ### Remove softmax from PoolingParams
299
300 We have already removed `softmax` and `activation` from PoolingParams. Instead, use `use_activation`, since we allow `classify` and `token_classify` to use any activation function.