3650a74ed8 · tduncan/Nvidia-Optimization-Stack-VLLM

@@ -1,15 +1,15 @@

1 # Pooling Models

2

3 -vLLM also supports pooling models, such as embedding, classification and reward models.

3 +vLLM also supports pooling models, such as embedding, classification, and reward models.

4

5 In vLLM, pooling models implement the [VllmModelForPooling][vllm.model_executor.models.VllmModelForPooling] interface.

6 These models use a [Pooler][vllm.model_executor.layers.pooler.Pooler] to extract the final hidden states of the input

7 before returning them.

8

9 !!! note

10 - We currently support pooling models primarily as a matter of convenience. This is not guaranteed to have any performance improvement over using HF Transformers / Sentence Transformers directly.

10 + We currently support pooling models primarily for convenience. This is not guaranteed to provide any performance improvements over using Hugging Face Transformers or Sentence Transformers directly.

11

12 - We are now planning to optimize pooling models in vLLM. Please comment on <https://github.com/vllm-project/vllm/issues/21796> if you have any suggestions!

12 + We plan to optimize pooling models in vLLM. Please comment on <https://github.com/vllm-project/vllm/issues/21796> if you have any suggestions!

13

14 ## Configuration

15

@@ -19,7 +19,7 @@ Run a model in pooling mode via the option `--runner pooling`.

19

20 !!! tip

21 There is no need to set this option in the vast majority of cases as vLLM can automatically

22 - detect the model runner to use via `--runner auto`.

22 + detect the appropriate model runner via `--runner auto`.

23

24 ### Model Conversion

25

@@ -78,7 +78,7 @@ When loading [Sentence Transformers](https://huggingface.co/sentence-transformer

78 its Sentence Transformers configuration file (`modules.json`) takes priority over the model's defaults.

79

80 You can further customize this via the `--pooler-config` option,

81 -which takes priority over both the model's and Sentence Transformers's defaults.

81 +which takes priority over both the model's and Sentence Transformers' defaults.

82

83 ## Offline Inference

84

@@ -168,11 +168,11 @@ The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.

168

169 - For embeddings, use `LLM.embed(...)` or `pooling_task="embed"`.

170 - For classification logits, use `LLM.classify(...)` or `pooling_task="classify"`.

171 - - For similarity scores, use `LLM.score(...)`.

171 + - For similarity scores, use `LLM.score(...)`.

172 - For rewards, use `LLM.reward(...)` or `pooling_task="token_classify"`.

173 - For token classification, use `pooling_task="token_classify"`.

174 - - For multi-vector retrieval, use `pooling_task="token_embed"`

175 - - For IO Processor Plugins , use `pooling_task="plugin"`

174 + - For multi-vector retrieval, use `pooling_task="token_embed"`.

175 + - For IO Processor Plugins, use `pooling_task="plugin"`.

176

177 ```python

178 from vllm import LLM

@@ -194,15 +194,15 @@ Our [OpenAI-Compatible Server](../serving/openai_compatible_server.md) provides

194 - [Pooling API](../serving/openai_compatible_server.md#pooling-api) is similar to `LLM.encode`, being applicable to all types of pooling models.

195

196 !!! note

197 - Please use one of the more specific methods or set the task directly when using [Pooling API](../serving/openai_compatible_server.md#pooling-api) api.:

197 + Please use one of the more specific endpoints or set the task directly when using the [Pooling API](../serving/openai_compatible_server.md#pooling-api):

198

199 - For embeddings, use [Embeddings API](../serving/openai_compatible_server.md#embeddings-api) or `"task":"embed"`.

200 - - For classification logits, use [Classification API](../serving/openai_compatible_server.md#classification-api) or `task":"classify"`.

201 - - For similarity scores, use [Score API](../serving/openai_compatible_server.md#score-api).

202 - - For rewards, `task":"token_classify"`.

203 - - For token classification, use `task":"token_classify"`.

204 - - For multi-vector retrieval, use `task":"token_embed"`

205 - - For IO Processor Plugins , use `task":"plugin"`

200 + - For classification logits, use [Classification API](../serving/openai_compatible_server.md#classification-api) or `"task":"classify"`.

201 + - For similarity scores, use [Score API](../serving/openai_compatible_server.md#score-api).

202 + - For rewards, use `"task":"token_classify"`.

203 + - For token classification, use `"task":"token_classify"`.

204 + - For multi-vector retrieval, use `"task":"token_embed"`.

205 + - For IO Processor Plugins, use `"task":"plugin"`.

206

207 ```python

208 # start a supported embeddings model server with `vllm serve`, e.g.

@@ -232,7 +232,7 @@ for output in response.json()["data"]:

232

233 ## Matryoshka Embeddings

234

235 -[Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows user to trade off between performance and cost.

235 +[Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows users to trade off between performance and cost.

236

237 !!! warning

238 Not all embedding models are trained using Matryoshka Representation Learning. To avoid misuse of the `dimensions` parameter, vLLM returns an error for requests that attempt to change the output dimension of models that do not support Matryoshka Embeddings.

@@ -245,9 +245,9 @@ for output in response.json()["data"]:

245

246 ### Manually enable Matryoshka Embeddings

247

248 -There is currently no official interface for specifying support for Matryoshka Embeddings. In vLLM, if `is_matryoshka` is `True` in `config.json,` it is allowed to change the output to arbitrary dimensions. Using `matryoshka_dimensions` can control the allowed output dimensions.

248 +There is currently no official interface for specifying support for Matryoshka Embeddings. In vLLM, if `is_matryoshka` is `True` in `config.json`, you can change the output dimension to arbitrary values. Use `matryoshka_dimensions` to control the allowed output dimensions.

249

250 -For models that support Matryoshka Embeddings but not recognized by vLLM, please manually override the config using `hf_overrides={"is_matryoshka": True}`, `hf_overrides={"matryoshka_dimensions": [<allowed output dimensions>]}` (offline) or `--hf-overrides '{"is_matryoshka": true}'`, `--hf-overrides '{"matryoshka_dimensions": [<allowed output dimensions>]}'`(online).

250 +For models that support Matryoshka Embeddings but are not recognized by vLLM, manually override the config using `hf_overrides={"is_matryoshka": True}` or `hf_overrides={"matryoshka_dimensions": [<allowed output dimensions>]}` (offline), or `--hf-overrides '{"is_matryoshka": true}'` or `--hf-overrides '{"matryoshka_dimensions": [<allowed output dimensions>]}'` (online).

251

252 Here is an example to serve a model with Matryoshka Embeddings enabled.

253

@@ -278,7 +278,7 @@ A code example can be found here: [examples/offline_inference/pooling/embed_matr

278

279 ### Online Inference

280

281 -Use the following command to start vllm server.

281 +Use the following command to start the vLLM server.

282

283 ```bash

284 vllm serve jinaai/jina-embeddings-v3 --trust-remote-code

@@ -310,11 +310,11 @@ An OpenAI client example can be found here: [examples/online_serving/pooling/ope

310

311 ### Encode task

312

313 -We have split the `encode` task into two more specific token wise tasks: `token_embed` and `token_classify`:

313 +We have split the `encode` task into two more specific token-wise tasks: `token_embed` and `token_classify`:

314

315 -- `token_embed` is the same as embed, using normalize as activation.

316 -- `token_classify` is the same as classify, default using softmax as activation.

315 +- `token_embed` is the same as `embed`, using normalization as the activation.

316 +- `token_classify` is the same as `classify`, by default using softmax as the activation.

317

318 ### Remove softmax from PoolingParams

319

320 -We are going to remove `softmax` and `activation` from `PoolingParams`. Instead, you should set `use_activation`, since we actually allow `classify` and `token_classify` to use any activation function.

320 +We are going to remove `softmax` and `activation` from `PoolingParams`. Instead, use `use_activation`, since we allow `classify` and `token_classify` to use any activation function.

Optimize the wording of the document and unify the terminology and th… (#29491)