| 1 | # Embedding Usages |
| 2 | |
| 3 | Embedding models are a class of machine learning models designed to transform unstructured data—such as text, images, or audio—into a structured numerical representation known as an embedding. |
| 4 | |
| 5 | ## Summary |
| 6 | |
| 7 | - Model Usage: (sequence) embedding |
| 8 | - Pooling Task: `embed` |
| 9 | - Offline APIs: |
| 10 | - `LLM.embed(...)` |
| 11 | - `LLM.encode(..., pooling_task="embed")` |
| 12 | - `LLM.score(...)` |
| 13 | - Online APIs: |
| 14 | - [Cohere Embed API](embed.md#cohere-embed-api) (`/v2/embed`) |
| 15 | - [OpenAI-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) |
| 16 | - Pooling API (`/pooling`) |
| 17 | |
| 18 | The primary distinction between (sequence) embedding and token embedding lies in their output granularity: (sequence) embedding produces a single embedding vector for an entire input sequence, whereas token embedding generates an embedding for each individual token within the sequence. |
| 19 | |
| 20 | Many embedding models support both (sequence) embedding and token embedding. For further details on token embedding, please refer to [this page](token_embed.md). |
| 21 | |
| 22 | ## Typical Use Cases |
| 23 | |
| 24 | ### Embedding |
| 25 | |
| 26 | The most basic use case of embedding models is to embed the inputs, e.g. for RAG. |
| 27 | |
| 28 | ### Pairwise Similarity |
| 29 | |
| 30 | You can compute pairwise similarity scores to build a similarity matrix using the [Score API](scoring.md). |
| 31 | |
| 32 | ## Supported Models |
| 33 | |
| 34 | --8<-- [start:supported-embed-models] |
| 35 | |
| 36 | ### Text-only Models |
| 37 | |
| 38 | | Architecture | Models | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) | |
| 39 | | ------------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ | |
| 40 | | `BertModel` | BERT-based | `BAAI/bge-base-en-v1.5`, `Snowflake/snowflake-arctic-embed-xs`, etc. | | | |
| 41 | | `BertSpladeSparseEmbeddingModel` | SPLADE | `naver/splade-v3` | | | |
| 42 | | `BgeM3EmbeddingModel` | BGE-M3 | `BAAI/bge-m3` | | | |
| 43 | | `Gemma2Model`<sup>C</sup> | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | ✅︎ | |
| 44 | | `Gemma3TextModel`<sup>C</sup> | Gemma 3-based | `google/embeddinggemma-300m`, etc. | ✅︎ | ✅︎ | |
| 45 | | `GteModel` | Arctic-Embed-2.0-M | `Snowflake/snowflake-arctic-embed-m-v2.0`. | | | |
| 46 | | `GteNewModel` | mGTE-TRM (see note) | `Alibaba-NLP/gte-multilingual-base`, etc. | | | |
| 47 | | `JinaEmbeddingsV5Model`<sup>C</sup> | Qwen3-decoder or EuroBERT-encoder backbone with task-specific LoRA adapters | `jinaai/jina-embeddings-v5-text-small`, `jinaai/jina-embeddings-v5-text-nano` (see note) | ✅︎ | ✅︎ | |
| 48 | | `LlamaBidirectionalModel`<sup>C</sup> | Llama-based with bidirectional attention | `nvidia/llama-nemotron-embed-1b-v2`, etc. | ✅︎ | ✅︎ | |
| 49 | | `LlamaModel`<sup>C</sup>, `LlamaForCausalLM`<sup>C</sup>, `MistralModel`<sup>C</sup>, etc. | Llama-based | `intfloat/e5-mistral-7b-instruct`, etc. | ✅︎ | ✅︎ | |
| 50 | | `ModernBertModel` | ModernBERT-based | `Alibaba-NLP/gte-modernbert-base`, etc. | ✅︎ | | |
| 51 | | `NomicBertModel` | Nomic BERT | `nomic-ai/nomic-embed-text-v1`, `nomic-ai/nomic-embed-text-v2-moe`, `Snowflake/snowflake-arctic-embed-m-long`, etc. | | | |
| 52 | | `Qwen2Model`<sup>C</sup>, `Qwen2ForCausalLM`<sup>C</sup> | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ | |
| 53 | | `Qwen3Model`<sup>C</sup>, `Qwen3ForCausalLM`<sup>C</sup> | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, etc. | ✅︎ | ✅︎ | |
| 54 | | `RobertaModel`, `RobertaForMaskedLM` | RoBERTa-based | `sentence-transformers/all-roberta-large-v1`, etc. | | | |
| 55 | | `VoyageQwen3BidirectionalEmbedModel`<sup>C</sup> | Voyage Qwen3-based with bidirectional attention | `voyageai/voyage-4-nano`, etc. | ✅︎ | ✅︎ | |
| 56 | | `XLMRobertaModel` | XLMRobertaModel-based | `BAAI/bge-m3` (see note), `intfloat/multilingual-e5-base`, `jinaai/jina-embeddings-v3` (see note), etc. | | | |
| 57 | | `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* | |
| 58 | |
| 59 | !!! note |
| 60 | The second-generation GTE model (mGTE-TRM) is named `NewModel`. The name `NewModel` is too generic, you should set `--hf-overrides '{"architectures": ["GteNewModel"]}'` to specify the use of the `GteNewModel` architecture. |
| 61 | |
| 62 | !!! note |
| 63 | `ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config. |
| 64 | You need to manually set mean pooling by passing `--pooler-config '{"pooling_type": "MEAN"}'`. |
| 65 | |
| 66 | !!! note |
| 67 | For `Alibaba-NLP/gte-Qwen2-*`, you need to enable `--trust-remote-code` for the correct tokenizer to be loaded. |
| 68 | See [relevant issue on HF Transformers](https://github.com/huggingface/transformers/issues/34882). |
| 69 | |
| 70 | !!! note |
| 71 | The `BAAI/bge-m3` model comes with extra weights for sparse and colbert embeddings, See [this page](specific_models.md#baaibge-m3) for more information. |
| 72 | |
| 73 | !!! note |
| 74 | `jinaai/jina-embeddings-v3` supports multiple tasks through LoRA, while vllm temporarily only supports text-matching tasks by merging LoRA weights. |
| 75 | |
| 76 | !!! note |
| 77 | `jinaai/jina-embeddings-v5-text-small` (Qwen3 decoder) and |
| 78 | `jinaai/jina-embeddings-v5-text-nano` (bidirectional EuroBERT encoder, |
| 79 | `is_decoder=false`) ship with four task-specific LoRA adapters |
| 80 | (`retrieval`, `text-matching`, `classification`, `clustering`). vLLM merges the |
| 81 | selected adapter into the base weights at load time. Choose the task with |
| 82 | `--hf-overrides '{"jina_task": "<task>"}'`; the default is `retrieval`. |
| 83 | |
| 84 | ### Multimodal Models |
| 85 | |
| 86 | !!! note |
| 87 | For more information about multimodal models inputs, see [this page](../supported_models.md#list-of-multimodal-language-models). |
| 88 | |
| 89 | | Architecture | Models | Inputs | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) | |
| 90 | | ------------ | ------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ | |
| 91 | | `CLIPModel` | CLIP | T / I | `openai/clip-vit-base-patch32`, `openai/clip-vit-large-patch14`, etc. | | | |
| 92 | | `LlamaNemotronVLModel` | Llama Nemotron Embedding + SigLIP | T + I | `nvidia/llama-nemotron-embed-vl-1b-v2` | | | |
| 93 | | `LlavaNextForConditionalGeneration`<sup>C</sup> | LLaVA-NeXT-based | T / I | `royokong/e5-v` | | ✅︎ | |
| 94 | | `Phi3VForCausalLM`<sup>C</sup> | Phi-3-Vision-based | T + I | `TIGER-Lab/VLM2Vec-Full` | | ✅︎ | |
| 95 | | `Qwen3VLForConditionalGeneration`<sup>C</sup> (see note) | Qwen3-VL | T + I + V | `Qwen/Qwen3-VL-Embedding-2B`, etc. | ✅︎ | ✅︎ | |
| 96 | | `SiglipModel` | SigLIP, SigLIP2 | T / I | `google/siglip-base-patch16-224`, `google/siglip2-base-patch16-224` | | | |
| 97 | | `*ForConditionalGeneration`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | \* | N/A | \* | \* | |
| 98 | |
| 99 | <sup>C</sup> Automatically converted into an embedding model via `--convert embed`. ([details](./README.md#model-conversion)) |
| 100 | \* Feature support is the same as that of the original model. |
| 101 | |
| 102 | If your model is not in the above list, we will try to automatically convert the model using |
| 103 | [as_embedding_model][vllm.model_executor.models.adapters.as_embedding_model]. By default, the embeddings |
| 104 | of the whole prompt are extracted from the normalized hidden state corresponding to the last token. |
| 105 | |
| 106 | !!! note |
| 107 | `Qwen3-VL-Embedding` officially uses `qwen_vl_utils` for image preprocessing, while vLLM uses `transformers`' `video_processing_qwen3_vl`, which leads to slightly different results compared to the official Hugging Face repository examples. Example code for offline inference using `qwen_vl_utils` can be found in the [vision_embedding_offline.py](../../../examples/pooling/embed/vision_embedding_offline.py) example. |
| 108 | |
| 109 | !!! note |
| 110 | Although vLLM supports automatically converting models of any architecture into embedding models via --convert embed, to get the best results, you should use pooling models that are specifically trained as such. |
| 111 | |
| 112 | --8<-- [end:supported-embed-models] |
| 113 | |
| 114 | ## Offline Inference |
| 115 | |
| 116 | ### Pooling Parameters |
| 117 | |
| 118 | The following [pooling parameters][vllm.PoolingParams] are supported. |
| 119 | |
| 120 | ```python |
| 121 | --8<-- "vllm/pooling_params.py:common-pooling-params" |
| 122 | --8<-- "vllm/pooling_params.py:embed-pooling-params" |
| 123 | ``` |
| 124 | |
| 125 | ### `LLM.embed` |
| 126 | |
| 127 | The [embed][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.embed] method outputs an embedding vector for each prompt. |
| 128 | |
| 129 | ```python |
| 130 | from vllm import LLM |
| 131 | |
| 132 | llm = LLM(model="intfloat/e5-small", runner="pooling") |
| 133 | (output,) = llm.embed("Hello, my name is") |
| 134 | |
| 135 | embeds = output.outputs.embedding |
| 136 | print(f"Embeddings: {embeds!r} (size={len(embeds)})") |
| 137 | ``` |
| 138 | |
| 139 | A code example can be found here: [examples/basic/offline_inference/embed.py](../../../examples/basic/offline_inference/embed.py) |
| 140 | |
| 141 | ### `LLM.encode` |
| 142 | |
| 143 | The [encode][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.encode] method is available to all pooling models in vLLM. |
| 144 | |
| 145 | Set `pooling_task="embed"` when using `LLM.encode` for embedding Models: |
| 146 | |
| 147 | ```python |
| 148 | from vllm import LLM |
| 149 | |
| 150 | llm = LLM(model="intfloat/e5-small", runner="pooling") |
| 151 | (output,) = llm.encode("Hello, my name is", pooling_task="embed") |
| 152 | |
| 153 | data = output.outputs.data |
| 154 | print(f"Data: {data!r}") |
| 155 | ``` |
| 156 | |
| 157 | ### `LLM.score` |
| 158 | |
| 159 | The [score][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.score] method outputs similarity scores between sentence pairs. |
| 160 | |
| 161 | All models that support embedding task also support using the score API to compute similarity scores by calculating the cosine similarity of two input prompt's embeddings. |
| 162 | |
| 163 | ```python |
| 164 | from vllm import LLM |
| 165 | |
| 166 | llm = LLM(model="intfloat/e5-small", runner="pooling") |
| 167 | (output,) = llm.score( |
| 168 | "What is the capital of France?", |
| 169 | "The capital of Brazil is Brasilia.", |
| 170 | ) |
| 171 | |
| 172 | score = output.outputs.score |
| 173 | print(f"Score: {score}") |
| 174 | ``` |
| 175 | |
| 176 | ## Online Serving |
| 177 | |
| 178 | ### OpenAI-Compatible Embeddings API |
| 179 | |
| 180 | Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings); |
| 181 | you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. |
| 182 | |
| 183 | Code example: [examples/pooling/embed/openai_embedding_client.py](../../../examples/pooling/embed/openai_embedding_client.py) |
| 184 | |
| 185 | #### Completion Parameters |
| 186 | |
| 187 | The following Classification API parameters are supported: |
| 188 | |
| 189 | ??? code |
| 190 | |
| 191 | ```python |
| 192 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params" |
| 193 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-params" |
| 194 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-params" |
| 195 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-params" |
| 196 | ``` |
| 197 | |
| 198 | The following extra parameters are supported: |
| 199 | |
| 200 | ??? code |
| 201 | |
| 202 | ```python |
| 203 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params" |
| 204 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-extra-params" |
| 205 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-extra-params" |
| 206 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-extra-params" |
| 207 | ``` |
| 208 | |
| 209 | #### Chat Parameters |
| 210 | |
| 211 | For chat-like input (i.e. if `messages` is passed), the following parameters are supported: |
| 212 | |
| 213 | ??? code |
| 214 | |
| 215 | ```python |
| 216 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params" |
| 217 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-params" |
| 218 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-params" |
| 219 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-params" |
| 220 | ``` |
| 221 | |
| 222 | these extra parameters are supported instead: |
| 223 | |
| 224 | ??? code |
| 225 | |
| 226 | ```python |
| 227 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params" |
| 228 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-extra-params" |
| 229 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-extra-params" |
| 230 | --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-extra-params" |
| 231 | ``` |
| 232 | |
| 233 | #### Examples |
| 234 | |
| 235 | If the model has a [chat template](../../serving/online_serving/README.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](../../serving/online_serving/openai_compatible_server.md#chat-api)) |
| 236 | which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations: |
| 237 | |
| 238 | ??? code |
| 239 | |
| 240 | ```python |
| 241 | from openai import OpenAI |
| 242 | from openai._types import NOT_GIVEN, NotGiven |
| 243 | from openai.types.chat import ChatCompletionMessageParam |
| 244 | from openai.types.create_embedding_response import CreateEmbeddingResponse |
| 245 | |
| 246 | def create_chat_embeddings( |
| 247 | client: OpenAI, |
| 248 | *, |
| 249 | messages: list[ChatCompletionMessageParam], |
| 250 | model: str, |
| 251 | encoding_format: Union[Literal["base64", "float"], NotGiven] = NOT_GIVEN, |
| 252 | ) -> CreateEmbeddingResponse: |
| 253 | return client.post( |
| 254 | "/embeddings", |
| 255 | cast_to=CreateEmbeddingResponse, |
| 256 | body={"messages": messages, "model": model, "encoding_format": encoding_format}, |
| 257 | ) |
| 258 | ``` |
| 259 | |
| 260 | ##### Multi-modal inputs |
| 261 | |
| 262 | You can pass multi-modal inputs to embedding models by defining a custom chat template for the server |
| 263 | and passing a list of `messages` in the request. Refer to the examples below for illustration. |
| 264 | |
| 265 | === "VLM2Vec" |
| 266 | |
| 267 | To serve the model: |
| 268 | |
| 269 | ```bash |
| 270 | vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \ |
| 271 | --trust-remote-code \ |
| 272 | --max-model-len 4096 \ |
| 273 | --chat-template examples/pooling/embed/template/vlm2vec_phi3v.jinja |
| 274 | ``` |
| 275 | |
| 276 | !!! important |
| 277 | Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--runner pooling` |
| 278 | to run this model in embedding mode instead of text generation mode. |
| 279 | |
| 280 | The custom chat template is completely different from the original one for this model, |
| 281 | and can be found here: [examples/pooling/embed/template/vlm2vec_phi3v.jinja](../../../examples/pooling/embed/template/vlm2vec_phi3v.jinja) |
| 282 | |
| 283 | Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library: |
| 284 | |
| 285 | ??? code |
| 286 | |
| 287 | ```python |
| 288 | from openai import OpenAI |
| 289 | client = OpenAI( |
| 290 | base_url="http://localhost:8000/v1", |
| 291 | api_key="EMPTY", |
| 292 | ) |
| 293 | image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" |
| 294 | |
| 295 | response = create_chat_embeddings( |
| 296 | client, |
| 297 | model="TIGER-Lab/VLM2Vec-Full", |
| 298 | messages=[ |
| 299 | { |
| 300 | "role": "user", |
| 301 | "content": [ |
| 302 | {"type": "image_url", "image_url": {"url": image_url}}, |
| 303 | {"type": "text", "text": "Represent the given image."}, |
| 304 | ], |
| 305 | } |
| 306 | ], |
| 307 | encoding_format="float", |
| 308 | ) |
| 309 | |
| 310 | print("Image embedding output:", response.data[0].embedding) |
| 311 | ``` |
| 312 | |
| 313 | === "DSE-Qwen2-MRL" |
| 314 | |
| 315 | To serve the model: |
| 316 | |
| 317 | ```bash |
| 318 | vllm serve MrLight/dse-qwen2-2b-mrl-v1 --runner pooling \ |
| 319 | --trust-remote-code \ |
| 320 | --max-model-len 8192 \ |
| 321 | --chat-template examples/pooling/embed/template/dse_qwen2_vl.jinja |
| 322 | ``` |
| 323 | |
| 324 | !!! important |
| 325 | Like with VLM2Vec, we have to explicitly pass `--runner pooling`. |
| 326 | |
| 327 | Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled |
| 328 | by a custom chat template: [examples/pooling/embed/template/dse_qwen2_vl.jinja](../../../examples/pooling/embed/template/dse_qwen2_vl.jinja) |
| 329 | |
| 330 | !!! important |
| 331 | `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code |
| 332 | example below for details. |
| 333 | |
| 334 | Full example: [examples/pooling/embed/vision_embedding_online.py](../../../examples/pooling/embed/vision_embedding_online.py) |
| 335 | |
| 336 | ### Cohere Embed API |
| 337 | |
| 338 | Our API is also compatible with [Cohere's Embed v2 API](https://docs.cohere.com/reference/embed) which adds support for some modern embedding feature such as truncation, output dimensions, embedding types, and input types. This endpoint works with any embedding model (including multimodal models). |
| 339 | |
| 340 | #### Cohere Embed API request parameters |
| 341 | |
| 342 | | Parameter | Type | Required | Description | |
| 343 | | --------- | ---- | -------- | ----------- | |
| 344 | | `model` | string | Yes | Model name | |
| 345 | | `input_type` | string | No | Prompt prefix key (model-dependent, see below) | |
| 346 | | `texts` | list[string] | No | Text inputs (use one of `texts`, `images`, or `inputs`) | |
| 347 | | `images` | list[string] | No | Base64 data URI images | |
| 348 | | `inputs` | list[object] | No | Mixed text and image content objects | |
| 349 | | `embedding_types` | list[string] | No | Output types (default: `["float"]`) | |
| 350 | | `output_dimension` | int | No | Truncate embeddings to this dimension (Matryoshka) | |
| 351 | | `truncate` | string | No | `END`, `START`, or `NONE` (default: `END`) | |
| 352 | |
| 353 | #### Text embedding |
| 354 | |
| 355 | ```bash |
| 356 | curl -X POST "http://localhost:8000/v2/embed" \ |
| 357 | -H "Content-Type: application/json" \ |
| 358 | -d '{ |
| 359 | "model": "Snowflake/snowflake-arctic-embed-m-v1.5", |
| 360 | "input_type": "query", |
| 361 | "texts": ["Hello world", "How are you?"], |
| 362 | "embedding_types": ["float"] |
| 363 | }' |
| 364 | ``` |
| 365 | |
| 366 | ??? console "Response" |
| 367 | |
| 368 | ```json |
| 369 | { |
| 370 | "id": "embd-...", |
| 371 | "embeddings": { |
| 372 | "float": [ |
| 373 | [0.012, -0.034, ...], |
| 374 | [0.056, 0.078, ...] |
| 375 | ] |
| 376 | }, |
| 377 | "texts": ["Hello world", "How are you?"], |
| 378 | "meta": { |
| 379 | "api_version": {"version": "2"}, |
| 380 | "billed_units": {"input_tokens": 12} |
| 381 | } |
| 382 | } |
| 383 | ``` |
| 384 | |
| 385 | #### Mixed text and image inputs |
| 386 | |
| 387 | For multimodal models, you can embed images by passing base64 data URIs. The `inputs` field accepts a list of objects with mixed text and image content: |
| 388 | |
| 389 | ```bash |
| 390 | curl -X POST "http://localhost:8000/v2/embed" \ |
| 391 | -H "Content-Type: application/json" \ |
| 392 | -d '{ |
| 393 | "model": "google/siglip-so400m-patch14-384", |
| 394 | "inputs": [ |
| 395 | { |
| 396 | "content": [ |
| 397 | {"type": "text", "text": "A photo of a cat"}, |
| 398 | {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBOR..."}} |
| 399 | ] |
| 400 | } |
| 401 | ], |
| 402 | "embedding_types": ["float"] |
| 403 | }' |
| 404 | ``` |
| 405 | |
| 406 | #### Embedding types |
| 407 | |
| 408 | The `embedding_types` parameter controls the output format. Multiple types can be requested in a single call: |
| 409 | |
| 410 | | Type | Description | |
| 411 | | ---- | ----------- | |
| 412 | | `float` | Raw float32 embeddings (default) | |
| 413 | | `binary` | Bit-packed signed binary | |
| 414 | | `ubinary` | Bit-packed unsigned binary | |
| 415 | | `base64` | Little-endian float32 encoded as base64 | |
| 416 | |
| 417 | ```bash |
| 418 | curl -X POST "http://localhost:8000/v2/embed" \ |
| 419 | -H "Content-Type: application/json" \ |
| 420 | -d '{ |
| 421 | "model": "Snowflake/snowflake-arctic-embed-m-v1.5", |
| 422 | "input_type": "query", |
| 423 | "texts": ["What is machine learning?"], |
| 424 | "embedding_types": ["float", "binary"] |
| 425 | }' |
| 426 | ``` |
| 427 | |
| 428 | ??? console "Response" |
| 429 | |
| 430 | ```json |
| 431 | { |
| 432 | "id": "embd-...", |
| 433 | "embeddings": { |
| 434 | "float": [[0.012, -0.034, ...]], |
| 435 | "binary": [[42, -117, ...]] |
| 436 | }, |
| 437 | "texts": ["What is machine learning?"], |
| 438 | "meta": { |
| 439 | "api_version": {"version": "2"}, |
| 440 | "billed_units": {"input_tokens": 8} |
| 441 | } |
| 442 | } |
| 443 | ``` |
| 444 | |
| 445 | #### Truncation |
| 446 | |
| 447 | The `truncate` parameter controls how inputs exceeding the model's maximum sequence length are handled: |
| 448 | |
| 449 | | Value | Behavior | |
| 450 | | ----- | --------- | |
| 451 | | `END` (default) | Keep the first tokens, drop the end | |
| 452 | | `START` | Keep the last tokens, drop the beginning | |
| 453 | | `NONE` | Return an error if the input is too long | |
| 454 | |
| 455 | #### Input type and prompt prefixes |
| 456 | |
| 457 | The `input_type` field selects a prompt prefix to prepend to each text input. The available values |
| 458 | depend on the model: |
| 459 | |
| 460 | - **Models with `task_instructions` in `config.json`**: The keys from the `task_instructions` dict are |
| 461 | the valid `input_type` values and the corresponding value is prepended to each text. |
| 462 | - **Models with `config_sentence_transformers.json` prompts**: The keys from the `prompts` dict are |
| 463 | the valid `input_type` values. For example, `Snowflake/snowflake-arctic-embed-xs` defines `"query"`, |
| 464 | so setting `input_type: "query"` prepends `"Represent this sentence for searching relevant passages: "`. |
| 465 | - **Other models**: `input_type` is not accepted and will raise a validation error if passed. |
| 466 | |
| 467 | ## More examples |
| 468 | |
| 469 | More examples can be found here: [examples/pooling/embed](../../../examples/pooling/embed) |
| 470 | |
| 471 | ## Supported Features |
| 472 | |
| 473 | ### Enable/disable normalize |
| 474 | |
| 475 | You can enable or disable normalize via `use_activation`. |
| 476 | |
| 477 | ### Matryoshka Embeddings |
| 478 | |
| 479 | [Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows users to trade off between performance and cost. |
| 480 | |
| 481 | !!! warning |
| 482 | Not all embedding models are trained using Matryoshka Representation Learning. To avoid misuse of the `dimensions` parameter, vLLM returns an error for requests that attempt to change the output dimension of models that do not support Matryoshka Embeddings. |
| 483 | |
| 484 | For example, setting `dimensions` parameter while using the `BAAI/bge-m3` model will result in the following error. |
| 485 | |
| 486 | ```json |
| 487 | {"object":"error","message":"Model \"BAAI/bge-m3\" does not support matryoshka representation, changing output dimensions will lead to poor results.","type":"BadRequestError","param":null,"code":400} |
| 488 | ``` |
| 489 | |
| 490 | #### Manually enable Matryoshka Embeddings |
| 491 | |
| 492 | There is currently no official interface for specifying support for Matryoshka Embeddings. In vLLM, if `is_matryoshka` is `True` in `config.json`, you can change the output dimension to arbitrary values. Use `matryoshka_dimensions` to control the allowed output dimensions. |
| 493 | |
| 494 | For models that support Matryoshka Embeddings but are not recognized by vLLM, manually override the config using `hf_overrides={"is_matryoshka": True}` or `hf_overrides={"matryoshka_dimensions": [<allowed output dimensions>]}` (offline), or `--hf-overrides '{"is_matryoshka": true}'` or `--hf-overrides '{"matryoshka_dimensions": [<allowed output dimensions>]}'` (online). |
| 495 | |
| 496 | Here is an example to serve a model with Matryoshka Embeddings enabled. |
| 497 | |
| 498 | ```bash |
| 499 | vllm serve Snowflake/snowflake-arctic-embed-m-v1.5 --hf-overrides '{"matryoshka_dimensions":[256]}' |
| 500 | ``` |
| 501 | |
| 502 | #### Offline Inference |
| 503 | |
| 504 | You can change the output dimensions of embedding models that support Matryoshka Embeddings by using the dimensions parameter in [PoolingParams][vllm.PoolingParams]. |
| 505 | |
| 506 | ```python |
| 507 | from vllm import LLM, PoolingParams |
| 508 | |
| 509 | llm = LLM( |
| 510 | model="jinaai/jina-embeddings-v3", |
| 511 | runner="pooling", |
| 512 | trust_remote_code=True, |
| 513 | ) |
| 514 | outputs = llm.embed( |
| 515 | ["Follow the white rabbit."], |
| 516 | pooling_params=PoolingParams(dimensions=32), |
| 517 | ) |
| 518 | print(outputs[0].outputs) |
| 519 | ``` |
| 520 | |
| 521 | A code example can be found here: [examples/pooling/embed/embed_matryoshka_fy_offline.py](../../../examples/pooling/embed/embed_matryoshka_fy_offline.py) |
| 522 | |
| 523 | #### Online Inference |
| 524 | |
| 525 | Use the following command to start the vLLM server. |
| 526 | |
| 527 | ```bash |
| 528 | vllm serve jinaai/jina-embeddings-v3 --trust-remote-code |
| 529 | ``` |
| 530 | |
| 531 | You can change the output dimensions of embedding models that support Matryoshka Embeddings by using the dimensions parameter. |
| 532 | |
| 533 | ```bash |
| 534 | curl http://127.0.0.1:8000/v1/embeddings \ |
| 535 | -H 'accept: application/json' \ |
| 536 | -H 'Content-Type: application/json' \ |
| 537 | -d '{ |
| 538 | "input": "Follow the white rabbit.", |
| 539 | "model": "jinaai/jina-embeddings-v3", |
| 540 | "encoding_format": "float", |
| 541 | "dimensions": 32 |
| 542 | }' |
| 543 | ``` |
| 544 | |
| 545 | Expected output: |
| 546 | |
| 547 | ```json |
| 548 | {"id":"embd-5c21fc9a5c9d4384a1b021daccaf9f64","object":"list","created":1745476417,"model":"jinaai/jina-embeddings-v3","data":[{"index":0,"object":"embedding","embedding":[-0.3828125,-0.1357421875,0.03759765625,0.125,0.21875,0.09521484375,-0.003662109375,0.1591796875,-0.130859375,-0.0869140625,-0.1982421875,0.1689453125,-0.220703125,0.1728515625,-0.2275390625,-0.0712890625,-0.162109375,-0.283203125,-0.055419921875,-0.0693359375,0.031982421875,-0.04052734375,-0.2734375,0.1826171875,-0.091796875,0.220703125,0.37890625,-0.0888671875,-0.12890625,-0.021484375,-0.0091552734375,0.23046875]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0,"prompt_tokens_details":null}} |
| 549 | ``` |
| 550 | |
| 551 | An OpenAI client example can be found here: [examples/pooling/embed/openai_embedding_matryoshka_fy_client.py](../../../examples/pooling/embed/openai_embedding_matryoshka_fy_client.py) |
| 552 | |
| 553 | ## Removed Features |
| 554 | |
| 555 | ### Remove `normalize` from PoolingParams |
| 556 | |
| 557 | We have already removed `normalize` from PoolingParams, use `use_activation` instead. |