@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f9e2a38386

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 6, 10:45 UTC

[Docs] Reorganize pooling docs. (#35592)

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

wang.yuqi committed Mar 19, 2026 at 19:25 UTC f9e2a383869c56a1fbee048afc9501ced9194c7e

17 files changed +2393 -1736

.github/CODEOWNERS

@@ -171,6 +171,7 @@ mkdocs.yaml @hmellor

171

172 # Pooling models

173 /examples/pooling @noooop

174 +/docs/models/pooling_models @noooop

175 /tests/models/*/pooling* @noooop

176 /tests/entrypoints/pooling @noooop

177 /vllm/config/pooler.py @noooop

docs/.nav.yml

+1 -1

@@ -25,7 +25,7 @@ nav:

25 - Models:

26 - models/supported_models.md

27 - models/generative_models.md

28 - - models/pooling_models.md

28 + - Pooling Models: models/pooling_models

29 - models/extensions

30 - Hardware Supported Models:

31 - models/hardware_supported_models/*

docs/contributing/model/tests.md

+1 -1

@@ -37,7 +37,7 @@ For [generative models](../../models/generative_models.md), there are two levels

38 #### Pooling models

40 -For [pooling models](../../models/pooling_models.md), we simply check the cosine similarity, as defined in [tests/models/utils.py](../../../tests/models/utils.py).

40 +For [pooling models](../../models/pooling_models/README.md), we simply check the cosine similarity, as defined in [tests/models/utils.py](../../../tests/models/utils.py).

42 ### Multi-modal processing

docs/features/README.md

+3 -3

@@ -36,14 +36,14 @@ th:not(:first-child) {

36 }

37 </style>

40 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |

41 | [CP](../configuration/optimization.md#chunked-prefill) | ✅ | | | | | | | | | | | | | | |

42 | [APC](automatic_prefix_caching.md) | ✅ | ✅ | | | | | | | | | | | | | |

43 | [LoRA](lora.md) | ✅ | ✅ | ✅ | | | | | | | | | | | | |

44 | [SD](speculative_decoding/README.md) | ✅ | ✅ | ❌ | ✅ | | | | | | | | | | | |

45 | CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | |

46 -| [pooling](../models/pooling_models.md) | 🟠\* | 🟠\* | ✅ | ❌ | ✅ | ✅ | | | | | | | | | |

46 +| [pooling](../models/pooling_models/README.md) | 🟠\* | 🟠\* | ✅ | ❌ | ✅ | ✅ | | | | | | | | | |

47 | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ❌ | [❌](https://github.com/vllm-project/vllm/issues/7366) | ❌ | [❌](https://github.com/vllm-project/vllm/issues/7366) | ✅ | ✅ | ✅ | | | | | | | | |

48 | <abbr title="Logprobs">logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | | | | | | | |

49 | <abbr title="Prompt Logprobs">prmpt logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | | | | | | |

@@ -66,7 +66,7 @@ th:not(:first-child) {

66 | [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

67 | [SD](speculative_decoding/README.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |

68 | CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/26970) |

69 -| [pooling](../models/pooling_models.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

69 +| [pooling](../models/pooling_models/README.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

70 | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |

71 | [mm](multimodal_inputs.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

72 | [prompt-embeds](prompt_embeds.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |

docs/mkdocs/hooks/url_schemes.py

+91 -43

@@ -1,7 +1,8 @@

1 # SPDX-License-Identifier: Apache-2.0

2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 """

4 -MkDocs hook to enable the following links to render correctly:

4 +MkDocs hook + markdown extension to enable the following links to render correctly,

5 +including inside content included via pymdownx.snippets:

7 - Relative file links outside of the `docs/` directory, e.g.:

8 - [Text](../some_file.py)

@@ -12,13 +13,17 @@ MkDocs hook to enable the following links to render correctly:

13 e.g. <...pull/123> -> [Pull Request #123](.../pull/123)

14 - Works for external repos too by including the `owner/repo` in the link title

15 -The goal is to simplify cross-referencing common GitHub resources

16 -in project docs.

16 +The link replacement runs as a markdown preprocessor (priority 25) so that it executes

17 +after pymdownx.snippets (priority 32) has expanded all included content.

18 +The on_page_markdown hook passes the current page context to the preprocessor before

19 +each page is converted.

20 """

22 from pathlib import Path

24 import regex as re

25 +from markdown import Extension

26 +from markdown.preprocessors import Preprocessor

27 from mkdocs.config.defaults import MkDocsConfig

28 from mkdocs.structure.files import Files

29 from mkdocs.structure.pages import Page

@@ -26,7 +31,6 @@ from mkdocs.structure.pages import Page

31 ROOT_DIR = Path(__file__).parent.parent.parent.parent.resolve()

32 DOC_DIR = ROOT_DIR / "docs"

29 -

34 gh_icon = ":octicons-mark-github-16:"

36 # Regex pieces

@@ -48,46 +52,90 @@ github_link = re.compile(rf"(\[{TITLE}\]\(|<){URL}(\)|>)")

52 relative_link = re.compile(rf"\[{TITLE}\]\({RELATIVE}\)")

55 +class UrlSchemesPreprocessor(Preprocessor):

56 + """Preprocessor that runs after pymdownx.snippets to process all links."""

57 +

58 + def __init__(self, md, ext):

59 + super().__init__(md)

60 + self.ext = ext

61 +

62 + def run(self, lines):

63 + page = self.ext.page

64 + if page is None or getattr(page.file, "abs_src_path", None) is None:

65 + return lines

66 +

67 + def replace_relative_link(match: re.Match) -> str:

68 + """

69 + Replace relative file links with URLs if they point outside the docs dir.

70 + """

71 + title = match.group("title")

72 + path = match.group("path")

73 + path = (Path(page.file.abs_src_path).parent / path).resolve()

74 + fragment = match.group("fragment") or ""

75 +

76 + # Check if the path exists and is outside the docs dir

77 + if not path.exists() or path.is_relative_to(DOC_DIR):

78 + return match.group(0)

79 +

80 + # Files and directories have different URL schemes on GitHub

81 + slug = "tree/main" if path.is_dir() else "blob/main"

82 +

83 + path = path.relative_to(ROOT_DIR)

84 + url = f"https://github.com/vllm-project/vllm/{slug}/{path}{fragment}"

85 + return f"[{gh_icon} {title}]({url})"

86 +

87 + def replace_github_link(match: re.Match) -> str:

88 + """

89 + Replace GitHub issue, PR, and project links with enhanced Markdown links.

90 + """

91 + repo = match.group("repo")

92 + type = match.group("type")

93 + number = match.group("number")

94 + # Title and fragment could be None

95 + title = match.group("title") or ""

96 + fragment = match.group("fragment") or ""

97 +

98 + # Use default titles for raw links

99 + if not title:

100 + title = TITLES[type]

101 + if "vllm-project" not in repo:

102 + title += repo

103 + title += f"#{number}"

104 +

105 + url = f"https://github.com/{repo}/{type}/{number}{fragment}"

106 + return f"[{gh_icon} {title}]({url})"

107 +

108 + markdown = "\n".join(lines)

109 + markdown = relative_link.sub(replace_relative_link, markdown)

110 + markdown = github_link.sub(replace_github_link, markdown)

111 + return markdown.split("\n")

112 +

113 +

114 +class UrlSchemesExtension(Extension):

115 + """Markdown extension that registers the URL schemes preprocessor."""

116 +

117 + def __init__(self, **kwargs):

118 + self.page = None

119 + super().__init__(**kwargs)

120 +

121 + def extendMarkdown(self, md):

122 + # Priority 25 runs after pymdownx.snippets (priority 32)

123 + md.preprocessors.register(UrlSchemesPreprocessor(md, self), "url_schemes", 25)

124 +

125 +

126 +# Singleton extension instance shared between the hook and the preprocessor.

127 +_ext = UrlSchemesExtension()

128 +

129 +

130 +def on_config(config: MkDocsConfig) -> MkDocsConfig:

131 + """Register the URL schemes markdown extension."""

132 + config["markdown_extensions"].append(_ext)

133 + return config

134 +

135 +

136 def on_page_markdown(

137 markdown: str, *, page: Page, config: MkDocsConfig, files: Files

138 ) -> str:

54 - def replace_relative_link(match: re.Match) -> str:

55 - """Replace relative file links with URLs if they point outside the docs dir."""

56 - title = match.group("title")

57 - path = match.group("path")

58 - path = (Path(page.file.abs_src_path).parent / path).resolve()

59 - fragment = match.group("fragment") or ""

60 -

61 - # Check if the path exists and is outside the docs dir

62 - if not path.exists() or path.is_relative_to(DOC_DIR):

63 - return match.group(0)

64 -

65 - # Files and directories have different URL schemes on GitHub

66 - slug = "tree/main" if path.is_dir() else "blob/main"

67 -

68 - path = path.relative_to(ROOT_DIR)

69 - url = f"https://github.com/vllm-project/vllm/{slug}/{path}{fragment}"

70 - return f"[{gh_icon} {title}]({url})"

71 -

72 - def replace_github_link(match: re.Match) -> str:

73 - """Replace GitHub issue, PR, and project links with enhanced Markdown links."""

74 - repo = match.group("repo")

75 - type = match.group("type")

76 - number = match.group("number")

77 - # Title and fragment could be None

78 - title = match.group("title") or ""

79 - fragment = match.group("fragment") or ""

80 -

81 - # Use default titles for raw links

82 - if not title:

83 - title = TITLES[type]

84 - if "vllm-project" not in repo:

85 - title += repo

86 - title += f"#{number}"

87 -

88 - url = f"https://github.com/{repo}/{type}/{number}{fragment}"

89 - return f"[{gh_icon} {title}]({url})"

90 -

91 - markdown = relative_link.sub(replace_relative_link, markdown)

92 - markdown = github_link.sub(replace_github_link, markdown)

139 + """Pass the current page context to the preprocessor."""

140 + _ext.page = page

141 return markdown

docs/models/pooling_models.md deleted

-716

@@ -1,716 +0,0 @@

1 -# Pooling Models

2 -

3 -vLLM also supports pooling models, such as embedding, classification, and reward models.

4 -

5 -In vLLM, pooling models implement the [VllmModelForPooling][vllm.model_executor.models.VllmModelForPooling] interface.

6 -These models use a [Pooler][vllm.model_executor.layers.pooler.Pooler] to extract the final hidden states of the input

7 -before returning them.

8 -

9 -!!! note

10 - We currently support pooling models primarily for convenience. This is not guaranteed to provide any performance improvements over using Hugging Face Transformers or Sentence Transformers directly.

11 -

12 - We plan to optimize pooling models in vLLM. Please comment on <https://github.com/vllm-project/vllm/issues/21796> if you have any suggestions!

13 -

14 -## Configuration

15 -

16 -### Model Runner

17 -

18 -Run a model in pooling mode via the option `--runner pooling`.

19 -

20 -!!! tip

21 - There is no need to set this option in the vast majority of cases as vLLM can automatically

22 - detect the appropriate model runner via `--runner auto`.

23 -

24 -### Model Conversion

25 -

26 -vLLM can adapt models for various pooling tasks via the option `--convert <type>`.

27 -

28 -If `--runner pooling` has been set (manually or automatically) but the model does not implement the

29 -[VllmModelForPooling][vllm.model_executor.models.VllmModelForPooling] interface,

30 -vLLM will attempt to automatically convert the model according to the architecture names

31 -shown in the table below.

32 -

33 -| Architecture | `--convert` | Supported pooling tasks |

34 -| ----------------------------------------------- | ----------- | ------------------------------------- |

35 -| `*ForTextEncoding`, `*EmbeddingModel`, `*Model` | `embed` | `token_embed`, `embed` |

36 -| `*ForRewardModeling`, `*RewardModel` | `embed` | `token_embed`, `embed` |

37 -| `*For*Classification`, `*ClassificationModel` | `classify` | `token_classify`, `classify`, `score` |

38 -

39 -!!! tip

40 - You can explicitly set `--convert <type>` to specify how to convert the model.

41 -

42 -### Pooling Tasks

43 -

44 -Each pooling model in vLLM supports one or more of these tasks according to

45 -[Pooler.get_supported_tasks][vllm.model_executor.layers.pooler.Pooler.get_supported_tasks],

46 -enabling the corresponding APIs:

47 -

48 -| Task | APIs |

49 -| ---------------- | ----------------------------------------------------------------------------- |

50 -| `embed` | `LLM.embed(...)`, `LLM.score(...)`\*, `LLM.encode(..., pooling_task="embed")` |

51 -| `classify` | `LLM.classify(...)`, `LLM.encode(..., pooling_task="classify")` |

52 -| `score` | `LLM.score(...)` |

53 -| `token_classify` | `LLM.reward(...)`, `LLM.encode(..., pooling_task="token_classify")` |

54 -| `token_embed` | `LLM.encode(..., pooling_task="token_embed")` |

55 -| `plugin` | `LLM.encode(..., pooling_task="plugin")` |

56 -

57 -\* The `LLM.score(...)` API falls back to `embed` task if the model does not support `score` task.

58 -

59 -### Pooler Configuration

60 -

61 -#### Predefined models

62 -

63 -If the [Pooler][vllm.model_executor.layers.pooler.Pooler] defined by the model accepts `pooler_config`,

64 -you can override some of its attributes via the `--pooler-config` option.

65 -

66 -#### Converted models

67 -

68 -If the model has been converted via `--convert` (see above),

69 -the pooler assigned to each task has the following attributes by default:

70 -

72 -| ---------- | ------------ | ------------- | ------- |

73 -| `embed` | `LAST` | ✅︎ | ❌ |

74 -| `classify` | `LAST` | ❌ | ✅︎ |

75 -

76 -When loading [Sentence Transformers](https://huggingface.co/sentence-transformers) models,

77 -its Sentence Transformers configuration file (`modules.json`) takes priority over the model's defaults.

78 -

79 -You can further customize this via the `--pooler-config` option,

80 -which takes priority over both the model's and Sentence Transformers' defaults.

81 -

82 -## Offline Inference

83 -

84 -The [LLM][vllm.LLM] class provides various methods for offline inference.

85 -See [configuration](../api/README.md#configuration) for a list of options when initializing the model.

86 -

87 -### `LLM.embed`

88 -

89 -The [embed][vllm.LLM.embed] method outputs an embedding vector for each prompt.

90 -It is primarily designed for embedding models.

91 -

92 -```python

93 -from vllm import LLM

94 -

95 -llm = LLM(model="intfloat/e5-small", runner="pooling")

96 -(output,) = llm.embed("Hello, my name is")

97 -

98 -embeds = output.outputs.embedding

99 -print(f"Embeddings: {embeds!r} (size={len(embeds)})")

100 -```

101 -

102 -A code example can be found here: [examples/basic/offline_inference/embed.py](../../examples/basic/offline_inference/embed.py)

103 -

104 -### `LLM.classify`

105 -

106 -The [classify][vllm.LLM.classify] method outputs a probability vector for each prompt.

107 -It is primarily designed for classification models.

108 -

109 -```python

110 -from vllm import LLM

111 -

112 -llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", runner="pooling")

113 -(output,) = llm.classify("Hello, my name is")

114 -

115 -probs = output.outputs.probs

116 -print(f"Class Probabilities: {probs!r} (size={len(probs)})")

117 -```

118 -

119 -A code example can be found here: [examples/basic/offline_inference/classify.py](../../examples/basic/offline_inference/classify.py)

120 -

121 -### `LLM.score`

122 -

123 -The [score][vllm.LLM.score] method outputs similarity scores between sentence pairs.

124 -It is designed for embedding models and cross-encoder models. Embedding models use cosine similarity, and [cross-encoder models](https://www.sbert.net/examples/applications/cross-encoder/README.html) serve as rerankers between candidate query-document pairs in RAG systems.

125 -

126 -!!! note

127 - vLLM can only perform the model inference component (e.g. embedding, reranking) of RAG.

128 - To handle RAG at a higher level, you should use integration frameworks such as [LangChain](https://github.com/langchain-ai/langchain).

129 -

130 -```python

131 -from vllm import LLM

132 -

133 -llm = LLM(model="BAAI/bge-reranker-v2-m3", runner="pooling")

134 -(output,) = llm.score(

135 - "What is the capital of France?",

136 - "The capital of Brazil is Brasilia.",

137 -)

138 -

139 -score = output.outputs.score

140 -print(f"Score: {score}")

141 -```

142 -

143 -A code example can be found here: [examples/basic/offline_inference/score.py](../../examples/basic/offline_inference/score.py)

144 -

145 -### `LLM.reward`

146 -

147 -The [reward][vllm.LLM.reward] method is available to all reward models in vLLM.

148 -

149 -```python

150 -from vllm import LLM

151 -

152 -llm = LLM(model="internlm/internlm2-1_8b-reward", runner="pooling", trust_remote_code=True)

153 -(output,) = llm.reward("Hello, my name is")

154 -

155 -data = output.outputs.data

156 -print(f"Data: {data!r}")

157 -```

158 -

159 -A code example can be found here: [examples/basic/offline_inference/reward.py](../../examples/basic/offline_inference/reward.py)

160 -

161 -### `LLM.encode`

162 -

163 -The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.

164 -

165 -!!! note

166 - Please use one of the more specific methods or set the task directly when using `LLM.encode`:

167 -

168 - - For embeddings, use `LLM.embed(...)` or `pooling_task="embed"`.

169 - - For classification logits, use `LLM.classify(...)` or `pooling_task="classify"`.

170 - - For similarity scores, use `LLM.score(...)`.

171 - - For rewards, use `LLM.reward(...)` or `pooling_task="token_classify"`.

172 - - For token classification, use `pooling_task="token_classify"`.

173 - - For multi-vector retrieval, use `pooling_task="token_embed"`.

174 - - For IO Processor Plugins, use `pooling_task="plugin"`.

175 -

176 -```python

177 -from vllm import LLM

178 -

179 -llm = LLM(model="intfloat/e5-small", runner="pooling")

180 -(output,) = llm.encode("Hello, my name is", pooling_task="embed")

181 -

182 -data = output.outputs.data

183 -print(f"Data: {data!r}")

184 -```

185 -

186 -## Online Serving

187 -

188 -Our [OpenAI-Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:

189 -

190 -- [Embeddings API](../serving/openai_compatible_server.md#embeddings-api) is similar to `LLM.embed`, accepting both text and [multi-modal inputs](../features/multimodal_inputs.md) for embedding models.

191 -- [Classification API](../serving/openai_compatible_server.md#classification-api) is similar to `LLM.classify` and is applicable to sequence classification models.

192 -- [Score API](../serving/openai_compatible_server.md#score-api) is similar to `LLM.score` for cross-encoder models.

193 -- [Pooling API](../serving/openai_compatible_server.md#pooling-api) is similar to `LLM.encode`, being applicable to all types of pooling models.

194 -

195 -!!! note

196 - Please use one of the more specific endpoints or set the task directly when using the [Pooling API](../serving/openai_compatible_server.md#pooling-api):

197 -

198 - - For embeddings, use [Embeddings API](../serving/openai_compatible_server.md#embeddings-api) or `"task":"embed"`.

199 - - For classification logits, use [Classification API](../serving/openai_compatible_server.md#classification-api) or `"task":"classify"`.

200 - - For similarity scores, use [Score API](../serving/openai_compatible_server.md#score-api).

201 - - For rewards, use `"task":"token_classify"`.

202 - - For token classification, use `"task":"token_classify"`.

203 - - For multi-vector retrieval, use `"task":"token_embed"`.

204 - - For IO Processor Plugins, use `"task":"plugin"`.

205 -

206 -```python

207 -# start a supported embeddings model server with `vllm serve`, e.g.

208 -# vllm serve intfloat/e5-small

209 -import requests

210 -

211 -host = "localhost"

212 -port = "8000"

213 -model_name = "intfloat/e5-small"

214 -

215 -api_url = f"http://{host}:{port}/pooling"

216 -

217 -prompts = [

218 - "Hello, my name is",

219 - "The president of the United States is",

220 - "The capital of France is",

221 - "The future of AI is",

222 -]

223 -prompt = {"model": model_name, "input": prompts, "task": "embed"}

224 -

225 -response = requests.post(api_url, json=prompt)

226 -

227 -for output in response.json()["data"]:

228 - data = output["data"]

229 - print(f"Data: {data!r} (size={len(data)})")

230 -```

231 -

232 -## Matryoshka Embeddings

233 -

234 -[Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows users to trade off between performance and cost.

235 -

236 -!!! warning

237 - Not all embedding models are trained using Matryoshka Representation Learning. To avoid misuse of the `dimensions` parameter, vLLM returns an error for requests that attempt to change the output dimension of models that do not support Matryoshka Embeddings.

238 -

239 - For example, setting `dimensions` parameter while using the `BAAI/bge-m3` model will result in the following error.

240 -

241 - ```json

242 - {"object":"error","message":"Model \"BAAI/bge-m3\" does not support matryoshka representation, changing output dimensions will lead to poor results.","type":"BadRequestError","param":null,"code":400}

243 - ```

244 -

245 -### Manually enable Matryoshka Embeddings

246 -

247 -There is currently no official interface for specifying support for Matryoshka Embeddings. In vLLM, if `is_matryoshka` is `True` in `config.json`, you can change the output dimension to arbitrary values. Use `matryoshka_dimensions` to control the allowed output dimensions.

248 -

249 -For models that support Matryoshka Embeddings but are not recognized by vLLM, manually override the config using `hf_overrides={"is_matryoshka": True}` or `hf_overrides={"matryoshka_dimensions": [<allowed output dimensions>]}` (offline), or `--hf-overrides '{"is_matryoshka": true}'` or `--hf-overrides '{"matryoshka_dimensions": [<allowed output dimensions>]}'` (online).

250 -

251 -Here is an example to serve a model with Matryoshka Embeddings enabled.

252 -

253 -```bash

254 -vllm serve Snowflake/snowflake-arctic-embed-m-v1.5 --hf-overrides '{"matryoshka_dimensions":[256]}'

255 -```

256 -

257 -### Offline Inference

258 -

259 -You can change the output dimensions of embedding models that support Matryoshka Embeddings by using the dimensions parameter in [PoolingParams][vllm.PoolingParams].

260 -

261 -```python

262 -from vllm import LLM, PoolingParams

263 -

264 -llm = LLM(

265 - model="jinaai/jina-embeddings-v3",

266 - runner="pooling",

267 - trust_remote_code=True,

268 -)

269 -outputs = llm.embed(

270 - ["Follow the white rabbit."],

271 - pooling_params=PoolingParams(dimensions=32),

272 -)

273 -print(outputs[0].outputs)

274 -```

275 -

276 -A code example can be found here: [examples/pooling/embed/embed_matryoshka_fy_offline.py](../../examples/pooling/embed/embed_matryoshka_fy_offline.py)

277 -

278 -### Online Inference

279 -

280 -Use the following command to start the vLLM server.

281 -

282 -```bash

283 -vllm serve jinaai/jina-embeddings-v3 --trust-remote-code

284 -```

285 -

286 -You can change the output dimensions of embedding models that support Matryoshka Embeddings by using the dimensions parameter.

287 -

288 -```bash

289 -curl http://127.0.0.1:8000/v1/embeddings \

290 - -H 'accept: application/json' \

291 - -H 'Content-Type: application/json' \

292 - -d '{

293 - "input": "Follow the white rabbit.",

294 - "model": "jinaai/jina-embeddings-v3",

295 - "encoding_format": "float",

296 - "dimensions": 32

297 - }'

298 -```

299 -

300 -Expected output:

301 -

302 -```json

303 -{"id":"embd-5c21fc9a5c9d4384a1b021daccaf9f64","object":"list","created":1745476417,"model":"jinaai/jina-embeddings-v3","data":[{"index":0,"object":"embedding","embedding":[-0.3828125,-0.1357421875,0.03759765625,0.125,0.21875,0.09521484375,-0.003662109375,0.1591796875,-0.130859375,-0.0869140625,-0.1982421875,0.1689453125,-0.220703125,0.1728515625,-0.2275390625,-0.0712890625,-0.162109375,-0.283203125,-0.055419921875,-0.0693359375,0.031982421875,-0.04052734375,-0.2734375,0.1826171875,-0.091796875,0.220703125,0.37890625,-0.0888671875,-0.12890625,-0.021484375,-0.0091552734375,0.23046875]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0,"prompt_tokens_details":null}}

304 -```

305 -

306 -An OpenAI client example can be found here: [examples/pooling/embed/openai_embedding_matryoshka_fy_client.py](../../examples/pooling/embed/openai_embedding_matryoshka_fy_client.py)

307 -

308 -## Specific models

309 -

310 -### ColBERT Late Interaction Models

311 -

312 -[ColBERT](https://arxiv.org/abs/2004.12832) (Contextualized Late Interaction over BERT) is a retrieval model that uses per-token embeddings and MaxSim scoring for document ranking. Unlike single-vector embedding models, ColBERT retains token-level representations and computes relevance scores through late interaction, providing better accuracy while being more efficient than cross-encoders.

313 -

314 -vLLM supports ColBERT models with multiple encoder backbones:

315 -

316 -| Architecture | Backbone | Example HF Models |

317 -| - | - | - |

318 -| `HF_ColBERT` | BERT | `answerdotai/answerai-colbert-small-v1`, `colbert-ir/colbertv2.0` |

319 -| `ColBERTModernBertModel` | ModernBERT | `lightonai/GTE-ModernColBERT-v1` |

320 -| `ColBERTJinaRobertaModel` | Jina XLM-RoBERTa | `jinaai/jina-colbert-v2` |

321 -

322 -**BERT-based ColBERT** models work out of the box:

323 -

324 -```shell

325 -vllm serve answerdotai/answerai-colbert-small-v1

326 -```

327 -

328 -For **non-BERT backbones**, use `--hf-overrides` to set the correct architecture:

329 -

330 -```shell

331 -# ModernBERT backbone

332 -vllm serve lightonai/GTE-ModernColBERT-v1 \

333 - --hf-overrides '{"architectures": ["ColBERTModernBertModel"]}'

334 -

335 -# Jina XLM-RoBERTa backbone

336 -vllm serve jinaai/jina-colbert-v2 \

337 - --hf-overrides '{"architectures": ["ColBERTJinaRobertaModel"]}' \

338 - --trust-remote-code

339 -```

340 -

341 -Then you can use the rerank endpoint:

342 -

343 -```shell

344 -curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

345 - "model": "answerdotai/answerai-colbert-small-v1",

346 - "query": "What is machine learning?",

347 - "documents": [

348 - "Machine learning is a subset of artificial intelligence.",

349 - "Python is a programming language.",

350 - "Deep learning uses neural networks."

351 - ]

352 -}'

353 -```

354 -

355 -Or the score endpoint:

356 -

357 -```shell

358 -curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

359 - "model": "answerdotai/answerai-colbert-small-v1",

360 - "text_1": "What is machine learning?",

361 - "text_2": ["Machine learning is a subset of AI.", "The weather is sunny."]

362 -}'

363 -```

364 -

365 -You can also get the raw token embeddings using the pooling endpoint with `token_embed` task:

366 -

367 -```shell

368 -curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

369 - "model": "answerdotai/answerai-colbert-small-v1",

370 - "input": "What is machine learning?",

371 - "task": "token_embed"

372 -}'

373 -```

374 -

375 -An example can be found here: [examples/pooling/score/colbert_rerank_online.py](../../examples/pooling/score/colbert_rerank_online.py)

376 -

377 -### ColQwen3 Multi-Modal Late Interaction Models

378 -

379 -ColQwen3 is based on [ColPali](https://arxiv.org/abs/2407.01449), which extends ColBERT's late interaction approach to **multi-modal** inputs. While ColBERT operates on text-only token embeddings, ColPali/ColQwen3 can embed both **text and images** (e.g. PDF pages, screenshots, diagrams) into per-token L2-normalized vectors and compute relevance via MaxSim scoring. ColQwen3 specifically uses Qwen3-VL as its vision-language backbone.

380 -

381 -| Architecture | Backbone | Example HF Models |

382 -| - | - | - |

383 -| `ColQwen3` | Qwen3-VL | `TomoroAI/tomoro-colqwen3-embed-4b`, `TomoroAI/tomoro-colqwen3-embed-8b` |

384 -| `OpsColQwen3Model` | Qwen3-VL | `OpenSearch-AI/Ops-Colqwen3-4B`, `OpenSearch-AI/Ops-Colqwen3-8B` |

385 -| `Qwen3VLNemotronEmbedModel` | Qwen3-VL | `nvidia/nemotron-colembed-vl-4b-v2`, `nvidia/nemotron-colembed-vl-8b-v2` |

386 -

387 -Start the server:

388 -

389 -```shell

390 -vllm serve TomoroAI/tomoro-colqwen3-embed-4b --max-model-len 4096

391 -```

392 -

393 -#### Text-only scoring and reranking

394 -

395 -Use the `/rerank` endpoint:

396 -

397 -```shell

398 -curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

399 - "model": "TomoroAI/tomoro-colqwen3-embed-4b",

400 - "query": "What is machine learning?",

401 - "documents": [

402 - "Machine learning is a subset of artificial intelligence.",

403 - "Python is a programming language.",

404 - "Deep learning uses neural networks."

405 - ]

406 -}'

407 -```

408 -

409 -Or the `/score` endpoint:

410 -

411 -```shell

412 -curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

413 - "model": "TomoroAI/tomoro-colqwen3-embed-4b",

414 - "text_1": "What is the capital of France?",

415 - "text_2": ["The capital of France is Paris.", "Python is a programming language."]

416 -}'

417 -```

418 -

419 -#### Multi-modal scoring and reranking (text query × image documents)

420 -

421 -The `/score` and `/rerank` endpoints also accept multi-modal inputs directly.

422 -Pass image documents using the `data_1`/`data_2` (for `/score`) or `documents` (for `/rerank`) fields

423 -with a `content` list containing `image_url` and `text` parts — the same format used by the

424 -OpenAI chat completion API:

425 -

426 -Score a text query against image documents:

427 -

428 -```shell

429 -curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

430 - "model": "TomoroAI/tomoro-colqwen3-embed-4b",

431 - "data_1": "Retrieve the city of Beijing",

432 - "data_2": [

433 - {

434 - "content": [

435 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

436 - {"type": "text", "text": "Describe the image."}

437 - ]

438 - }

439 - ]

440 -}'

441 -```

442 -

443 -Rerank image documents by a text query:

444 -

445 -```shell

446 -curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

447 - "model": "TomoroAI/tomoro-colqwen3-embed-4b",

448 - "query": "Retrieve the city of Beijing",

449 - "documents": [

450 - {

451 - "content": [

452 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_1>"}},

453 - {"type": "text", "text": "Describe the image."}

454 - ]

455 - },

456 - {

457 - "content": [

458 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_2>"}},

459 - {"type": "text", "text": "Describe the image."}

460 - ]

461 - }

462 - ],

463 - "top_n": 2

464 -}'

465 -```

466 -

467 -#### Raw token embeddings

468 -

469 -You can also get the raw token embeddings using the `/pooling` endpoint with `token_embed` task:

470 -

471 -```shell

472 -curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

473 - "model": "TomoroAI/tomoro-colqwen3-embed-4b",

474 - "input": "What is machine learning?",

475 - "task": "token_embed"

476 -}'

477 -```

478 -

479 -For **image inputs** via the pooling endpoint, use the chat-style `messages` field:

480 -

481 -```shell

482 -curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

483 - "model": "TomoroAI/tomoro-colqwen3-embed-4b",

484 - "messages": [

485 - {

486 - "role": "user",

487 - "content": [

488 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

489 - {"type": "text", "text": "Describe the image."}

490 - ]

491 - }

492 - ]

493 -}'

494 -```

495 -

496 -#### Examples

497 -

498 -- Multi-vector retrieval: [examples/pooling/token_embed/colqwen3_token_embed_online.py](../../examples/pooling/token_embed/colqwen3_token_embed_online.py)

499 -- Reranking (text + multi-modal): [examples/pooling/score/colqwen3_rerank_online.py](../../examples/pooling/score/colqwen3_rerank_online.py)

500 -

501 -### Llama Nemotron Multimodal

502 -

503 -#### Embedding Model

504 -

505 -Llama Nemotron VL Embedding models combine the bidirectional Llama embedding backbone

506 -(from `nvidia/llama-nemotron-embed-1b-v2`) with SigLIP as the vision encoder to produce

507 -single-vector embeddings from text and/or images.

508 -

509 -| Architecture | Backbone | Example HF Models |

510 -| - | - | - |

511 -| `LlamaNemotronVLModel` | Bidirectional Llama + SigLIP | `nvidia/llama-nemotron-embed-vl-1b-v2` |

512 -

513 -Start the server:

514 -

515 -```shell

516 -vllm serve nvidia/llama-nemotron-embed-vl-1b-v2 \

517 - --trust-remote-code \

518 - --chat-template examples/pooling/embed/template/nemotron_embed_vl.jinja

519 -```

520 -

521 -!!! note

522 - The chat template bundled with this model's tokenizer is not suitable for

523 - the embeddings API. Use the provided override template above when serving

524 - with the `messages`-based (chat-style) embeddings endpoint.

525 -

526 - The override template uses the message `role` to automatically prepend the

527 - appropriate prefix: set `role` to `"query"` for queries (prepends `query: `)

528 - or `"document"` for passages (prepends `passage: `). Any other role omits

529 - the prefix.

530 -

531 -Embed text queries:

532 -

533 -```shell

534 -curl -s http://localhost:8000/v1/embeddings -H "Content-Type: application/json" -d '{

535 - "model": "nvidia/llama-nemotron-embed-vl-1b-v2",

536 - "messages": [

537 - {

538 - "role": "query",

539 - "content": [

540 - {"type": "text", "text": "What is machine learning?"}

541 - ]

542 - }

543 - ]

544 -}'

545 -```

546 -

547 -Embed images via the chat-style `messages` field:

548 -

549 -```shell

550 -curl -s http://localhost:8000/v1/embeddings -H "Content-Type: application/json" -d '{

551 - "model": "nvidia/llama-nemotron-embed-vl-1b-v2",

552 - "messages": [

553 - {

554 - "role": "document",

555 - "content": [

556 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

557 - {"type": "text", "text": "Describe the image."}

558 - ]

559 - }

560 - ]

561 -}'

562 -```

563 -

564 -#### Reranker Model

565 -

566 -Llama Nemotron VL reranker models combine the same bidirectional Llama + SigLIP

567 -backbone with a sequence-classification head for cross-encoder scoring and reranking.

568 -

569 -| Architecture | Backbone | Example HF Models |

570 -| - | - | - |

571 -| `LlamaNemotronVLForSequenceClassification` | Bidirectional Llama + SigLIP | `nvidia/llama-nemotron-rerank-vl-1b-v2` |

572 -

573 -Start the server:

574 -

575 -```shell

576 -vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \

577 - --runner pooling \

578 - --trust-remote-code \

579 - --chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja

580 -```

581 -

582 -!!! note

583 - The chat template bundled with this checkpoint's tokenizer is not suitable

584 - for the Score/Rerank APIs. Use the provided override template when serving:

585 - `examples/pooling/score/template/nemotron-vl-rerank.jinja`.

586 -

587 -Score a text query against an image document:

588 -

589 -```shell

590 -curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

591 - "model": "nvidia/llama-nemotron-rerank-vl-1b-v2",

592 - "data_1": "Find diagrams about autonomous robots",

593 - "data_2": [

594 - {

595 - "content": [

596 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

597 - {"type": "text", "text": "Robotics workflow diagram."}

598 - ]

599 - }

600 - ]

601 -}'

602 -```

603 -

604 -Rerank image documents by a text query:

605 -

606 -```shell

607 -curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

608 - "model": "nvidia/llama-nemotron-rerank-vl-1b-v2",

609 - "query": "Find diagrams about autonomous robots",

610 - "documents": [

611 - {

612 - "content": [

613 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_1>"}},

614 - {"type": "text", "text": "Robotics workflow diagram."}

615 - ]

616 - },

617 - {

618 - "content": [

619 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_2>"}},

620 - {"type": "text", "text": "General skyline photo."}

621 - ]

622 - }

623 - ],

624 - "top_n": 2

625 -}'

626 -```

627 -

628 -### ColQwen3.5 Multi-Modal Late Interaction Models

629 -

630 -ColQwen3.5 is based on [ColPali](https://arxiv.org/abs/2407.01449), extending ColBERT's late interaction approach to **multi-modal** inputs. It uses the Qwen3.5 hybrid backbone (linear + full attention) and produces per-token L2-normalized vectors for MaxSim scoring.

631 -

632 -| Architecture | Backbone | Example HF Models |

633 -| - | - | - |

634 -| `ColQwen3_5` | Qwen3.5 | `athrael-soju/colqwen3.5-4.5B` |

635 -

636 -Start the server:

637 -

638 -```shell

639 -vllm serve athrael-soju/colqwen3.5-4.5B --max-model-len 4096

640 -```

641 -

642 -Then you can use the rerank endpoint:

643 -

644 -```shell

645 -curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

646 - "model": "athrael-soju/colqwen3.5-4.5B",

647 - "query": "What is machine learning?",

648 - "documents": [

649 - "Machine learning is a subset of artificial intelligence.",

650 - "Python is a programming language.",

651 - "Deep learning uses neural networks."

652 - ]

653 -}'

654 -```

655 -

656 -Or the score endpoint:

657 -

658 -```shell

659 -curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

660 - "model": "athrael-soju/colqwen3.5-4.5B",

661 - "text_1": "What is the capital of France?",

662 - "text_2": ["The capital of France is Paris.", "Python is a programming language."]

663 -}'

664 -```

665 -

666 -An example can be found here: [examples/pooling/score/colqwen3_5_rerank_online.py](../../examples/pooling/score/colqwen3_5_rerank_online.py)

667 -

668 -### BAAI/bge-m3

669 -

670 -The `BAAI/bge-m3` model comes with extra weights for sparse and colbert embeddings but unfortunately in its `config.json`

671 -the architecture is declared as `XLMRobertaModel`, which makes `vLLM` load it as a vanilla ROBERTA model without the

672 -extra weights. To load the full model weights, override its architecture like this:

673 -

674 -```shell

675 -vllm serve BAAI/bge-m3 --hf-overrides '{"architectures": ["BgeM3EmbeddingModel"]}'

676 -```

677 -

678 -Then you obtain the sparse embeddings like this:

679 -

680 -```shell

681 -curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

682 - "model": "BAAI/bge-m3",

683 - "task": "token_classify",

684 - "input": ["What is BGE M3?", "Definition of BM25"]

685 -}'

686 -```

687 -

688 -Due to limitations in the output schema, the output consists of a list of

689 -token scores for each token for each input. This means that you'll have to call

690 -`/tokenize` as well to be able to pair tokens with scores.

691 -Refer to the tests in `tests/models/language/pooling/test_bge_m3.py` to see how

692 -to do that.

693 -

694 -You can obtain the colbert embeddings like this:

695 -

696 -```shell

697 -curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

698 - "model": "BAAI/bge-m3",

699 - "task": "token_embed",

700 - "input": ["What is BGE M3?", "Definition of BM25"]

701 -}'

702 -```

703 -

704 -## Deprecated Features

705 -

706 -### Encode task

707 -

708 -We have split the `encode` task into two more specific token-wise tasks: `token_embed` and `token_classify`:

709 -

710 -- `token_embed` is the same as `embed`, using normalization as the activation.

711 -- `token_classify` is the same as `classify`, by default using softmax as the activation.

712 -

713 -Pooling models now default support all pooling, you can use it without any settings.

714 -

715 -- Extracting hidden states prefers using `token_embed` task.

716 -- Reward models prefers using `token_classify` task.

docs/models/pooling_models/README.md new

+253

@@ -0,0 +1,253 @@

1 +# Pooling Models

2 +

3 +!!! note

4 + We currently support pooling models primarily for convenience. This is not guaranteed to provide any performance improvements over using Hugging Face Transformers or Sentence Transformers directly.

5 +

6 + We plan to optimize pooling models in vLLM. Please comment on <https://github.com/vllm-project/vllm/issues/21796> if you have any suggestions!

7 +

8 +## What are pooling models?

9 +

10 +Natural Language Processing (NLP) can be primarily divided into the following two types of tasks:

11 +

12 +- Natural Language Understanding (NLU)

13 +- Natural Language Generation (NLG)

14 +

15 +The generative models supported by vLLM cover a variety of task types, such as the large language models (LLMs) we are familiar with, multimodal models (VLM) that handle multimodal inputs like images, videos, and audio, speech-to-text transcription models, and real-time models that support streaming input. Their common feature is the ability to generate text. Taking it a step further, vLLM-Omni supports the generation of multimodal content, including images, videos, and audio.

16 +

17 +As the capabilities of generative models continue to improve, the boundaries of these models are also constantly expanding. However, certain application scenarios still require specialized small language models to efficiently complete specific tasks. These models typically have the following characteristics:

18 +

19 +- They do not require content generation.

20 +- They only need to perform very limited functions, without requiring strong generalization, creativity, or high intelligence.

21 +- They demand extremely low latency and may operate on cost-constrained hardware.

22 +- Text-only models typically have fewer than 1 billion parameters, while multimodal models generally have fewer than 10 billion parameters.

23 +

24 +Although these models are relatively small in scale, they are still based on the Transformer architecture, similar or even identical to the most advanced large language models today. Many recently released pooling models are also fine-tuned from large language models, allowing them to benefit from the continuous improvements in large models. This architecture similarity enables them to reuse much of vLLM’s infrastructure. If compatible, we would be happy to help them leverage the latest features of vLLM as well.

25 +

26 +### Sequence-wise Task and Token-wise Task

27 +

28 +The key distinction between sequence-wise task and token-wise task lies in their output granularity: sequence-wise task produces a single result for an entire input sequence, whereas token-wise task yields a result for each individual token within the sequence.

29 +

30 +Of course, we also have "plugin" tasks that allow users to customize input and output processors. For more information, please refer to [IO Processor Plugins](../../design/io_processor_plugins.md).

31 +

32 +### Pooling Tasks

33 +

34 +| Pooling Tasks | Granularity | Outputs |

35 +|--------------------|---------------|-------------------------------------------------|

36 +| `classify` | Sequence-wise | probability vector of classes for each sequence |

37 +| `score` (see note) | Sequence-wise | reranker score for each sequence |

38 +| `embed` | Sequence-wise | vector representations for each sequence |

39 +| `token_classify` | Token-wise | probability vector of classes for each token |

40 +| `token_embed` | Token-wise | vector representations for each token |

41 +

42 +!!! note

43 + Within classification tasks, there is a specialized subcategory: Cross-encoder (aka reranker) models. These models are a subset of classification models that accept two prompts as input and output num_labels equal to 1.

44 +

45 +### Score Types

46 +

48 +|--------------------|---------------|-------------------------------------------------|--------------------|--------------------------|

49 +| `classify` | Sequence-wise | probability vector of classes for each sequence | nan | nan |

52 +| `token_classify` | Token-wise | probability vector of classes for each token | nan | nan |

54 +

55 +The score models is designed to compute similarity scores between two input prompts. It supports three model types (aka `score_type`): `cross-encoder`, `late-interaction`, and `bi-encoder`.

56 +

57 +### Pooling Usages

58 +

59 +| Pooling Usages | Description |

60 +|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|

61 +| Classification Usages | Predicting which predefined category, class, or label best corresponds to a given input. |

62 +| Embedding Usages | Converts unstructured data (text, images, audio, etc.) into structured numerical vectors (embeddings). |

63 +| Token Classification Usages | Token-wise classification |

64 +| Token Embedding Usages | Token-wise embedding |

65 +| Scoring Usages | Computes similarity scores between two inputs. It supports three model types (aka `score_type`): `cross-encoder`, `late-interaction`, and `bi-encoder`. |

66 +| Reward Usages | Evaluates the quality of outputs generated by a language model, acting as a proxy for human preferences. |

67 +

68 +We also have some special models that support multiple pooling tasks, or have specific usage scenarios, or support special inputs and outputs.

69 +

70 +For more detailed information, please refer to the link below.

71 +

72 +- [Classification Usages](classify.md)

73 +- [Embedding Usages](embed.md)

74 +- [Reward Usages](reward.md)

75 +- [Token Classification Usages](token_classify.md)

76 +- [Token Embedding Usages](token_embed.md)

77 +- [Scoring Usages](scoring.md)

78 +- [Specific Model Examples](specific_models.md)

79 +

80 +## Offline Inference

81 +

82 +Each pooling model in vLLM supports one or more of these tasks according to

83 +[Pooler.get_supported_tasks][vllm.model_executor.layers.pooler.Pooler.get_supported_tasks],

84 +enabling the corresponding APIs.

85 +

86 +### Offline APIs corresponding to pooling tasks

87 +

88 +| Task | APIs |

89 +|------------------|----------------------------------------------------------------------------|

90 +| `embed` | `LLM.embed(...)`,`LLM.encode(..., pooling_task="embed")`, `LLM.score(...)` |

91 +| `classify` | `LLM.classify(...)`, `LLM.encode(..., pooling_task="classify")` |

92 +| `score` | `LLM.score(...)` |

93 +| `token_classify` | `LLM.reward(...)`, `LLM.encode(..., pooling_task="token_classify")` |

94 +| `token_embed` | `LLM.encode(..., pooling_task="token_embed")`, `LLM.score(...)` |

95 +| `plugin` | `LLM.encode(..., pooling_task="plugin")` |

96 +

97 +### `LLM.classify`

98 +

99 +The [classify][vllm.LLM.classify] method outputs a probability vector for each prompt.

100 +It is primarily designed for [classification models](classify.md).

101 +For more information about `LLM.embed`, see [this page](classify.md#offline-inference).

102 +

103 +### `LLM.embed`

104 +

105 +The [embed][vllm.LLM.embed] method outputs an embedding vector for each prompt.

106 +It is primarily designed for [embedding models](embed.md).

107 +For more information about `LLM.embed`, see [this page](embed.md#offline-inference).

108 +

109 +### `LLM.score`

110 +

111 +The [score][vllm.LLM.score] method outputs similarity scores between sentence pairs.

112 +It is primarily designed for [score models](scoring.md).

113 +

114 +### `LLM.encode`

115 +

116 +The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.

117 +

118 +Please use one of the more specific methods or set the task directly when using `LLM.encode`, refer to the [table above](#offline-apis-corresponding-to-pooling-tasks).

119 +

120 +### Examples

121 +

122 +```python

123 +from vllm import LLM

124 +

125 +llm = LLM(model="intfloat/e5-small", runner="pooling")

126 +(output,) = llm.encode("Hello, my name is", pooling_task="embed")

127 +

128 +data = output.outputs.data

129 +print(f"Data: {data!r}")

130 +```

131 +

132 +## Online Serving

133 +

134 +Our online Server provides endpoints that correspond to the offline APIs:

135 +

136 +- Corresponding to `LLM.embed`:

137 + - [Cohere Embed API](embed.md#cohere-embed-api) (`/v2/embed`)

138 + - [Openai-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)

139 +- Corresponding to `LLM.classify`:

140 + - [Classification API](classify.md#online-serving)(`/classify`)

141 +- Corresponding to `LLM.score`:

142 + - [Score API](scoring.md#score-api)(`/score`)

143 + - [Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)

144 +- Pooling API (`/pooling`) is similar to `LLM.encode`, being applicable to all types of pooling models.

145 +

146 +The following introduces the Pooling API. For other APIs, please refer to the link above.

147 +

148 +### Pooling API

149 +

150 +Our Pooling API (`/pooling`) is similar to `LLM.encode`, being applicable to all types of pooling models.

151 +

152 +The input format is the same as [Embeddings API](embed.md#openai-compatible-embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.

153 +

154 +Please use one of the more specific APIs or set the task directly when using the Pooling API, refer to the [table above](#offline-apis-corresponding-to-pooling-tasks).

155 +

156 +Code example: [examples/pooling/pooling/pooling_online.py](../../../examples/pooling/pooling/pooling_online.py)

157 +

158 +### Examples

159 +

160 +```python

161 +# start a supported embeddings model server with `vllm serve`, e.g.

162 +# vllm serve intfloat/e5-small

163 +import requests

164 +

165 +host = "localhost"

166 +port = "8000"

167 +model_name = "intfloat/e5-small"

168 +

169 +api_url = f"http://{host}:{port}/pooling"

170 +

171 +prompts = [

172 + "Hello, my name is",

173 + "The president of the United States is",

174 + "The capital of France is",

175 + "The future of AI is",

176 +]

177 +prompt = {"model": model_name, "input": prompts, "task": "embed"}

178 +

179 +response = requests.post(api_url, json=prompt)

180 +

181 +for output in response.json()["data"]:

182 + data = output["data"]

183 + print(f"Data: {data!r} (size={len(data)})")

184 +```

185 +

186 +## Configuration

187 +

188 +In vLLM, pooling models implement the [VllmModelForPooling][vllm.model_executor.models.VllmModelForPooling] interface.

189 +These models use a [Pooler][vllm.model_executor.layers.pooler.Pooler] to extract the final hidden states of the input

190 +before returning them.

191 +

192 +### Model Runner

193 +

194 +Run a model in pooling mode via the option `--runner pooling`.

195 +

196 +!!! tip

197 + There is no need to set this option in the vast majority of cases as vLLM can automatically

198 + detect the appropriate model runner via `--runner auto`.

199 +

200 +### Model Conversion

201 +

202 +vLLM can adapt models for various pooling tasks via the option `--convert <type>`.

203 +

204 +If `--runner pooling` has been set (manually or automatically) but the model does not implement the

205 +[VllmModelForPooling][vllm.model_executor.models.VllmModelForPooling] interface,

206 +vLLM will attempt to automatically convert the model according to the architecture names

207 +shown in the table below.

208 +

209 +| Architecture | `--convert` | Supported pooling tasks |

210 +| ----------------------------------------------- | ----------- | ------------------------------------- |

211 +| `*ForTextEncoding`, `*EmbeddingModel`, `*Model` | `embed` | `token_embed`, `embed` |

212 +| `*ForRewardModeling`, `*RewardModel` | `embed` | `token_embed`, `embed` |

213 +| `*For*Classification`, `*ClassificationModel` | `classify` | `token_classify`, `classify`, `score` |

214 +

215 +!!! tip

216 + You can explicitly set `--convert <type>` to specify how to convert the model.

217 +

218 +### Pooler Configuration

219 +

220 +#### Predefined models

221 +

222 +If the [Pooler][vllm.model_executor.layers.pooler.Pooler] defined by the model accepts `pooler_config`,

223 +you can override some of its attributes via the `--pooler-config` option.

224 +

225 +#### Converted models

226 +

227 +If the model has been converted via `--convert` (see above),

228 +the pooler assigned to each task has the following attributes by default:

229 +

231 +| ---------- | ------------ | ------------- | ------- |

232 +| `embed` | `LAST` | ✅︎ | ❌ |

233 +| `classify` | `LAST` | ❌ | ✅︎ |

234 +

235 +When loading [Sentence Transformers](https://huggingface.co/sentence-transformers) models,

236 +its Sentence Transformers configuration file (`modules.json`) takes priority over the model's defaults.

237 +

238 +You can further customize this via the `--pooler-config` option,

239 +which takes priority over both the model's and Sentence Transformers' defaults.

240 +

241 +## Removed Features

242 +

243 +### Encode task

244 +

245 +We have split the `encode` task into two more specific token-wise tasks: `token_embed` and `token_classify`:

246 +

247 +- `token_embed` is the same as `embed`, using normalization as the activation.

248 +- `token_classify` is the same as `classify`, by default using softmax as the activation.

249 +

250 +Pooling models now default support all pooling, you can use it without any settings.

251 +

252 +- Extracting hidden states prefers using `token_embed` task.

253 +- Named Entity Recognition (NER) and reward models prefers using `token_classify` task.

docs/models/pooling_models/classify.md new

+276

@@ -0,0 +1,276 @@

1 +# Classification Usages

2 +

3 +Classification involves predicting which predefined category, class, or label best corresponds to a given input.

4 +

5 +## Summary

6 +

7 +- Model Usage: (sequence) classification

8 +- Pooling Task: `classify`

9 +- Offline APIs:

10 + - `LLM.classify(...)`

11 + - `LLM.encode(..., pooling_task="classify")`

12 +- Online APIs:

13 + - [Classification API](classify.md#online-serving) (`/classify`)

14 + - Pooling API (`/pooling`)

15 +

16 +The key distinction between (sequence) classification and token classification lies in their output granularity: (sequence) classification produces a single result for an entire input sequence, whereas token classification yields a result for each individual token within the sequence.

17 +

18 +Many classification models support both (sequence) classification and token classification. For further details on token classification, please refer to [this page](token_classify.md).

19 +

20 +## Typical Use Cases

21 +

22 +### Classification

23 +

24 +The most fundamental application of classification models is to categorize input data into predefined classes.

25 +

26 +## Supported Models

27 +

28 +### Text-only Models

29 +

31 +| ------------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |

33 +| `GPT2ForSequenceClassification` | GPT2 | `nie3e/sentiment-polish-gpt2-small` | | |

35 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

36 +

37 +### Multimodal Models

38 +

39 +!!! note

40 + For more information about multimodal models inputs, see [this page](../supported_models.md#list-of-multimodal-language-models).

41 +

43 +| ------------ | ------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |

45 +| `*ForConditionalGeneration`C, `*ForCausalLM`C, etc. | Generative models | \* | N/A | \* | \* |

46 +

47 +C Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))

48 +\* Feature support is the same as that of the original model.

49 +

50 +If your model is not in the above list, we will try to automatically convert the model using

51 +[as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.

52 +

53 +### Cross-encoder Models

54 +

55 +Cross-encoder (aka reranker) models are a subset of classification models that accept two prompts as input and output num_labels equal to 1. Most classification models can also be used as [cross-encoder models](scoring.md#cross-encoder-models). For more information on cross-encoder models, please refer to [this page](scoring.md).

56 +

57 +--8<-- "docs/models/pooling_models/scoring.md:supported-score-models"

58 +

59 +### Reward Models

60 +

61 +Using (sequence) classification models as reward models. For more information, see [Reward Models](reward.md).

62 +

63 +--8<-- "docs/models/pooling_models/reward.md:supported-sequence-reward-models"

64 +

65 +## Offline Inference

66 +

67 +### Pooling Parameters

68 +

69 +The following [pooling parameters][vllm.PoolingParams] are supported.

70 +

71 +```python

72 +--8<-- "vllm/pooling_params.py:common-pooling-params"

73 +--8<-- "vllm/pooling_params.py:classify-pooling-params"

74 +```

75 +

76 +### `LLM.classify`

77 +

78 +The [classify][vllm.LLM.classify] method outputs a probability vector for each prompt.

79 +

80 +```python

81 +from vllm import LLM

82 +

83 +llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", runner="pooling")

84 +(output,) = llm.classify("Hello, my name is")

85 +

86 +probs = output.outputs.probs

87 +print(f"Class Probabilities: {probs!r} (size={len(probs)})")

88 +```

89 +

90 +A code example can be found here: [examples/offline_inference/basic/classify.py](../../../examples/basic/offline_inference/classify.py)

91 +

92 +### `LLM.encode`

93 +

94 +The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.

95 +

96 +Set `pooling_task="classify"` when using `LLM.encode` for classification Models:

97 +

98 +```python

99 +from vllm import LLM

100 +

101 +llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", runner="pooling")

102 +(output,) = llm.encode("Hello, my name is", pooling_task="classify")

103 +

104 +data = output.outputs.data

105 +print(f"Data: {data!r}")

106 +```

107 +

108 +## Online Serving

109 +

110 +### Classification API

111 +

112 +Online `/classify` API is similar to `LLM.classify`.

113 +

114 +#### Completion Parameters

115 +

116 +The following Classification API parameters are supported:

117 +

118 +??? code

119 +

120 + ```python

121 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

122 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-params"

123 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-params"

124 + ```

125 +

126 +The following extra parameters are supported:

127 +

128 +??? code

129 +

130 + ```python

131 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

132 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-extra-params"

133 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

134 + ```

135 +

136 +#### Chat Parameters

137 +

138 +For chat-like input (i.e. if `messages` is passed), the following parameters are supported:

139 +

140 +??? code

141 +

142 + ```python

143 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

144 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-params"

145 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-params"

146 + ```

147 +

148 +these extra parameters are supported instead:

149 +

150 +??? code

151 +

152 + ```python

153 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

154 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-extra-params"

155 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

156 + ```

157 +

158 +#### Example Requests

159 +

160 +Code example: [examples/pooling/classify/classification_online.py](../../../examples/pooling/classify/classification_online.py)

161 +

162 +You can classify multiple texts by passing an array of strings:

163 +

164 +```bash

165 +curl -v "http://127.0.0.1:8000/classify" \

166 + -H "Content-Type: application/json" \

167 + -d '{

168 + "model": "jason9693/Qwen2.5-1.5B-apeach",

169 + "input": [

170 + "Loved the new café—coffee was great.",

171 + "This update broke everything. Frustrating."

172 + ]

173 + }'

174 +```

175 +

176 +??? console "Response"

177 +

178 + ```json

179 + {

180 + "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",

181 + "object": "list",

182 + "created": 1745383065,

183 + "model": "jason9693/Qwen2.5-1.5B-apeach",

184 + "data": [

185 + {

186 + "index": 0,

187 + "label": "Default",

188 + "probs": [

189 + 0.565970778465271,

190 + 0.4340292513370514

191 + ],

192 + "num_classes": 2

193 + },

194 + {

195 + "index": 1,

196 + "label": "Spoiled",

197 + "probs": [

198 + 0.26448777318000793,

199 + 0.7355121970176697

200 + ],

201 + "num_classes": 2

202 + }

203 + ],

204 + "usage": {

205 + "prompt_tokens": 20,

206 + "total_tokens": 20,

207 + "completion_tokens": 0,

208 + "prompt_tokens_details": null

209 + }

210 + }

211 + ```

212 +

213 +You can also pass a string directly to the `input` field:

214 +

215 +```bash

216 +curl -v "http://127.0.0.1:8000/classify" \

217 + -H "Content-Type: application/json" \

218 + -d '{

219 + "model": "jason9693/Qwen2.5-1.5B-apeach",

220 + "input": "Loved the new café—coffee was great."

221 + }'

222 +```

223 +

224 +??? console "Response"

225 +

226 + ```json

227 + {

228 + "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",

229 + "object": "list",

230 + "created": 1745383213,

231 + "model": "jason9693/Qwen2.5-1.5B-apeach",

232 + "data": [

233 + {

234 + "index": 0,

235 + "label": "Default",

236 + "probs": [

237 + 0.565970778465271,

238 + 0.4340292513370514

239 + ],

240 + "num_classes": 2

241 + }

242 + ],

243 + "usage": {

244 + "prompt_tokens": 10,

245 + "total_tokens": 10,

246 + "completion_tokens": 0,

247 + "prompt_tokens_details": null

248 + }

249 + }

250 + ```

251 +

252 +## More examples

253 +

254 +More examples can be found here: [examples/pooling/classify](../../../examples/pooling/classify)

255 +

256 +## Supported Features

257 +

258 +### Enable/disable activation

259 +

260 +You can enable or disable activation via `use_activation`.

261 +

262 +### Problem type (e.g. `multi_label_classification`)

263 +

264 +You can modify the `problem_type` via problem_type in the Hugging Face config. The supported problem types are: `single_label_classification`, `multi_label_classification`, and `regression`.

265 +

266 +Implement alignment with transformers [ForSequenceClassificationLoss](https://github.com/huggingface/transformers/blob/57bb6db6ee4cfaccc45b8d474dfad5a17811ca60/src/transformers/loss/loss_utils.py#L92).

267 +

268 +### Logit bias

269 +

270 +You can modify the `logit_bias` (aka `sigmoid_normalize`) through the logit_bias parameter in `vllm.config.PoolerConfig`.

271 +

272 +## Removed Features

273 +

274 +### Remove softmax from PoolingParams

275 +

276 +We have already removed `softmax` and `activation` from PoolingParams. Instead, use `use_activation`, since we allow `classify` and `token_classify` to use any activation function.

docs/models/pooling_models/embed.md new

+546

@@ -0,0 +1,546 @@

1 +# Embedding Usages

2 +

3 +Embedding models are a class of machine learning models designed to transform unstructured data—such as text, images, or audio—into a structured numerical representation known as an embedding.

4 +

5 +## Summary

6 +

7 +- Model Usage: (sequence) embedding

8 +- Pooling Task: `embed`

9 +- Offline APIs:

10 + - `LLM.embed(...)`

11 + - `LLM.encode(..., pooling_task="embed")`

12 + - `LLM.score(...)`

13 +- Online APIs:

14 + - [Cohere Embed API](embed.md#cohere-embed-api) (`/v2/embed`)

15 + - [Openai-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)

16 + - Pooling API (`/pooling`)

17 +

18 +The primary distinction between (sequence) embedding and token embedding lies in their output granularity: (sequence) embedding produces a single embedding vector for an entire input sequence, whereas token embedding generates an embedding for each individual token within the sequence.

19 +

20 +Many embedding models support both (sequence) embedding and token embedding. For further details on token embedding, please refer to [this page](token_embed.md).

21 +

22 +## Typical Use Cases

23 +

24 +### Embedding

25 +

26 +The most basic use case of embedding models is to embed the inputs, e.g. for RAG.

27 +

28 +### Pairwise Similarity

29 +

30 +You can compute pairwise similarity scores to build a similarity matrix using the [Score API](scoring.md).

31 +

32 +## Supported Models

33 +

34 +--8<-- [start:supported-embed-models]

35 +

36 +### Text-only Models

37 +

39 +| ------------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |

43 +| `Gemma2Model`C | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | ✅︎ |

44 +| `Gemma3TextModel`C | Gemma 3-based | `google/embeddinggemma-300m`, etc. | ✅︎ | ✅︎ |

45 +| `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ |

48 +| `LlamaBidirectionalModel`C | Llama-based with bidirectional attention | `nvidia/llama-nemotron-embed-1b-v2`, etc. | ✅︎ | ✅︎ |

49 +| `LlamaModel`C, `LlamaForCausalLM`C, `MistralModel`C, etc. | Llama-based | `intfloat/e5-mistral-7b-instruct`, etc. | ✅︎ | ✅︎ |

52 +| `Qwen2Model`C, `Qwen2ForCausalLM`C | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ |

53 +| `Qwen3Model`C, `Qwen3ForCausalLM`C | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, etc. | ✅︎ | ✅︎ |

55 +| `VoyageQwen3BidirectionalEmbedModel`C | Voyage Qwen3-based with bidirectional attention | `voyageai/voyage-4-nano`, etc. | ✅︎ | ✅︎ |

57 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

58 +

59 +!!! note

60 + The second-generation GTE model (mGTE-TRM) is named `NewModel`. The name `NewModel` is too generic, you should set `--hf-overrides '{"architectures": ["GteNewModel"]}'` to specify the use of the `GteNewModel` architecture.

61 +

62 +!!! note

63 + `ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.

64 + You need to manually set mean pooling by passing `--pooler-config '{"pooling_type": "MEAN"}'`.

65 +

66 +!!! note

67 + For `Alibaba-NLP/gte-Qwen2-*`, you need to enable `--trust-remote-code` for the correct tokenizer to be loaded.

68 + See [relevant issue on HF Transformers](https://github.com/huggingface/transformers/issues/34882).

69 +

70 +!!! note

71 + The `BAAI/bge-m3` model comes with extra weights for sparse and colbert embeddings, See [this page](specific_models.md#baaibge-m3) for more information.

72 +

73 +!!! note

74 + `jinaai/jina-embeddings-v3` supports multiple tasks through LoRA, while vllm temporarily only supports text-matching tasks by merging LoRA weights.

75 +

76 +### Multimodal Models

77 +

78 +!!! note

79 + For more information about multimodal models inputs, see [this page](../supported_models.md#list-of-multimodal-language-models).

80 +

82 +| ------------ | ------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |

85 +| `LlavaNextForConditionalGeneration`C | LLaVA-NeXT-based | T / I | `royokong/e5-v` | | ✅︎ |

86 +| `Phi3VForCausalLM`C | Phi-3-Vision-based | T + I | `TIGER-Lab/VLM2Vec-Full` | | ✅︎ |

87 +| `Qwen3VLForConditionalGeneration`C | Qwen3-VL | T + I + V | `Qwen/Qwen3-VL-Embedding-2B`, etc. | ✅︎ | ✅︎ |

89 +| `*ForConditionalGeneration`C, `*ForCausalLM`C, etc. | Generative models | \* | N/A | \* | \* |

90 +

91 +C Automatically converted into an embedding model via `--convert embed`. ([details](./README.md#model-conversion))

92 +\* Feature support is the same as that of the original model.

93 +

94 +If your model is not in the above list, we will try to automatically convert the model using

95 +[as_embedding_model][vllm.model_executor.models.adapters.as_embedding_model]. By default, the embeddings

96 +of the whole prompt are extracted from the normalized hidden state corresponding to the last token.

97 +

98 +!!! note

99 + Although vLLM supports automatically converting models of any architecture into embedding models via --convert embed, to get the best results, you should use pooling models that are specifically trained as such.

100 +

101 +--8<-- [end:supported-embed-models]

102 +

103 +## Offline Inference

104 +

105 +### Pooling Parameters

106 +

107 +The following [pooling parameters][vllm.PoolingParams] are supported.

108 +

109 +```python

110 +--8<-- "vllm/pooling_params.py:common-pooling-params"

111 +--8<-- "vllm/pooling_params.py:embed-pooling-params"

112 +```

113 +

114 +### `LLM.embed`

115 +

116 +The [embed][vllm.LLM.embed] method outputs an embedding vector for each prompt.

117 +

118 +```python

119 +from vllm import LLM

120 +

121 +llm = LLM(model="intfloat/e5-small", runner="pooling")

122 +(output,) = llm.embed("Hello, my name is")

123 +

124 +embeds = output.outputs.embedding

125 +print(f"Embeddings: {embeds!r} (size={len(embeds)})")

126 +```

127 +

128 +A code example can be found here: [examples/offline_inference/basic/embed.py](../../../examples/basic/offline_inference/embed.py)

129 +

130 +### `LLM.encode`

131 +

132 +The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.

133 +

134 +Set `pooling_task="embed"` when using `LLM.encode` for embedding Models:

135 +

136 +```python

137 +from vllm import LLM

138 +

139 +llm = LLM(model="intfloat/e5-small", runner="pooling")

140 +(output,) = llm.encode("Hello, my name is", pooling_task="embed")

141 +

142 +data = output.outputs.data

143 +print(f"Data: {data!r}")

144 +```

145 +

146 +### `LLM.score`

147 +

148 +The [score][vllm.LLM.score] method outputs similarity scores between sentence pairs.

149 +

150 +All models that support embedding task also support using the score API to compute similarity scores by calculating the cosine similarity of two input prompt's embeddings.

151 +

152 +```python

153 +from vllm import LLM

154 +

155 +llm = LLM(model="intfloat/e5-small", runner="pooling")

156 +(output,) = llm.score(

157 + "What is the capital of France?",

158 + "The capital of Brazil is Brasilia.",

159 +)

160 +

161 +score = output.outputs.score

162 +print(f"Score: {score}")

163 +```

164 +

165 +## Online Serving

166 +

167 +### OpenAI-Compatible Embeddings API

168 +

169 +Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);

170 +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

171 +

172 +Code example: [examples/pooling/embed/openai_embedding_client.py](../../../examples/pooling/embed/openai_embedding_client.py)

173 +

174 +#### Completion Parameters

175 +

176 +The following Classification API parameters are supported:

177 +

178 +??? code

179 +

180 + ```python

181 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

182 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-params"

183 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-params"

184 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-params"

185 + ```

186 +

187 +The following extra parameters are supported:

188 +

189 +??? code

190 +

191 + ```python

192 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

193 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-extra-params"

194 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-extra-params"

195 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-extra-params"

196 + ```

197 +

198 +#### Chat Parameters

199 +

200 +For chat-like input (i.e. if `messages` is passed), the following parameters are supported:

201 +

202 +??? code

203 +

204 + ```python

205 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

206 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-params"

207 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-params"

208 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-params"

209 + ```

210 +

211 +these extra parameters are supported instead:

212 +

213 +??? code

214 +

215 + ```python

216 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

217 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-extra-params"

218 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-extra-params"

219 + --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-extra-params"

220 + ```

221 +

222 +#### Examples

223 +

224 +If the model has a [chat template](../../serving/openai_compatible_server.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](../../serving/openai_compatible_server.md#chat-api))

225 +which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations:

226 +

227 +??? code

228 +

229 + ```python

230 + from openai import OpenAI

231 + from openai._types import NOT_GIVEN, NotGiven

232 + from openai.types.chat import ChatCompletionMessageParam

233 + from openai.types.create_embedding_response import CreateEmbeddingResponse

234 +

235 + def create_chat_embeddings(

236 + client: OpenAI,

237 + *,

238 + messages: list[ChatCompletionMessageParam],

239 + model: str,

240 + encoding_format: Union[Literal["base64", "float"], NotGiven] = NOT_GIVEN,

241 + ) -> CreateEmbeddingResponse:

242 + return client.post(

243 + "/embeddings",

244 + cast_to=CreateEmbeddingResponse,

245 + body={"messages": messages, "model": model, "encoding_format": encoding_format},

246 + )

247 + ```

248 +

249 +##### Multi-modal inputs

250 +

251 +You can pass multi-modal inputs to embedding models by defining a custom chat template for the server

252 +and passing a list of `messages` in the request. Refer to the examples below for illustration.

253 +

254 +=== "VLM2Vec"

255 +

256 + To serve the model:

257 +

258 + ```bash

259 + vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \

260 + --trust-remote-code \

261 + --max-model-len 4096 \

262 + --chat-template examples/pooling/embed/template/vlm2vec_phi3v.jinja

263 + ```

264 +

265 + !!! important

266 + Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--runner pooling`

267 + to run this model in embedding mode instead of text generation mode.

268 +

269 + The custom chat template is completely different from the original one for this model,

270 + and can be found here: [examples/pooling/embed/template/vlm2vec_phi3v.jinja](../../../examples/pooling/embed/template/vlm2vec_phi3v.jinja)

271 +

272 + Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

273 +

274 + ??? code

275 +

276 + ```python

277 + from openai import OpenAI

278 + client = OpenAI(

279 + base_url="http://localhost:8000/v1",

280 + api_key="EMPTY",

281 + )

282 + image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

283 +

284 + response = create_chat_embeddings(

285 + client,

286 + model="TIGER-Lab/VLM2Vec-Full",

287 + messages=[

288 + {

289 + "role": "user",

290 + "content": [

291 + {"type": "image_url", "image_url": {"url": image_url}},

292 + {"type": "text", "text": "Represent the given image."},

293 + ],

294 + }

295 + ],

296 + encoding_format="float",

297 + )

298 +

299 + print("Image embedding output:", response.data[0].embedding)

300 + ```

301 +

302 +=== "DSE-Qwen2-MRL"

303 +

304 + To serve the model:

305 +

306 + ```bash

307 + vllm serve MrLight/dse-qwen2-2b-mrl-v1 --runner pooling \

308 + --trust-remote-code \

309 + --max-model-len 8192 \

310 + --chat-template examples/pooling/embed/template/dse_qwen2_vl.jinja

311 + ```

312 +

313 + !!! important

314 + Like with VLM2Vec, we have to explicitly pass `--runner pooling`.

315 +

316 + Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled

317 + by a custom chat template: [examples/pooling/embed/template/dse_qwen2_vl.jinja](../../../examples/pooling/embed/template/dse_qwen2_vl.jinja)

318 +

319 + !!! important

320 + `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code

321 + example below for details.

322 +

323 +Full example: [examples/pooling/embed/vision_embedding_online.py](../../../examples/pooling/embed/vision_embedding_online.py)

324 +

325 +### Cohere Embed API

326 +

327 +Our API is also compatible with [Cohere's Embed v2 API](https://docs.cohere.com/reference/embed) which adds support for some modern embedding feature such as truncation, output dimensions, embedding types, and input types. This endpoint works with any embedding model (including multimodal models).

328 +

329 +#### Cohere Embed API request parameters

330 +

332 +| --------- | ---- | -------- | ----------- |

339 +| `output_dimension` | int | No | Truncate embeddings to this dimension (Matryoshka) |

341 +

342 +#### Text embedding

343 +

344 +```bash

345 +curl -X POST "http://localhost:8000/v2/embed" \

346 + -H "Content-Type: application/json" \

347 + -d '{

348 + "model": "Snowflake/snowflake-arctic-embed-m-v1.5",

349 + "input_type": "query",

350 + "texts": ["Hello world", "How are you?"],

351 + "embedding_types": ["float"]

352 + }'

353 +```

354 +

355 +??? console "Response"

356 +

357 + ```json

358 + {

359 + "id": "embd-...",

360 + "embeddings": {

361 + "float": [

362 + [0.012, -0.034, ...],

363 + [0.056, 0.078, ...]

364 + ]

365 + },

366 + "texts": ["Hello world", "How are you?"],

367 + "meta": {

368 + "api_version": {"version": "2"},

369 + "billed_units": {"input_tokens": 12}

370 + }

371 + }

372 + ```

373 +

374 +#### Mixed text and image inputs

375 +

376 +For multimodal models, you can embed images by passing base64 data URIs. The `inputs` field accepts a list of objects with mixed text and image content:

377 +

378 +```bash

379 +curl -X POST "http://localhost:8000/v2/embed" \

380 + -H "Content-Type: application/json" \

381 + -d '{

382 + "model": "google/siglip-so400m-patch14-384",

383 + "inputs": [

384 + {

385 + "content": [

386 + {"type": "text", "text": "A photo of a cat"},

387 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBOR..."}}

388 + ]

389 + }

390 + ],

391 + "embedding_types": ["float"]

392 + }'

393 +```

394 +

395 +#### Embedding types

396 +

397 +The `embedding_types` parameter controls the output format. Multiple types can be requested in a single call:

398 +

399 +| Type | Description |

400 +| ---- | ----------- |

401 +| `float` | Raw float32 embeddings (default) |

402 +| `binary` | Bit-packed signed binary |

403 +| `ubinary` | Bit-packed unsigned binary |

404 +| `base64` | Little-endian float32 encoded as base64 |

405 +

406 +```bash

407 +curl -X POST "http://localhost:8000/v2/embed" \

408 + -H "Content-Type: application/json" \

409 + -d '{

410 + "model": "Snowflake/snowflake-arctic-embed-m-v1.5",

411 + "input_type": "query",

412 + "texts": ["What is machine learning?"],

413 + "embedding_types": ["float", "binary"]

414 + }'

415 +```

416 +

417 +??? console "Response"

418 +

419 + ```json

420 + {

421 + "id": "embd-...",

422 + "embeddings": {

423 + "float": [[0.012, -0.034, ...]],

424 + "binary": [[42, -117, ...]]

425 + },

426 + "texts": ["What is machine learning?"],

427 + "meta": {

428 + "api_version": {"version": "2"},

429 + "billed_units": {"input_tokens": 8}

430 + }

431 + }

432 + ```

433 +

434 +#### Truncation

435 +

436 +The `truncate` parameter controls how inputs exceeding the model's maximum sequence length are handled:

437 +

438 +| Value | Behavior |

439 +| ----- | --------- |

440 +| `END` (default) | Keep the first tokens, drop the end |

441 +| `START` | Keep the last tokens, drop the beginning |

442 +| `NONE` | Return an error if the input is too long |

443 +

444 +#### Input type and prompt prefixes

445 +

446 +The `input_type` field selects a prompt prefix to prepend to each text input. The available values

447 +depend on the model:

448 +

449 +- **Models with `task_instructions` in `config.json`**: The keys from the `task_instructions` dict are

450 + the valid `input_type` values and the corresponding value is prepended to each text.

451 +- **Models with `config_sentence_transformers.json` prompts**: The keys from the `prompts` dict are

452 + the valid `input_type` values. For example, `Snowflake/snowflake-arctic-embed-xs` defines `"query"`,

453 + so setting `input_type: "query"` prepends `"Represent this sentence for searching relevant passages: "`.

454 +- **Other models**: `input_type` is not accepted and will raise a validation error if passed.

455 +

456 +## More examples

457 +

458 +More examples can be found here: [examples/pooling/embed](../../../examples/pooling/embed)

459 +

460 +## Supported Features

461 +

462 +### Enable/disable normalize

463 +

464 +You can enable or disable normalize via `use_activation`.

465 +

466 +### Matryoshka Embeddings

467 +

468 +[Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows users to trade off between performance and cost.

469 +

470 +!!! warning

471 + Not all embedding models are trained using Matryoshka Representation Learning. To avoid misuse of the `dimensions` parameter, vLLM returns an error for requests that attempt to change the output dimension of models that do not support Matryoshka Embeddings.

472 +

473 + For example, setting `dimensions` parameter while using the `BAAI/bge-m3` model will result in the following error.

474 +

475 + ```json

476 + {"object":"error","message":"Model \"BAAI/bge-m3\" does not support matryoshka representation, changing output dimensions will lead to poor results.","type":"BadRequestError","param":null,"code":400}

477 + ```

478 +

479 +#### Manually enable Matryoshka Embeddings

480 +

481 +There is currently no official interface for specifying support for Matryoshka Embeddings. In vLLM, if `is_matryoshka` is `True` in `config.json`, you can change the output dimension to arbitrary values. Use `matryoshka_dimensions` to control the allowed output dimensions.

482 +

483 +For models that support Matryoshka Embeddings but are not recognized by vLLM, manually override the config using `hf_overrides={"is_matryoshka": True}` or `hf_overrides={"matryoshka_dimensions": [<allowed output dimensions>]}` (offline), or `--hf-overrides '{"is_matryoshka": true}'` or `--hf-overrides '{"matryoshka_dimensions": [<allowed output dimensions>]}'` (online).

484 +

485 +Here is an example to serve a model with Matryoshka Embeddings enabled.

486 +

487 +```bash

488 +vllm serve Snowflake/snowflake-arctic-embed-m-v1.5 --hf-overrides '{"matryoshka_dimensions":[256]}'

489 +```

490 +

491 +#### Offline Inference

492 +

493 +You can change the output dimensions of embedding models that support Matryoshka Embeddings by using the dimensions parameter in [PoolingParams][vllm.PoolingParams].

494 +

495 +```python

496 +from vllm import LLM, PoolingParams

497 +

498 +llm = LLM(

499 + model="jinaai/jina-embeddings-v3",

500 + runner="pooling",

501 + trust_remote_code=True,

502 +)

503 +outputs = llm.embed(

504 + ["Follow the white rabbit."],

505 + pooling_params=PoolingParams(dimensions=32),

506 +)

507 +print(outputs[0].outputs)

508 +```

509 +

510 +A code example can be found here: [examples/pooling/embed/embed_matryoshka_fy_offline.py](../../../examples/pooling/embed/embed_matryoshka_fy_offline.py)

511 +

512 +#### Online Inference

513 +

514 +Use the following command to start the vLLM server.

515 +

516 +```bash

517 +vllm serve jinaai/jina-embeddings-v3 --trust-remote-code

518 +```

519 +

520 +You can change the output dimensions of embedding models that support Matryoshka Embeddings by using the dimensions parameter.

521 +

522 +```bash

523 +curl http://127.0.0.1:8000/v1/embeddings \

524 + -H 'accept: application/json' \

525 + -H 'Content-Type: application/json' \

526 + -d '{

527 + "input": "Follow the white rabbit.",

528 + "model": "jinaai/jina-embeddings-v3",

529 + "encoding_format": "float",

530 + "dimensions": 32

531 + }'

532 +```

533 +

534 +Expected output:

535 +

536 +```json

537 +{"id":"embd-5c21fc9a5c9d4384a1b021daccaf9f64","object":"list","created":1745476417,"model":"jinaai/jina-embeddings-v3","data":[{"index":0,"object":"embedding","embedding":[-0.3828125,-0.1357421875,0.03759765625,0.125,0.21875,0.09521484375,-0.003662109375,0.1591796875,-0.130859375,-0.0869140625,-0.1982421875,0.1689453125,-0.220703125,0.1728515625,-0.2275390625,-0.0712890625,-0.162109375,-0.283203125,-0.055419921875,-0.0693359375,0.031982421875,-0.04052734375,-0.2734375,0.1826171875,-0.091796875,0.220703125,0.37890625,-0.0888671875,-0.12890625,-0.021484375,-0.0091552734375,0.23046875]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0,"prompt_tokens_details":null}}

538 +```

539 +

540 +An OpenAI client example can be found here: [examples/pooling/embed/openai_embedding_matryoshka_fy_client.py](../../../examples/pooling/embed/openai_embedding_matryoshka_fy_client.py)

541 +

542 +## Removed Features

543 +

544 +### Remove `normalize` from PoolingParams

545 +

546 +We have already removed `normalize` from PoolingParams, use `use_activation` instead.

docs/models/pooling_models/reward.md new

+136

@@ -0,0 +1,136 @@

1 +# Reward Usages

2 +

3 +A reward model (RM) is designed to evaluate and score the quality of outputs generated by a language model, acting as a proxy for human preferences.

4 +

5 +## Summary

6 +

7 +- Model Usage: reward

8 +- Pooling Task:

9 +

10 +| Model Types | Pooling Tasks |

11 +|------------------------------------|----------------|

12 +| (sequence) (outcome) reward models | classify |

13 +| token (outcome) reward models | token_classify |

14 +| process reward models | token_classify |

15 +

16 +- Offline APIs:

17 + - `LLM.encode(..., pooling_task="...")`

18 +- Online APIs:

19 + - Pooling API (`/pooling`)

20 +

21 +## Supported Models

22 +

23 +### Reward Models

24 +

25 +Using sequence classification models as (sequence) (outcome) reward models, the usage and supported features are the same as for normal [classification models](classify.md).

26 +

27 +--8<-- [start:supported-sequence-reward-models]

28 +

30 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

31 +| `JambaForSequenceClassification` | Jamba | `ai21labs/Jamba-tiny-reward-dev`, etc. | ✅︎ | ✅︎ |

32 +| `Qwen3ForSequenceClassification`C | Qwen3-based | `Skywork/Skywork-Reward-V2-Qwen3-0.6B`, etc. | ✅︎ | ✅︎ |

33 +| `LlamaForSequenceClassification`C | Llama-based | `Skywork/Skywork-Reward-V2-Llama-3.2-1B`, etc. | ✅︎ | ✅︎ |

34 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

35 +

36 +C Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))

37 +

38 +If your model is not in the above list, we will try to automatically convert the model using

39 +[as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.

40 +

41 +--8<-- [end:supported-sequence-reward-models]

42 +

43 +### Token Reward Models

44 +

45 +The key distinction between (sequence) classification and token classification lies in their output granularity: (sequence) classification produces a single result for an entire input sequence, whereas token classification yields a result for each individual token within the sequence.

46 +

47 +Using token classification models as token (outcome) reward models, the usage and supported features are the same as for normal [token classification models](token_classify.md).

48 +

49 +--8<-- [start:supported-token-reward-models]

50 +

52 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

53 +| `InternLM2ForRewardModel` | InternLM2-based | `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc. | ✅︎ | ✅︎ |

54 +| `Qwen2ForRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-RM-72B`, etc. | ✅︎ | ✅︎ |

55 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

56 +

57 +C Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))

58 +

59 +If your model is not in the above list, we will try to automatically convert the model using

60 +[as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model].

61 +

62 +--8<-- [end:supported-token-reward-models]

63 +

64 +### Process Reward Models

65 +

66 +The process reward models used for evaluating intermediate steps are crucial to achieving the desired outcome.

67 +

69 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

70 +| `LlamaForCausalLM` | Llama-based | `peiyi9979/math-shepherd-mistral-7b-prm`, etc. | ✅︎ | ✅︎ |

71 +| `Qwen2ForProcessRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-PRM-7B`, etc. | ✅︎ | ✅︎ |

72 +

73 +!!! important

74 + For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,

75 + e.g.: `--pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.

76 +

77 +## Offline Inference

78 +

79 +### Pooling Parameters

80 +

81 +The following [pooling parameters][vllm.PoolingParams] are supported.

82 +

83 +```python

84 +--8<-- "vllm/pooling_params.py:common-pooling-params"

85 +--8<-- "vllm/pooling_params.py:classify-pooling-params"

86 +```

87 +

88 +### `LLM.encode`

89 +

90 +The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.

91 +

92 +- Reward Models

93 +

94 +Set `pooling_task="classify"` when using `LLM.encode` for (sequence) (outcome) reward models:

95 +

96 +```python

97 +from vllm import LLM

98 +

99 +llm = LLM(model="Skywork/Skywork-Reward-V2-Qwen3-0.6B", runner="pooling")

100 +(output,) = llm.encode("Hello, my name is", pooling_task="classify")

101 +

102 +data = output.outputs.data

103 +print(f"Data: {data!r}")

104 +```

105 +

106 +- Token Reward Models

107 +

108 +Set `pooling_task="token_classify"` when using `LLM.encode` for token (outcome) reward models:

109 +

110 +```python

111 +from vllm import LLM

112 +

113 +llm = LLM(model="internlm/internlm2-1_8b-reward", runner="pooling", trust_remote_code=True)

114 +(output,) = llm.encode("Hello, my name is", pooling_task="token_classify")

115 +

116 +data = output.outputs.data

117 +print(f"Data: {data!r}")

118 +```

119 +

120 +- Process Reward Models

121 +

122 +Set `pooling_task="token_classify"` when using `LLM.encode` for token (outcome) reward models:

123 +

124 +```python

125 +from vllm import LLM

126 +

127 +llm = LLM(model="Qwen/Qwen2.5-Math-PRM-7B", runner="pooling")

128 +(output,) = llm.encode("Hello, my name is<extra_0><extra_0><extra_0>", pooling_task="token_classify")

129 +

130 +data = output.outputs.data

131 +print(f"Data: {data!r}")

132 +```

133 +

134 +## Online Serving

135 +

136 +Please refer to the [pooling API](README.md#pooling-api). Pooling task corresponding to reward model types refer to the [table above](#summary).

docs/models/pooling_models/scoring.md new

+448

@@ -0,0 +1,448 @@

1 +# Scoring Usages

2 +

3 +The score models is designed to compute similarity scores between two input prompts. It supports three model types (aka `score_type`): `cross-encoder`, `late-interaction`, and `bi-encoder`.

4 +

5 +!!! note

6 + vLLM handles only the model inference component of RAG pipelines (such as embedding generation and reranking). For higher-level RAG orchestration, you should leverage integration frameworks like [LangChain](https://github.com/langchain-ai/langchain).

7 +

8 +## Summary

9 +

10 +- Model Usage: Scoring

11 +- Pooling Task:

12 +

13 +| Score Types | Pooling Tasks | scoring function |

14 +|--------------------|---------------|--------------------------|

15 +| `cross-encoder` | `score` | linear classifier |

16 +| `late-interaction` | `token_embed` | late interaction(MaxSim) |

17 +| `bi-encoder` | `embed` | cosine similarity |

18 +

19 +- Offline APIs:

20 + - `LLM.score`

21 +- Online APIs:

22 + - [Score API](scoring.md#score-api) (`/score`)

23 + - [Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)

24 +

25 +## Supported Models

26 +

27 +### Cross-encoder models

28 +

29 +[Cross-encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) (aka reranker) models are a subset of classification models that accept two prompts as input and output num_labels equal to 1.

30 +

31 +--8<-- [start:supported-score-models]

32 +

33 +#### Text-only Models

34 +

36 +| ------------ | ------ | ----------------- | ------------------------- | --------------------------- | --------------------------------------- |

38 +| `GemmaForSequenceClassification` | Gemma-based | `BAAI/bge-reranker-v2-gemma`(see note), etc. | [bge-reranker-v2-gemma.jinja](../../../examples/pooling/score/template/bge-reranker-v2-gemma.jinja) | ✅︎ | ✅︎ |

40 +| `LlamaBidirectionalForSequenceClassification`C | Llama-based with bidirectional attention | `nvidia/llama-nemotron-rerank-1b-v2`, etc. | [nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja) | ✅︎ | ✅︎ |

41 +| `Qwen2ForSequenceClassification`C | Qwen2-based | `mixedbread-ai/mxbai-rerank-base-v2`(see note), etc. | [mxbai_rerank_v2.jinja](../../../examples/pooling/score/template/mxbai_rerank_v2.jinja) | ✅︎ | ✅︎ |

42 +| `Qwen3ForSequenceClassification`C | Qwen3-based | `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `Qwen/Qwen3-Reranker-0.6B`(see note), etc. | [qwen3_reranker.jinja](../../../examples/pooling/score/template/qwen3_reranker.jinja) | ✅︎ | ✅︎ |

45 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | N/A | \* | \* |

46 +

47 +C Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))

48 +\* Feature support is the same as that of the original model.

49 +

50 +!!! note

51 + Some models require a specific prompt format to work correctly.

52 +

53 + You can find Example HF Models's corresponding score template in [examples/pooling/score/template/](../../../examples/pooling/score/template)

54 +

55 + Examples : [examples/pooling/score/using_template_offline.py](../../../examples/pooling/score/using_template_offline.py) [examples/pooling/score/using_template_online.py](../../../examples/pooling/score/using_template_online.py)

56 +

57 +!!! note

58 + Load the official original `BAAI/bge-reranker-v2-gemma` by using the following command.

59 +

60 + ```bash

61 + vllm serve BAAI/bge-reranker-v2-gemma --hf_overrides '{"architectures": ["GemmaForSequenceClassification"],"classifier_from_token": ["Yes"],"method": "no_post_processing"}'

62 + ```

63 +

64 +!!! note

65 + The second-generation GTE model (mGTE-TRM) is named `NewForSequenceClassification`. The name `NewForSequenceClassification` is too generic, you should set `--hf-overrides '{"architectures": ["GteNewForSequenceClassification"]}'` to specify the use of the `GteNewForSequenceClassification` architecture.

66 +

67 +!!! note

68 + Load the official original `mxbai-rerank-v2` by using the following command.

69 +

70 + ```bash

71 + vllm serve mixedbread-ai/mxbai-rerank-base-v2 --hf_overrides '{"architectures": ["Qwen2ForSequenceClassification"],"classifier_from_token": ["0", "1"], "method": "from_2_way_softmax"}'

72 + ```

73 +

74 +!!! note

75 + Load the official original `Qwen3 Reranker` by using the following command. More information can be found at: [examples/pooling/score/qwen3_reranker_offline.py](../../../examples/pooling/score/qwen3_reranker_offline.py) [examples/pooling/score/qwen3_reranker_online.py](../../../examples/pooling/score/qwen3_reranker_online.py).

76 +

77 + ```bash

78 + vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

79 + ```

80 +

81 +#### Multimodal Models

82 +

83 +!!! note

84 + For more information about multimodal models inputs, see [this page](../supported_models.md#list-of-multimodal-language-models).

85 +

87 +| ------------ | ------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |

88 +| `JinaVLForSequenceClassification` | JinaVL-based | T + IE+ | `jinaai/jina-reranker-m0`, etc. | ✅︎ | ✅︎ |

90 +| `Qwen3VLForSequenceClassification` | Qwen3-VL-Reranker | T + IE+ + VE+ | `Qwen/Qwen3-VL-Reranker-2B`(see note), etc. | ✅︎ | ✅︎ |

91 +

92 +C Automatically converted into a classification model via `--convert classify`. ([details](README.md#model-conversion))

93 +\* Feature support is the same as that of the original model.

94 +

95 +!!! note

96 + Similar to Qwen3-Reranker, you need to use the following `--hf_overrides` to load the official original `Qwen3-VL-Reranker`.

97 +

98 + ```bash

99 + vllm serve Qwen/Qwen3-VL-Reranker-2B --hf_overrides '{"architectures": ["Qwen3VLForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

100 + ```

101 +

102 +--8<-- [end:supported-score-models]

103 +

104 +### Late-interaction models

105 +

106 +All models that support token embedding task also support using the score API to compute similarity scores by calculating the late interaction of two input prompts. See [this page](token_embed.md) for more information about token embedding models.

107 +

108 +--8<-- "docs/models/pooling_models/token_embed.md:supported-token-embed-models"

109 +

110 +### Bi-encoder

111 +

112 +All models that support embedding task also support using the score API to compute similarity scores by calculating the cosine similarity of two input prompt's embeddings. See [this page](embed.md) for more information about embedding models.

113 +

114 +--8<-- "docs/models/pooling_models/embed.md:supported-embed-models"

115 +

116 +## Offline Inference

117 +

118 +### Pooling Parameters

119 +

120 +The following [pooling parameters][vllm.PoolingParams] are only supported by cross-encoder models and do not work for late-interaction and bi-encoder models.

121 +

122 +```python

123 +--8<-- "vllm/pooling_params.py:common-pooling-params"

124 +--8<-- "vllm/pooling_params.py:classify-pooling-params"

125 +```

126 +

127 +### `LLM.score`

128 +

129 +The [score][vllm.LLM.score] method outputs similarity scores between sentence pairs.

130 +

131 +```python

132 +from vllm import LLM

133 +

134 +llm = LLM(model="BAAI/bge-reranker-v2-m3", runner="pooling")

135 +(output,) = llm.score(

136 + "What is the capital of France?",

137 + "The capital of Brazil is Brasilia.",

138 +)

139 +

140 +score = output.outputs.score

141 +print(f"Score: {score}")

142 +```

143 +

144 +A code example can be found here: [examples/basic/offline_inference/score.py](../../../examples/basic/offline_inference/score.py)

145 +

146 +## Online Serving

147 +

148 +### Score API

149 +

150 +Our Score API (`/score`) is similar to `LLM.score`, compute similarity scores between two input prompts.

151 +

152 +#### Parameters

153 +

154 +The following Score API parameters are supported:

155 +

156 +```python

157 +--8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

158 +--8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

159 +--8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

160 +```

161 +

162 +#### Examples

163 +

164 +##### Single inference

165 +

166 +You can pass a string to both `queries` and `documents`, forming a single sentence pair.

167 +

168 +```bash

169 +curl -X 'POST' \

170 + 'http://127.0.0.1:8000/score' \

171 + -H 'accept: application/json' \

172 + -H 'Content-Type: application/json' \

173 + -d '{

174 + "model": "BAAI/bge-reranker-v2-m3",

175 + "encoding_format": "float",

176 + "queries": "What is the capital of France?",

177 + "documents": "The capital of France is Paris."

178 +}'

179 +```

180 +

181 +??? console "Response"

182 +

183 + ```json

184 + {

185 + "id": "score-request-id",

186 + "object": "list",

187 + "created": 693447,

188 + "model": "BAAI/bge-reranker-v2-m3",

189 + "data": [

190 + {

191 + "index": 0,

192 + "object": "score",

193 + "score": 1

194 + }

195 + ],

196 + "usage": {}

197 + }

198 + ```

199 +

200 +##### Batch inference

201 +

202 +You can pass a string to `queries` and a list to `documents`, forming multiple sentence pairs

203 +where each pair is built from `queries` and a string in `documents`.

204 +The total number of pairs is `len(documents)`.

205 +

206 +??? console "Request"

207 +

208 + ```bash

209 + curl -X 'POST' \

210 + 'http://127.0.0.1:8000/score' \

211 + -H 'accept: application/json' \

212 + -H 'Content-Type: application/json' \

213 + -d '{

214 + "model": "BAAI/bge-reranker-v2-m3",

215 + "queries": "What is the capital of France?",

216 + "documents": [

217 + "The capital of Brazil is Brasilia.",

218 + "The capital of France is Paris."

219 + ]

220 + }'

221 + ```

222 +

223 +??? console "Response"

224 +

225 + ```json

226 + {

227 + "id": "score-request-id",

228 + "object": "list",

229 + "created": 693570,

230 + "model": "BAAI/bge-reranker-v2-m3",

231 + "data": [

232 + {

233 + "index": 0,

234 + "object": "score",

235 + "score": 0.001094818115234375

236 + },

237 + {

238 + "index": 1,

239 + "object": "score",

240 + "score": 1

241 + }

242 + ],

243 + "usage": {}

244 + }

245 + ```

246 +

247 +You can pass a list to both `queries` and `documents`, forming multiple sentence pairs

248 +where each pair is built from a string in `queries` and the corresponding string in `documents` (similar to `zip()`).

249 +The total number of pairs is `len(documents)`.

250 +

251 +??? console "Request"

252 +

253 + ```bash

254 + curl -X 'POST' \

255 + 'http://127.0.0.1:8000/score' \

256 + -H 'accept: application/json' \

257 + -H 'Content-Type: application/json' \

258 + -d '{

259 + "model": "BAAI/bge-reranker-v2-m3",

260 + "encoding_format": "float",

261 + "queries": [

262 + "What is the capital of Brazil?",

263 + "What is the capital of France?"

264 + ],

265 + "documents": [

266 + "The capital of Brazil is Brasilia.",

267 + "The capital of France is Paris."

268 + ]

269 + }'

270 + ```

271 +

272 +??? console "Response"

273 +

274 + ```json

275 + {

276 + "id": "score-request-id",

277 + "object": "list",

278 + "created": 693447,

279 + "model": "BAAI/bge-reranker-v2-m3",

280 + "data": [

281 + {

282 + "index": 0,

283 + "object": "score",

284 + "score": 1

285 + },

286 + {

287 + "index": 1,

288 + "object": "score",

289 + "score": 1

290 + }

291 + ],

292 + "usage": {}

293 + }

294 + ```

295 +

296 +##### Multi-modal inputs

297 +

298 +You can pass multi-modal inputs to scoring models by passing `content` including a list of multi-modal input (image, etc.) in the request. Refer to the examples below for illustration.

299 +

300 +=== "JinaVL-Reranker"

301 +

302 + To serve the model:

303 +

304 + ```bash

305 + vllm serve jinaai/jina-reranker-m0

306 + ```

307 +

308 + Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

309 +

310 + ??? Code

311 +

312 + ```python

313 + import requests

314 +

315 + response = requests.post(

316 + "http://localhost:8000/v1/score",

317 + json={

318 + "model": "jinaai/jina-reranker-m0",

319 + "queries": "slm markdown",

320 + "documents": [

321 + {

322 + "content": [

323 + {

324 + "type": "image_url",

325 + "image_url": {

326 + "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"

327 + },

328 + }

329 + ],

330 + },

331 + {

332 + "content": [

333 + {

334 + "type": "image_url",

335 + "image_url": {

336 + "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"

337 + },

338 + }

339 + ]

340 + },

341 + ],

342 + },

343 + )

344 + response.raise_for_status()

345 + response_json = response.json()

346 + print("Scoring output:", response_json["data"][0]["score"])

347 + print("Scoring output:", response_json["data"][1]["score"])

348 + ```

349 +Full example:

350 +

351 +- [examples/pooling/score/vision_score_api_online.py](../../../examples/pooling/score/vision_score_api_online.py)

352 +- [examples/pooling/score/vision_rerank_api_online.py](../../../examples/pooling/score/vision_rerank_api_online.py)

353 +

354 +### Rerank API

355 +

356 +`/rerank`, `/v1/rerank`, and `/v2/rerank` APIs are compatible with both [Jina AI's rerank API interface](https://jina.ai/reranker/) and

357 +[Cohere's rerank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with

358 +popular open-source tools.

359 +

360 +Code example: [examples/pooling/score/rerank_api_online.py](../../../examples/pooling/score/rerank_api_online.py)

361 +

362 +#### Parameters

363 +

364 +The following rerank api parameters are supported:

365 +

366 +```python

367 +--8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

368 +--8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

369 +--8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

370 +```

371 +

372 +#### Examples

373 +

374 +Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.

375 +Result documents will be sorted by relevance, and the `index` property can be used to determine original order.

376 +

377 +??? console "Request"

378 +

379 + ```bash

380 + curl -X 'POST' \

381 + 'http://127.0.0.1:8000/v1/rerank' \

382 + -H 'accept: application/json' \

383 + -H 'Content-Type: application/json' \

384 + -d '{

385 + "model": "BAAI/bge-reranker-base",

386 + "query": "What is the capital of France?",

387 + "documents": [

388 + "The capital of Brazil is Brasilia.",

389 + "The capital of France is Paris.",

390 + "Horses and cows are both animals"

391 + ]

392 + }'

393 + ```

394 +

395 +??? console "Response"

396 +

397 + ```json

398 + {

399 + "id": "rerank-fae51b2b664d4ed38f5969b612edff77",

400 + "model": "BAAI/bge-reranker-base",

401 + "usage": {

402 + "total_tokens": 56

403 + },

404 + "results": [

405 + {

406 + "index": 1,

407 + "document": {

408 + "text": "The capital of France is Paris."

409 + },

410 + "relevance_score": 0.99853515625

411 + },

412 + {

413 + "index": 0,

414 + "document": {

415 + "text": "The capital of Brazil is Brasilia."

416 + },

417 + "relevance_score": 0.0005860328674316406

418 + }

419 + ]

420 + }

421 + ```

422 +

423 +## More examples

424 +

425 +More examples can be found here: [examples/pooling/score](../../../examples/pooling/score)

426 +

427 +## Supported Features

428 +

429 +AS cross-encoder models are a subset of classification models that accept two prompts as input and output num_labels equal to 1, cross-encoder features should be consistent with (sequence) classification. For more information, see [this page](classify.md#supported-features).

430 +

431 +### Score Template

432 +

433 +Score templates are supported for **cross-encoder** models only. If you are using an **embedding** model for scoring, vLLM does not apply a score template.

434 +

435 +Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/openai_compatible_server.md#chat-template)).

436 +

437 +Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter:

438 +

439 +- **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}`

440 +- **Document**: `{{ (messages | selectattr("role", "eq", "document") | first).content }}`

441 +

442 +This approach is more robust than index-based access (`messages[0]`, `messages[1]`) because it selects messages by their semantic role. It also avoids assumptions about message ordering if additional message types are added to `messages` in the future.

443 +

444 +Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)

445 +

446 +### Enable/disable activation

447 +

448 +You can enable or disable activation via `use_activation` only works for cross-encoder models.

docs/models/pooling_models/specific_models.md new

+395

@@ -0,0 +1,395 @@

1 +# Specific Model Examples

2 +

3 +## ColBERT Late Interaction Models

4 +

5 +[ColBERT](https://arxiv.org/abs/2004.12832) (Contextualized Late Interaction over BERT) is a retrieval model that uses per-token embeddings and MaxSim scoring for document ranking. Unlike single-vector embedding models, ColBERT retains token-level representations and computes relevance scores through late interaction, providing better accuracy while being more efficient than cross-encoders.

6 +

7 +vLLM supports ColBERT models with multiple encoder backbones:

8 +

9 +| Architecture | Backbone | Example HF Models |

10 +| - | - | - |

11 +| `HF_ColBERT` | BERT | `answerdotai/answerai-colbert-small-v1`, `colbert-ir/colbertv2.0` |

12 +| `ColBERTModernBertModel` | ModernBERT | `lightonai/GTE-ModernColBERT-v1` |

13 +| `ColBERTJinaRobertaModel` | Jina XLM-RoBERTa | `jinaai/jina-colbert-v2` |

14 +

15 +**BERT-based ColBERT** models work out of the box:

16 +

17 +```shell

18 +vllm serve answerdotai/answerai-colbert-small-v1

19 +```

20 +

21 +For **non-BERT backbones**, use `--hf-overrides` to set the correct architecture:

22 +

23 +```shell

24 +# ModernBERT backbone

25 +vllm serve lightonai/GTE-ModernColBERT-v1 \

26 + --hf-overrides '{"architectures": ["ColBERTModernBertModel"]}'

27 +

28 +# Jina XLM-RoBERTa backbone

29 +vllm serve jinaai/jina-colbert-v2 \

30 + --hf-overrides '{"architectures": ["ColBERTJinaRobertaModel"]}' \

31 + --trust-remote-code

32 +```

33 +

34 +Then you can use the rerank API:

35 +

36 +```shell

37 +curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

38 + "model": "answerdotai/answerai-colbert-small-v1",

39 + "query": "What is machine learning?",

40 + "documents": [

41 + "Machine learning is a subset of artificial intelligence.",

42 + "Python is a programming language.",

43 + "Deep learning uses neural networks."

44 + ]

45 +}'

46 +```

47 +

48 +Or the score API:

49 +

50 +```shell

51 +curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

52 + "model": "answerdotai/answerai-colbert-small-v1",

53 + "text_1": "What is machine learning?",

54 + "text_2": ["Machine learning is a subset of AI.", "The weather is sunny."]

55 +}'

56 +```

57 +

58 +You can also get the raw token embeddings using the pooling API with `token_embed` task:

59 +

60 +```shell

61 +curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

62 + "model": "answerdotai/answerai-colbert-small-v1",

63 + "input": "What is machine learning?",

64 + "task": "token_embed"

65 +}'

66 +```

67 +

68 +An example can be found here: [examples/pooling/score/colbert_rerank_online.py](../../../examples/pooling/score/colbert_rerank_online.py)

69 +

70 +## ColQwen3 Multi-Modal Late Interaction Models

71 +

72 +ColQwen3 is based on [ColPali](https://arxiv.org/abs/2407.01449), which extends ColBERT's late interaction approach to **multi-modal** inputs. While ColBERT operates on text-only token embeddings, ColPali/ColQwen3 can embed both **text and images** (e.g. PDF pages, screenshots, diagrams) into per-token L2-normalized vectors and compute relevance via MaxSim scoring. ColQwen3 specifically uses Qwen3-VL as its vision-language backbone.

73 +

74 +| Architecture | Backbone | Example HF Models |

75 +| - | - | - |

76 +| `ColQwen3` | Qwen3-VL | `TomoroAI/tomoro-colqwen3-embed-4b`, `TomoroAI/tomoro-colqwen3-embed-8b` |

77 +| `OpsColQwen3Model` | Qwen3-VL | `OpenSearch-AI/Ops-Colqwen3-4B`, `OpenSearch-AI/Ops-Colqwen3-8B` |

78 +| `Qwen3VLNemotronEmbedModel` | Qwen3-VL | `nvidia/nemotron-colembed-vl-4b-v2`, `nvidia/nemotron-colembed-vl-8b-v2` |

79 +

80 +Start the server:

81 +

82 +```shell

83 +vllm serve TomoroAI/tomoro-colqwen3-embed-4b --max-model-len 4096

84 +```

85 +

86 +### Text-only scoring and reranking

87 +

88 +Use the `/rerank` API:

89 +

90 +```shell

91 +curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

92 + "model": "TomoroAI/tomoro-colqwen3-embed-4b",

93 + "query": "What is machine learning?",

94 + "documents": [

95 + "Machine learning is a subset of artificial intelligence.",

96 + "Python is a programming language.",

97 + "Deep learning uses neural networks."

98 + ]

99 +}'

100 +```

101 +

102 +Or the `/score` API:

103 +

104 +```shell

105 +curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

106 + "model": "TomoroAI/tomoro-colqwen3-embed-4b",

107 + "text_1": "What is the capital of France?",

108 + "text_2": ["The capital of France is Paris.", "Python is a programming language."]

109 +}'

110 +```

111 +

112 +### Multi-modal scoring and reranking (text query × image documents)

113 +

114 +The `/score` and `/rerank` APIs also accept multi-modal inputs directly.

115 +Pass image documents using the `data_1`/`data_2` (for `/score`) or `documents` (for `/rerank`) fields

116 +with a `content` list containing `image_url` and `text` parts — the same format used by the

117 +OpenAI chat completion API:

118 +

119 +Score a text query against image documents:

120 +

121 +```shell

122 +curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

123 + "model": "TomoroAI/tomoro-colqwen3-embed-4b",

124 + "data_1": "Retrieve the city of Beijing",

125 + "data_2": [

126 + {

127 + "content": [

128 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

129 + {"type": "text", "text": "Describe the image."}

130 + ]

131 + }

132 + ]

133 +}'

134 +```

135 +

136 +Rerank image documents by a text query:

137 +

138 +```shell

139 +curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

140 + "model": "TomoroAI/tomoro-colqwen3-embed-4b",

141 + "query": "Retrieve the city of Beijing",

142 + "documents": [

143 + {

144 + "content": [

145 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_1>"}},

146 + {"type": "text", "text": "Describe the image."}

147 + ]

148 + },

149 + {

150 + "content": [

151 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_2>"}},

152 + {"type": "text", "text": "Describe the image."}

153 + ]

154 + }

155 + ],

156 + "top_n": 2

157 +}'

158 +```

159 +

160 +### Raw token embeddings

161 +

162 +You can also get the raw token embeddings using the `/pooling` API with `token_embed` task:

163 +

164 +```shell

165 +curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

166 + "model": "TomoroAI/tomoro-colqwen3-embed-4b",

167 + "input": "What is machine learning?",

168 + "task": "token_embed"

169 +}'

170 +```

171 +

172 +For **image inputs** via the pooling API, use the chat-style `messages` field:

173 +

174 +```shell

175 +curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

176 + "model": "TomoroAI/tomoro-colqwen3-embed-4b",

177 + "messages": [

178 + {

179 + "role": "user",

180 + "content": [

181 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

182 + {"type": "text", "text": "Describe the image."}

183 + ]

184 + }

185 + ]

186 +}'

187 +```

188 +

189 +### Examples

190 +

191 +- Multi-vector retrieval: [examples/pooling/token_embed/colqwen3_token_embed_online.py](../../../examples/pooling/token_embed/colqwen3_token_embed_online.py)

192 +- Reranking (text + multi-modal): [examples/pooling/score/colqwen3_rerank_online.py](../../../examples/pooling/score/colqwen3_rerank_online.py)

193 +

194 +## ColQwen3.5 Multi-Modal Late Interaction Models

195 +

196 +ColQwen3.5 is based on [ColPali](https://arxiv.org/abs/2407.01449), extending ColBERT's late interaction approach to **multi-modal** inputs. It uses the Qwen3.5 hybrid backbone (linear + full attention) and produces per-token L2-normalized vectors for MaxSim scoring.

197 +

198 +| Architecture | Backbone | Example HF Models |

199 +| - | - | - |

200 +| `ColQwen3_5` | Qwen3.5 | `athrael-soju/colqwen3.5-4.5B` |

201 +

202 +Start the server:

203 +

204 +```shell

205 +vllm serve athrael-soju/colqwen3.5-4.5B --max-model-len 4096

206 +```

207 +

208 +Then you can use the rerank endpoint:

209 +

210 +```shell

211 +curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

212 + "model": "athrael-soju/colqwen3.5-4.5B",

213 + "query": "What is machine learning?",

214 + "documents": [

215 + "Machine learning is a subset of artificial intelligence.",

216 + "Python is a programming language.",

217 + "Deep learning uses neural networks."

218 + ]

219 +}'

220 +```

221 +

222 +Or the score endpoint:

223 +

224 +```shell

225 +curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

226 + "model": "athrael-soju/colqwen3.5-4.5B",

227 + "text_1": "What is the capital of France?",

228 + "text_2": ["The capital of France is Paris.", "Python is a programming language."]

229 +}'

230 +```

231 +

232 +An example can be found here: [examples/pooling/score/colqwen3_5_rerank_online.py](../../../examples/pooling/score/colqwen3_5_rerank_online.py)

233 +

234 +## Llama Nemotron Multimodal

235 +

236 +### Embedding Model

237 +

238 +Llama Nemotron VL Embedding models combine the bidirectional Llama embedding backbone

239 +(from `nvidia/llama-nemotron-embed-1b-v2`) with SigLIP as the vision encoder to produce

240 +single-vector embeddings from text and/or images.

241 +

242 +| Architecture | Backbone | Example HF Models |

243 +| - | - | - |

244 +| `LlamaNemotronVLModel` | Bidirectional Llama + SigLIP | `nvidia/llama-nemotron-embed-vl-1b-v2` |

245 +

246 +Start the server:

247 +

248 +```shell

249 +vllm serve nvidia/llama-nemotron-embed-vl-1b-v2 \

250 + --trust-remote-code \

251 + --chat-template examples/pooling/embed/template/nemotron_embed_vl.jinja

252 +```

253 +

254 +!!! note

255 + The chat template bundled with this model's tokenizer is not suitable for

256 + the embeddings API. Use the provided override template above when serving

257 + with the `messages`-based (chat-style) embeddings API.

258 +

259 + The override template uses the message `role` to automatically prepend the

260 + appropriate prefix: set `role` to `"query"` for queries (prepends `query: `)

261 + or `"document"` for passages (prepends `passage: `). Any other role omits

262 + the prefix.

263 +

264 +Embed text queries:

265 +

266 +```shell

267 +curl -s http://localhost:8000/v1/embeddings -H "Content-Type: application/json" -d '{

268 + "model": "nvidia/llama-nemotron-embed-vl-1b-v2",

269 + "messages": [

270 + {

271 + "role": "query",

272 + "content": [

273 + {"type": "text", "text": "What is machine learning?"}

274 + ]

275 + }

276 + ]

277 +}'

278 +```

279 +

280 +Embed images via the chat-style `messages` field:

281 +

282 +```shell

283 +curl -s http://localhost:8000/v1/embeddings -H "Content-Type: application/json" -d '{

284 + "model": "nvidia/llama-nemotron-embed-vl-1b-v2",

285 + "messages": [

286 + {

287 + "role": "document",

288 + "content": [

289 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

290 + {"type": "text", "text": "Describe the image."}

291 + ]

292 + }

293 + ]

294 +}'

295 +```

296 +

297 +### Reranker Model

298 +

299 +Llama Nemotron VL reranker models combine the same bidirectional Llama + SigLIP

300 +backbone with a sequence-classification head for cross-encoder scoring and reranking.

301 +

302 +| Architecture | Backbone | Example HF Models |

303 +| - | - | - |

304 +| `LlamaNemotronVLForSequenceClassification` | Bidirectional Llama + SigLIP | `nvidia/llama-nemotron-rerank-vl-1b-v2` |

305 +

306 +Start the server:

307 +

308 +```shell

309 +vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \

310 + --runner pooling \

311 + --trust-remote-code \

312 + --chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja

313 +```

314 +

315 +!!! note

316 + The chat template bundled with this checkpoint's tokenizer is not suitable

317 + for the Score/Rerank APIs. Use the provided override template when serving:

318 + `examples/pooling/score/template/nemotron-vl-rerank.jinja`.

319 +

320 +Score a text query against an image document:

321 +

322 +```shell

323 +curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

324 + "model": "nvidia/llama-nemotron-rerank-vl-1b-v2",

325 + "data_1": "Find diagrams about autonomous robots",

326 + "data_2": [

327 + {

328 + "content": [

329 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

330 + {"type": "text", "text": "Robotics workflow diagram."}

331 + ]

332 + }

333 + ]

334 +}'

335 +```

336 +

337 +Rerank image documents by a text query:

338 +

339 +```shell

340 +curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

341 + "model": "nvidia/llama-nemotron-rerank-vl-1b-v2",

342 + "query": "Find diagrams about autonomous robots",

343 + "documents": [

344 + {

345 + "content": [

346 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_1>"}},

347 + {"type": "text", "text": "Robotics workflow diagram."}

348 + ]

349 + },

350 + {

351 + "content": [

352 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_2>"}},

353 + {"type": "text", "text": "General skyline photo."}

354 + ]

355 + }

356 + ],

357 + "top_n": 2

358 +}'

359 +```

360 +

361 +## BAAI/bge-m3

362 +

363 +The `BAAI/bge-m3` model comes with extra weights for sparse and colbert embeddings but unfortunately in its `config.json`

364 +the architecture is declared as `XLMRobertaModel`, which makes `vLLM` load it as a vanilla ROBERTA model without the

365 +extra weights. To load the full model weights, override its architecture like this:

366 +

367 +```shell

368 +vllm serve BAAI/bge-m3 --hf-overrides '{"architectures": ["BgeM3EmbeddingModel"]}'

369 +```

370 +

371 +Then you obtain the sparse embeddings like this:

372 +

373 +```shell

374 +curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

375 + "model": "BAAI/bge-m3",

376 + "task": "token_classify",

377 + "input": ["What is BGE M3?", "Definition of BM25"]

378 +}'

379 +```

380 +

381 +Due to limitations in the output schema, the output consists of a list of

382 +token scores for each token for each input. This means that you'll have to call

383 +`/tokenize` as well to be able to pair tokens with scores.

384 +Refer to the tests in `tests/models/language/pooling/test_bge_m3.py` to see how

385 +to do that.

386 +

387 +You can obtain the colbert embeddings like this:

388 +

389 +```shell

390 +curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

391 + "model": "BAAI/bge-m3",

392 + "task": "token_embed",

393 + "input": ["What is BGE M3?", "Definition of BM25"]

394 +}'

395 +```

docs/models/pooling_models/token_classify.md new

+89

@@ -0,0 +1,89 @@

1 +# Token Classification Usages

2 +

3 +## Summary

4 +

5 +- Model Usage: token classification

6 +- Pooling Tasks: `token_classify`

7 +- Offline APIs:

8 + - `LLM.encode(..., pooling_task="token_classify")`

9 +- Online APIs:

10 + - Pooling API (`/pooling`)

11 +

12 +The key distinction between (sequence) classification and token classification lies in their output granularity: (sequence) classification produces a single result for an entire input sequence, whereas token classification yields a result for each individual token within the sequence.

13 +

14 +Many classification models support both (sequence) classification and token classification. For further details on (sequence) classification, please refer to [this page](classify.md).

15 +

16 +## Typical Use Cases

17 +

18 +### Named Entity Recognition (NER)

19 +

20 +For implementation examples, see:

21 +

22 +Offline: [examples/pooling/token_classify/ner_offline.py](../../../examples/pooling/token_classify/ner_offline.py)

23 +

24 +Online: [examples/pooling/token_classify/ner_online.py](../../../examples/pooling/token_classify/ner_online.py)

25 +

26 +### Sparse retrieval (lexical matching)

27 +

28 +The BAAI/bge-m3 model leverages token classification for sparse retrieval. For more information, see [this page](specific_models.md#baaibge-m3).

29 +

30 +## Supported Models

31 +

33 +| ------------ | ------ | ----------------- | --------------------------- | --------------------------------------- |

38 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

39 +

40 +C Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))

41 +\* Feature support is the same as that of the original model.

42 +

43 +If your model is not in the above list, we will try to automatically convert the model using

44 +[as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.

45 +

46 +### As Reward Models

47 +

48 +Using token classification models as reward models. For details on reward models, see [Reward Models](reward.md).

49 +

50 +--8<-- "docs/models/pooling_models/reward.md:supported-token-reward-models"

51 +

52 +## Offline Inference

53 +

54 +### Pooling Parameters

55 +

56 +The following [pooling parameters][vllm.PoolingParams] are supported.

57 +

58 +```python

59 +--8<-- "vllm/pooling_params.py:common-pooling-params"

60 +--8<-- "vllm/pooling_params.py:classify-pooling-params"

61 +```

62 +

63 +### `LLM.encode`

64 +

65 +The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.

66 +

67 +Set `pooling_task="token_classify"` when using `LLM.encode` for token classification Models:

68 +

69 +```python

70 +from vllm import LLM

71 +

72 +llm = LLM(model="boltuix/NeuroBERT-NER", runner="pooling")

73 +(output,) = llm.encode("Hello, my name is", pooling_task="token_classify")

74 +

75 +data = output.outputs.data

76 +print(f"Data: {data!r}")

77 +```

78 +

79 +## Online Serving

80 +

81 +Please refer to the [pooling API](README.md#pooling-api) and use `"task":"token_classify"`.

82 +

83 +## More examples

84 +

85 +More examples can be found here: [examples/pooling/token_classify](../../../examples/pooling/token_classify)

86 +

87 +## Supported Features

88 +

89 +Token classification features should be consistent with (sequence) classification. For more information, see [this page](classify.md#supported-features).

docs/models/pooling_models/token_embed.md new

+125

@@ -0,0 +1,125 @@

1 +# Token Embedding Usages

2 +

3 +## Summary

4 +

5 +- Model Usage: Token classification models

6 +- Pooling Tasks: `token_embed`

7 +- Offline APIs:

8 + - `LLM.encode(..., pooling_task="token_embed")`

9 +- Online APIs:

10 + - Pooling API (`/pooling`)

11 +

12 +The difference between the (sequence) embedding task and the token embedding task is that (sequence) embedding outputs one embedding for each sequence, while token embedding outputs a embedding for each token.

13 +

14 +Many embedding models support both (sequence) embedding and token embedding. For further details on (sequence) embedding, please refer to [this page](embed.md).

15 +

16 +## Typical Use Cases

17 +

18 +### Multi-Vector Retrieval

19 +

20 +For implementation examples, see:

21 +

22 +Offline: [examples/pooling/token_embed/multi_vector_retrieval_offline.py](../../../examples/pooling/token_embed/multi_vector_retrieval_offline.py)

23 +

24 +Online: [examples/pooling/token_embed/multi_vector_retrieval_online.py](../../../examples/pooling/token_embed/multi_vector_retrieval_online.py)

25 +

26 +### Late interaction

27 +

28 +Similarity scores can be computed using late interaction between two input prompts via the score API. For more information, see [Score API](scoring.md).

29 +

30 +### Extract last hidden states

31 +

32 +Models of any architecture can be converted into embedding models using `--convert embed`. Token embedding can then be used to extract the last hidden states from these models.

33 +

34 +## Supported Models

35 +

36 +--8<-- [start:supported-token-embed-models]

37 +

38 +### Text-only Models

39 +

41 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

45 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

46 +

47 +### Multimodal Models

48 +

49 +!!! note

50 + For more information about multimodal models inputs, see [this page](../supported_models.md#list-of-multimodal-language-models).

51 +

53 +| ------------ | ------ | ----- | ----------------- | ------------------------------ | ------------------------------------------ |

59 +| `Qwen3VLNemotronEmbedModel` | Qwen3-VL | T / I | `nvidia/nemotron-colembed-vl-4b-v2`, `nvidia/nemotron-colembed-vl-8b-v2` | ✅︎ | ✅︎ |

60 +| `*ForConditionalGeneration`C, `*ForCausalLM`C, etc. | Generative models | \* | N/A | \* | \* |

61 +

62 +C Automatically converted into an embedding model via `--convert embed`. ([details](./README.md#model-conversion))

63 +\* Feature support is the same as that of the original model.

64 +

65 +If your model is not in the above list, we will try to automatically convert the model using [as_embedding_model][vllm.model_executor.models.adapters.as_embedding_model].

66 +

67 +--8<-- [end:supported-token-embed-models]

68 +

69 +## Offline Inference

70 +

71 +### Pooling Parameters

72 +

73 +The following [pooling parameters][vllm.PoolingParams] are supported.

74 +

75 +```python

76 +--8<-- "vllm/pooling_params.py:common-pooling-params"

77 +--8<-- "vllm/pooling_params.py:embed-pooling-params"

78 +```

79 +

80 +### `LLM.encode`

81 +

82 +The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.

83 +

84 +Set `pooling_task="token_embed"` when using `LLM.encode` for token embedding Models:

85 +

86 +```python

87 +from vllm import LLM

88 +

89 +llm = LLM(model="answerdotai/answerai-colbert-small-v1", runner="pooling")

90 +(output,) = llm.encode("Hello, my name is", pooling_task="token_embed")

91 +

92 +data = output.outputs.data

93 +print(f"Data: {data!r}")

94 +```

95 +

96 +### `LLM.score`

97 +

98 +The [score][vllm.LLM.score] method outputs similarity scores between sentence pairs.

99 +

100 +All models that support token embedding task also support using the score API to compute similarity scores by calculating the late interaction of two input prompts.

101 +

102 +```python

103 +from vllm import LLM

104 +

105 +llm = LLM(model="answerdotai/answerai-colbert-small-v1", runner="pooling")

106 +(output,) = llm.score(

107 + "What is the capital of France?",

108 + "The capital of Brazil is Brasilia.",

109 +)

110 +

111 +score = output.outputs.score

112 +print(f"Score: {score}")

113 +```

114 +

115 +## Online Serving

116 +

117 +Please refer to the [pooling API](README.md#pooling-api) and use `"task":"token_embed"`.

118 +

119 +## More examples

120 +

121 +More examples can be found here: [examples/pooling/token_embed](../../../examples/pooling/token_embed)

122 +

123 +## Supported Features

124 +

125 +Token embedding features should be consistent with (sequence) embedding. For more information, see [this page](embed.md#supported-features).

docs/models/supported_models.md

+14 -198

@@ -1,6 +1,6 @@

1 # Supported Models

3 -vLLM supports [generative](./generative_models.md) and [pooling](./pooling_models.md) models across various tasks.

3 +vLLM supports [generative](./generative_models.md) and [pooling](./pooling_models/README.md) models across various tasks.

5 For each task, we list the model architectures that have been implemented in vLLM.

6 Alongside each architecture, we include some popular models that use it.

@@ -499,156 +499,6 @@ Some models are supported only via the [Transformers modeling backend](#transfor

499 !!! note

500 Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.

501

502 -### Pooling Models

503 -

504 -See [this page](./pooling_models.md) for more information on how to use pooling models.

505 -

506 -!!! important

507 - Since some model architectures support both generative and pooling tasks,

508 - you should explicitly specify `--runner pooling` to ensure that the model is used in pooling mode instead of generative mode.

509 -

510 -#### Embedding

511 -

512 -These models primarily support the [`LLM.embed`](./pooling_models.md#llmembed) API.

513 -

515 -| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

519 -| `Gemma2Model`C | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | ✅︎ |

520 -| `Gemma3TextModel`C | Gemma 3-based | `google/embeddinggemma-300m`, etc. | ✅︎ | ✅︎ |

521 -| `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ |

526 -| `LlamaBidirectionalModel`C | Llama-based with bidirectional attention | `nvidia/llama-nemotron-embed-1b-v2`, etc. | ✅︎ | ✅︎ |

527 -| `LlamaModel`C, `LlamaForCausalLM`C, `MistralModel`C, etc. | Llama-based | `intfloat/e5-mistral-7b-instruct`, etc. | ✅︎ | ✅︎ |

528 -| `Qwen2Model`C, `Qwen2ForCausalLM`C | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ |

529 -| `Qwen3Model`C, `Qwen3ForCausalLM`C | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, etc. | ✅︎ | ✅︎ |

530 -| `VoyageQwen3BidirectionalEmbedModel`C | Voyage Qwen3-based with bidirectional attention | `voyageai/voyage-4-nano`, etc. | ✅︎ | ✅︎ |

532 -| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

533 -

534 -C Automatically converted into an embedding model via `--convert embed`. ([details](./pooling_models.md#model-conversion))

535 -\* Feature support is the same as that of the original model.

536 -

537 -!!! note

538 - `ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.

539 - You need to manually set mean pooling by passing `--pooler-config '{"pooling_type": "MEAN"}'`.

540 -

541 -!!! note

542 - For `Alibaba-NLP/gte-Qwen2-*`, you need to enable `--trust-remote-code` for the correct tokenizer to be loaded.

543 - See [relevant issue on HF Transformers](https://github.com/huggingface/transformers/issues/34882).

544 -

545 -!!! note

546 - `jinaai/jina-embeddings-v3` supports multiple tasks through LoRA, while vllm temporarily only supports text-matching tasks by merging LoRA weights.

547 -

548 -!!! note

549 - The second-generation GTE model (mGTE-TRM) is named `NewModel`. The name `NewModel` is too generic, you should set `--hf-overrides '{"architectures": ["GteNewModel"]}'` to specify the use of the `GteNewModel` architecture.

550 -

551 -If your model is not in the above list, we will try to automatically convert the model using

552 -[as_embedding_model][vllm.model_executor.models.adapters.as_embedding_model]. By default, the embeddings

553 -of the whole prompt are extracted from the normalized hidden state corresponding to the last token.

554 -

555 -#### Classification

556 -

557 -These models primarily support the [`LLM.classify`](./pooling_models.md#llmclassify) API.

558 -

560 -| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

562 -| `GPT2ForSequenceClassification` | GPT2 | `nie3e/sentiment-polish-gpt2-small` | | |

563 -| `JambaForSequenceClassification` | Jamba | `ai21labs/Jamba-tiny-reward-dev`, etc. | ✅︎ | ✅︎ |

564 -| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

565 -

566 -C Automatically converted into a classification model via `--convert classify`. ([details](./pooling_models.md#model-conversion))

567 -\* Feature support is the same as that of the original model.

568 -

569 -If your model is not in the above list, we will try to automatically convert the model using

570 -[as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.

571 -

572 -#### Cross-encoder / Reranker

573 -

574 -Cross-encoder and reranker models are a subset of classification models that accept two prompts as input.

575 -These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) API.

576 -

578 -| ------------ | ------ | ----------------- | ------------------------- | --------------------------- | --------------------------------------- |

581 -| `GemmaForSequenceClassification` | Gemma-based | `BAAI/bge-reranker-v2-gemma`(see note), etc. | [bge-reranker-v2-gemma.jinja](../../examples/pooling/score/template/bge-reranker-v2-gemma.jinja) | ✅︎ | ✅︎ |

583 -| `LlamaBidirectionalForSequenceClassification`C | Llama-based with bidirectional attention | `nvidia/llama-nemotron-rerank-1b-v2`, etc. | [nemotron-rerank.jinja](../../examples/pooling/score/template/nemotron-rerank.jinja) | ✅︎ | ✅︎ |

584 -| `Qwen2ForSequenceClassification`C | Qwen2-based | `mixedbread-ai/mxbai-rerank-base-v2`(see note), etc. | [mxbai_rerank_v2.jinja](../../examples/pooling/score/template/mxbai_rerank_v2.jinja) | ✅︎ | ✅︎ |

585 -| `Qwen3ForSequenceClassification`C | Qwen3-based | `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `Qwen/Qwen3-Reranker-0.6B`(see note), etc. | [qwen3_reranker.jinja](../../examples/pooling/score/template/qwen3_reranker.jinja) | ✅︎ | ✅︎ |

588 -| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | N/A | \* | \* |

589 -

590 -C Automatically converted into a classification model via `--convert classify`. ([details](./pooling_models.md#model-conversion))

591 -\* Feature support is the same as that of the original model.

592 -

593 -!!! note

594 - Some models require a specific prompt format to work correctly.

595 -

596 - You can find Example HF Models's corresponding score template in [examples/pooling/score/template/](../../examples/pooling/score/template)

597 -

598 - Examples : [examples/pooling/score/using_template_offline.py](../../examples/pooling/score/using_template_offline.py) [examples/pooling/score/using_template_online.py](../../examples/pooling/score/using_template_online.py)

599 -

600 -!!! note

601 - Load the official original `BAAI/bge-reranker-v2-gemma` by using the following command.

602 -

603 - ```bash

604 - vllm serve BAAI/bge-reranker-v2-gemma --hf_overrides '{"architectures": ["GemmaForSequenceClassification"],"classifier_from_token": ["Yes"],"method": "no_post_processing"}'

605 - ```

606 -

607 -!!! note

608 - The second-generation GTE model (mGTE-TRM) is named `NewForSequenceClassification`. The name `NewForSequenceClassification` is too generic, you should set `--hf-overrides '{"architectures": ["GteNewForSequenceClassification"]}'` to specify the use of the `GteNewForSequenceClassification` architecture.

609 -

610 -!!! note

611 - Load the official original `mxbai-rerank-v2` by using the following command.

612 -

613 - ```bash

614 - vllm serve mixedbread-ai/mxbai-rerank-base-v2 --hf_overrides '{"architectures": ["Qwen2ForSequenceClassification"],"classifier_from_token": ["0", "1"], "method": "from_2_way_softmax"}'

615 - ```

616 -

617 -!!! note

618 - Load the official original `Qwen3 Reranker` by using the following command. More information can be found at: [examples/pooling/score/qwen3_reranker_offline.py](../../examples/pooling/score/qwen3_reranker_offline.py) [examples/pooling/score/qwen3_reranker_online.py](../../examples/pooling/score/qwen3_reranker_online.py).

619 -

620 - ```bash

621 - vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

622 - ```

623 -

624 -#### Reward Modeling

625 -

626 -These models primarily support the [`LLM.reward`](./pooling_models.md#llmreward) API.

627 -

629 -| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

630 -| `InternLM2ForRewardModel` | InternLM2-based | `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc. | ✅︎ | ✅︎ |

631 -| `LlamaForCausalLM` | Llama-based | `peiyi9979/math-shepherd-mistral-7b-prm`, etc. | ✅︎ | ✅︎ |

632 -| `Qwen2ForRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-RM-72B`, etc. | ✅︎ | ✅︎ |

633 -| `Qwen2ForProcessRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-PRM-7B`, etc. | ✅︎ | ✅︎ |

634 -

635 -!!! important

636 - For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,

637 - e.g.: `--pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.

638 -

639 -#### Token Classification

640 -

641 -These models primarily support the [`LLM.encode`](./pooling_models.md#llmencode) API.

642 -

644 -| ------------ | ------ | ----------------- | --------------------------- | --------------------------------------- |

648 -

649 -!!! note

650 - Named Entity Recognition (NER) usage, please refer to [examples/pooling/token_classify/ner_offline.py](../../examples/pooling/token_classify/ner_offline.py), [examples/pooling/token_classify/ner_online.py](../../examples/pooling/token_classify/ner_online.py).

651 -

502 ## List of Multimodal Language Models

503

504 The following modalities are supported depending on the model:

@@ -816,57 +666,23 @@ Speech2Text models trained specifically for Automatic Speech Recognition.

666 !!! note

667 `VoxtralForConditionalGeneration` requires `mistral-common[audio]` to be installed.

668

819 -### Pooling Models

820 -

821 -See [this page](./pooling_models.md) for more information on how to use pooling models.

669 +## Pooling Models

670

823 -#### Embedding

671 +See [this page](pooling_models/README.md) for more information on how to use pooling models.

672

825 -These models primarily support the [`LLM.embed`](./pooling_models.md#llmembed) API.

826 -

827 -!!! note

828 - To get the best results, you should use pooling models that are specifically trained as such.

829 -

830 -The following table lists those that are tested in vLLM.

831 -

833 -| ------------ | ------ | ------ | ----------------- | -------------------- | ------------------------- |

839 -| `LlavaNextForConditionalGeneration`C | LLaVA-NeXT-based | T / I | `royokong/e5-v` | | ✅︎ |

840 -| `Phi3VForCausalLM`C | Phi-3-Vision-based | T + I | `TIGER-Lab/VLM2Vec-Full` | | ✅︎ |

841 -| `Qwen3VLForConditionalGeneration`C | Qwen3-VL | T + I + V | `Qwen/Qwen3-VL-Embedding-2B`, etc. | ✅︎ | ✅︎ |

843 -| `*ForConditionalGeneration`C, `*ForCausalLM`C, etc. | Generative models | \* | N/A | \* | \* |

844 -

845 -C Automatically converted into an embedding model via `--convert embed`. ([details](./pooling_models.md#model-conversion))

846 -\* Feature support is the same as that of the original model.

847 -

848 ----

849 -

850 -#### Cross-encoder / Reranker

851 -

852 -Cross-encoder and reranker models are a subset of classification models that accept two prompts as input.

853 -These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) API.

854 -

856 -| ------------ | ------ | ------ | ----------------- | -------------------- | ------------------------- |

857 -| `JinaVLForSequenceClassification` | JinaVL-based | T + IE+ | `jinaai/jina-reranker-m0`, etc. | ✅︎ | ✅︎ |

859 -| `Qwen3VLForSequenceClassification` | Qwen3-VL-Reranker | T + IE+ + VE+ | `Qwen/Qwen3-VL-Reranker-2B`(see note), etc. | ✅︎ | ✅︎ |

860 -

861 -C Automatically converted into a classification model via `--convert classify`. ([details](./pooling_models.md#model-conversion))

862 -\* Feature support is the same as that of the original model.

673 +!!! important

674 + Since some model architectures support both generative and pooling tasks,

675 + you should explicitly specify `--runner pooling` to ensure that the model is used in pooling mode instead of generative mode.

676

864 -!!! note

865 - Similar to Qwen3-Reranker, you need to use the following `--hf_overrides` to load the official original `Qwen3-VL-Reranker`.

677 +See the link below for more information on the models supported for specific pooling tasks.

678

867 - ```bash

868 - vllm serve Qwen/Qwen3-VL-Reranker-2B --hf_overrides '{"architectures": ["Qwen3VLForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

869 - ```

679 +- [Classification Usages](pooling_models/classify.md)

680 +- [Embedding Usages](pooling_models/embed.md)

681 +- [Reward Usages](pooling_models/reward.md)

682 +- [Token Classification Usages](pooling_models/token_classify.md)

683 +- [Token Embedding Usages](pooling_models/token_embed.md)

684 +- [Scoring Usages](pooling_models/scoring.md)

685 +- [Specific Model Examples](pooling_models/specific_models.md)

686

687 ## Model Support Policy

688

docs/serving/offline_inference.md

+1 -1

@@ -16,7 +16,7 @@ After initializing the `LLM` instance, use the available APIs to perform model i

16 The available APIs depend on the model type:

18 - [Generative models](../models/generative_models.md) output logprobs which are sampled from to obtain the final output text.

19 -- [Pooling models](../models/pooling_models.md) output their hidden states directly.

19 +- [Pooling models](../models/pooling_models/README.md) output their hidden states directly.

21 !!! info

22 [API Reference](../api/README.md#offline-inference)

docs/serving/openai_compatible_server.md

+13 -773

@@ -53,8 +53,8 @@ We currently support the following OpenAI APIs:

53 - Only applicable to [text generation models](../models/generative_models.md) with a [chat template](../serving/openai_compatible_server.md#chat-template).

54 - *Note: `user` parameter is ignored.*

55 - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls.

56 -- [Embeddings API](#embeddings-api) (`/v1/embeddings`)

57 - - Only applicable to [embedding models](../models/pooling_models.md).

56 +- [Embeddings API](../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)

57 + - Only applicable to [embedding models](../models/pooling_models/embed.md).

58 - [Transcriptions API](#transcriptions-api) (`/v1/audio/transcriptions`)

59 - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).

60 - [Translation API](#translations-api) (`/v1/audio/translations`)

@@ -66,20 +66,19 @@ In addition, we have the following custom APIs:

67 - [Tokenizer API](#tokenizer-api) (`/tokenize`, `/detokenize`)

68 - Applicable to any model with a tokenizer.

69 -- [Pooling API](#pooling-api) (`/pooling`)

70 - - Applicable to all [pooling models](../models/pooling_models.md).

71 -- [Classification API](#classification-api) (`/classify`)

72 - - Only applicable to [classification models](../models/pooling_models.md).

73 -- [Score API](#score-api) (`/score`)

74 - - Applicable to [embedding models and cross-encoder models](../models/pooling_models.md).

75 -- [Cohere Embed API](#cohere-embed-api) (`/v2/embed`)

69 +- [pooling API](../models/pooling_models/README.md#pooling-api) (`/pooling`)

70 + - Applicable to all [pooling models](../models/pooling_models/README.md).

71 +- [Classification API](../models/pooling_models/classify.md#classification-api) (`/classify`)

72 + - Only applicable to [classification models](../models/pooling_models/classify.md).

73 +- [Cohere Embed API](../models/pooling_models/embed.md#cohere-embed-api) (`/v2/embed`)

74 - Compatible with [Cohere's Embed API](https://docs.cohere.com/reference/embed)

77 - - Works with any [embedding model](../models/pooling_models.md), including multimodal models.

78 -- [Re-rank API](#re-rank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)

79 - - Implements [Jina AI's v1 re-rank API](https://jina.ai/reranker/)

80 - - Also compatible with [Cohere's v1 & v2 re-rank APIs](https://docs.cohere.com/v2/reference/rerank)

75 + - Works with any [embedding model](../models/pooling_models/embed.md#supported-models), including multimodal models.

76 +- [Score API](../models/pooling_models/scoring.md#score-api) (`/score`)

77 + - Applicable to [score models](../models/pooling_models/scoring.md).

78 +- [Rerank API](../models/pooling_models/scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)

79 + - Implements [Jina AI's v1 rerank API](https://jina.ai/reranker/)

80 + - Also compatible with [Cohere's v1 & v2 rerank APIs](https://docs.cohere.com/v2/reference/rerank)

81 - Jina and Cohere's APIs are very similar; Jina's includes extra information in the rerank endpoint's response.

82 - - Only applicable to [cross-encoder models](../models/pooling_models.md).

83 ## Chat Template

@@ -269,300 +268,6 @@ The following extra parameters in the response object are supported:

268 --8<-- "vllm/entrypoints/openai/responses/protocol.py:responses-response-extra-params"

269 ```

270

272 -### Embeddings API

273 -

274 -Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);

275 -you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

276 -

277 -Code example: [examples/pooling/embed/openai_embedding_client.py](../../examples/pooling/embed/openai_embedding_client.py)

278 -

279 -If the model has a [chat template](../serving/openai_compatible_server.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))

280 -which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations:

281 -

282 -??? code

283 -

284 - ```python

285 - from openai import OpenAI

286 - from openai._types import NOT_GIVEN, NotGiven

287 - from openai.types.chat import ChatCompletionMessageParam

288 - from openai.types.create_embedding_response import CreateEmbeddingResponse

289 -

290 - def create_chat_embeddings(

291 - client: OpenAI,

292 - *,

293 - messages: list[ChatCompletionMessageParam],

294 - model: str,

295 - encoding_format: Union[Literal["base64", "float"], NotGiven] = NOT_GIVEN,

296 - ) -> CreateEmbeddingResponse:

297 - return client.post(

298 - "/embeddings",

299 - cast_to=CreateEmbeddingResponse,

300 - body={"messages": messages, "model": model, "encoding_format": encoding_format},

301 - )

302 - ```

303 -

304 -#### Multi-modal inputs

305 -

306 -You can pass multi-modal inputs to embedding models by defining a custom chat template for the server

307 -and passing a list of `messages` in the request. Refer to the examples below for illustration.

308 -

309 -=== "VLM2Vec"

310 -

311 - To serve the model:

312 -

313 - ```bash

314 - vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \

315 - --trust-remote-code \

316 - --max-model-len 4096 \

317 - --chat-template examples/pooling/embed/template/vlm2vec_phi3v.jinja

318 - ```

319 -

320 - !!! important

321 - Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--runner pooling`

322 - to run this model in embedding mode instead of text generation mode.

323 -

324 - The custom chat template is completely different from the original one for this model,

325 - and can be found here: [examples/pooling/embed/template/vlm2vec_phi3v.jinja](../../examples/pooling/embed/template/vlm2vec_phi3v.jinja)

326 -

327 - Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

328 -

329 - ??? code

330 -

331 - ```python

332 - from openai import OpenAI

333 - client = OpenAI(

334 - base_url="http://localhost:8000/v1",

335 - api_key="EMPTY",

336 - )

337 - image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

338 -

339 - response = create_chat_embeddings(

340 - client,

341 - model="TIGER-Lab/VLM2Vec-Full",

342 - messages=[

343 - {

344 - "role": "user",

345 - "content": [

346 - {"type": "image_url", "image_url": {"url": image_url}},

347 - {"type": "text", "text": "Represent the given image."},

348 - ],

349 - }

350 - ],

351 - encoding_format="float",

352 - )

353 -

354 - print("Image embedding output:", response.data[0].embedding)

355 - ```

356 -

357 -=== "DSE-Qwen2-MRL"

358 -

359 - To serve the model:

360 -

361 - ```bash

362 - vllm serve MrLight/dse-qwen2-2b-mrl-v1 --runner pooling \

363 - --trust-remote-code \

364 - --max-model-len 8192 \

365 - --chat-template examples/pooling/embed/template/dse_qwen2_vl.jinja

366 - ```

367 -

368 - !!! important

369 - Like with VLM2Vec, we have to explicitly pass `--runner pooling`.

370 -

371 - Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled

372 - by a custom chat template: [examples/pooling/embed/template/dse_qwen2_vl.jinja](../../examples/pooling/embed/template/dse_qwen2_vl.jinja)

373 -

374 - !!! important

375 - `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code

376 - example below for details.

377 -

378 -Full example: [examples/pooling/embed/vision_embedding_online.py](../../examples/pooling/embed/vision_embedding_online.py)

379 -

380 -#### Extra parameters

381 -

382 -The following [pooling parameters][vllm.PoolingParams] are supported.

383 -

384 -```python

385 ---8<-- "vllm/pooling_params.py:common-pooling-params"

386 ---8<-- "vllm/pooling_params.py:embed-pooling-params"

387 -```

388 -

389 -The following Embeddings API parameters are supported:

390 -

391 -??? code

392 -

393 - ```python

394 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

395 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-params"

396 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-params"

397 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-params"

398 - ```

399 -

400 -The following extra parameters are supported:

401 -

402 -??? code

403 -

404 - ```python

405 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

406 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-extra-params"

407 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-extra-params"

408 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-extra-params"

409 - ```

410 -

411 -For chat-like input (i.e. if `messages` is passed), the following parameters are supported:

412 -

413 -The following parameters are supported by default:

414 -

415 -??? code

416 -

417 - ```python

418 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

419 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-params"

420 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-params"

421 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-params"

422 - ```

423 -

424 -these extra parameters are supported instead:

425 -

426 -??? code

427 -

428 - ```python

429 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

430 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-extra-params"

431 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:encoding-extra-params"

432 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:embed-extra-params"

433 - ```

434 -

435 -### Cohere Embed API

436 -

437 -Our API is also compatible with [Cohere's Embed v2 API](https://docs.cohere.com/reference/embed) which adds support for some modern embedding feature such as truncation, output dimensions, embedding types, and input types. This endpoint works with any embedding model (including multimodal models).

438 -

439 -#### Cohere Embed API request parameters

440 -

442 -| --------- | ---- | -------- | ----------- |

449 -| `output_dimension` | int | No | Truncate embeddings to this dimension (Matryoshka) |

451 -

452 -#### Text embedding

453 -

454 -```bash

455 -curl -X POST "http://localhost:8000/v2/embed" \

456 - -H "Content-Type: application/json" \

457 - -d '{

458 - "model": "Snowflake/snowflake-arctic-embed-m-v1.5",

459 - "input_type": "query",

460 - "texts": ["Hello world", "How are you?"],

461 - "embedding_types": ["float"]

462 - }'

463 -```

464 -

465 -??? console "Response"

466 -

467 - ```json

468 - {

469 - "id": "embd-...",

470 - "embeddings": {

471 - "float": [

472 - [0.012, -0.034, ...],

473 - [0.056, 0.078, ...]

474 - ]

475 - },

476 - "texts": ["Hello world", "How are you?"],

477 - "meta": {

478 - "api_version": {"version": "2"},

479 - "billed_units": {"input_tokens": 12}

480 - }

481 - }

482 - ```

483 -

484 -#### Mixed text and image inputs

485 -

486 -For multimodal models, you can embed images by passing base64 data URIs. The `inputs` field accepts a list of objects with mixed text and image content:

487 -

488 -```bash

489 -curl -X POST "http://localhost:8000/v2/embed" \

490 - -H "Content-Type: application/json" \

491 - -d '{

492 - "model": "google/siglip-so400m-patch14-384",

493 - "inputs": [

494 - {

495 - "content": [

496 - {"type": "text", "text": "A photo of a cat"},

497 - {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBOR..."}}

498 - ]

499 - }

500 - ],

501 - "embedding_types": ["float"]

502 - }'

503 -```

504 -

505 -#### Embedding types

506 -

507 -The `embedding_types` parameter controls the output format. Multiple types can be requested in a single call:

508 -

509 -| Type | Description |

510 -| ---- | ----------- |

511 -| `float` | Raw float32 embeddings (default) |

512 -| `binary` | Bit-packed signed binary |

513 -| `ubinary` | Bit-packed unsigned binary |

514 -| `base64` | Little-endian float32 encoded as base64 |

515 -

516 -```bash

517 -curl -X POST "http://localhost:8000/v2/embed" \

518 - -H "Content-Type: application/json" \

519 - -d '{

520 - "model": "Snowflake/snowflake-arctic-embed-m-v1.5",

521 - "input_type": "query",

522 - "texts": ["What is machine learning?"],

523 - "embedding_types": ["float", "binary"]

524 - }'

525 -```

526 -

527 -??? console "Response"

528 -

529 - ```json

530 - {

531 - "id": "embd-...",

532 - "embeddings": {

533 - "float": [[0.012, -0.034, ...]],

534 - "binary": [[42, -117, ...]]

535 - },

536 - "texts": ["What is machine learning?"],

537 - "meta": {

538 - "api_version": {"version": "2"},

539 - "billed_units": {"input_tokens": 8}

540 - }

541 - }

542 - ```

543 -

544 -#### Truncation

545 -

546 -The `truncate` parameter controls how inputs exceeding the model's maximum sequence length are handled:

547 -

548 -| Value | Behavior |

549 -| ----- | --------- |

550 -| `END` (default) | Keep the first tokens, drop the end |

551 -| `START` | Keep the last tokens, drop the beginning |

552 -| `NONE` | Return an error if the input is too long |

553 -

554 -#### Input type and prompt prefixes

555 -

556 -The `input_type` field selects a prompt prefix to prepend to each text input. The available values

557 -depend on the model:

558 -

559 -- **Models with `task_instructions` in `config.json`**: The keys from the `task_instructions` dict are

560 - the valid `input_type` values and the corresponding value is prepended to each text.

561 -- **Models with `config_sentence_transformers.json` prompts**: The keys from the `prompts` dict are

562 - the valid `input_type` values. For example, `Snowflake/snowflake-arctic-embed-xs` defines `"query"`,

563 - so setting `input_type: "query"` prepends `"Represent this sentence for searching relevant passages: "`.

564 -- **Other models**: `input_type` is not accepted and will raise a validation error if passed.

565 -

271 ### Transcriptions API

272

273 Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription);

@@ -759,172 +464,8 @@ It consists of two endpoints:

464 - `/tokenize` corresponds to calling `tokenizer.encode()`.

465 - `/detokenize` corresponds to calling `tokenizer.decode()`.

466

762 -### Pooling API

763 -

764 -Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.

765 -

766 -The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.

767 -

768 -Code example: [examples/pooling/pooling/pooling_online.py](../../examples/pooling/pooling/pooling_online.py)

769 -

770 -### Classification API

771 -

772 -Our Classification API directly supports Hugging Face sequence-classification models such as [ai21labs/Jamba-tiny-reward-dev](https://huggingface.co/ai21labs/Jamba-tiny-reward-dev) and [jason9693/Qwen2.5-1.5B-apeach](https://huggingface.co/jason9693/Qwen2.5-1.5B-apeach).

773 -

774 -We automatically wrap any other transformer via `as_seq_cls_model()`, which pools on the last token, attaches a `RowParallelLinear` head, and applies a softmax to produce per-class probabilities.

775 -

776 -Code example: [examples/pooling/classify/classification_online.py](../../examples/pooling/classify/classification_online.py)

777 -

778 -#### Example Requests

779 -

780 -You can classify multiple texts by passing an array of strings:

781 -

782 -```bash

783 -curl -v "http://127.0.0.1:8000/classify" \

784 - -H "Content-Type: application/json" \

785 - -d '{

786 - "model": "jason9693/Qwen2.5-1.5B-apeach",

787 - "input": [

788 - "Loved the new café—coffee was great.",

789 - "This update broke everything. Frustrating."

790 - ]

791 - }'

792 -```

793 -

794 -??? console "Response"

795 -

796 - ```json

797 - {

798 - "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",

799 - "object": "list",

800 - "created": 1745383065,

801 - "model": "jason9693/Qwen2.5-1.5B-apeach",

802 - "data": [

803 - {

804 - "index": 0,

805 - "label": "Default",

806 - "probs": [

807 - 0.565970778465271,

808 - 0.4340292513370514

809 - ],

810 - "num_classes": 2

811 - },

812 - {

813 - "index": 1,

814 - "label": "Spoiled",

815 - "probs": [

816 - 0.26448777318000793,

817 - 0.7355121970176697

818 - ],

819 - "num_classes": 2

820 - }

821 - ],

822 - "usage": {

823 - "prompt_tokens": 20,

824 - "total_tokens": 20,

825 - "completion_tokens": 0,

826 - "prompt_tokens_details": null

827 - }

828 - }

829 - ```

830 -

831 -You can also pass a string directly to the `input` field:

832 -

833 -```bash

834 -curl -v "http://127.0.0.1:8000/classify" \

835 - -H "Content-Type: application/json" \

836 - -d '{

837 - "model": "jason9693/Qwen2.5-1.5B-apeach",

838 - "input": "Loved the new café—coffee was great."

839 - }'

840 -```

841 -

842 -??? console "Response"

843 -

844 - ```json

845 - {

846 - "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",

847 - "object": "list",

848 - "created": 1745383213,

849 - "model": "jason9693/Qwen2.5-1.5B-apeach",

850 - "data": [

851 - {

852 - "index": 0,

853 - "label": "Default",

854 - "probs": [

855 - 0.565970778465271,

856 - 0.4340292513370514

857 - ],

858 - "num_classes": 2

859 - }

860 - ],

861 - "usage": {

862 - "prompt_tokens": 10,

863 - "total_tokens": 10,

864 - "completion_tokens": 0,

865 - "prompt_tokens_details": null

866 - }

867 - }

868 - ```

869 -

870 -#### Extra parameters

871 -

872 -The following [pooling parameters][vllm.PoolingParams] are supported.

873 -

874 -```python

875 ---8<-- "vllm/pooling_params.py:common-pooling-params"

876 ---8<-- "vllm/pooling_params.py:classify-pooling-params"

877 -```

878 -

879 -The following Classification API parameters are supported:

880 -

881 -??? code

882 -

883 - ```python

884 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

885 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-params"

886 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-params"

887 - ```

888 -

889 -The following extra parameters are supported:

890 -

891 -??? code

892 -

893 - ```python

894 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

895 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:completion-extra-params"

896 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

897 - ```

898 -

899 -For chat-like input (i.e. if `messages` is passed), the following parameters are supported:

900 -

901 -??? code

902 -

903 - ```python

904 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

905 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-params"

906 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-params"

907 - ```

908 -

909 -these extra parameters are supported instead:

910 -

911 -??? code

912 -

913 - ```python

914 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

915 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:chat-extra-params"

916 - --8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

917 - ```

918 -

467 ### Score API

468

921 -Our Score API can apply a cross-encoder model or an embedding model to predict scores for sentence or multimodal pairs. When using an embedding model the score corresponds to the cosine similarity between each embedding pair.

922 -Usually, the score for a sentence pair refers to the similarity between two sentences, on a scale of 0 to 1.

923 -

924 -You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).

925 -

926 -Code example: [examples/pooling/score/score_api_online.py](../../examples/pooling/score/score_api_online.py)

927 -

469 #### Score Template

470

471 Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](#chat-template)).

@@ -940,307 +481,6 @@ This approach is more robust than index-based access (`messages[0]`, `messages[1

481

482 Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../examples/pooling/score/template/nemotron-rerank.jinja)

483

943 -#### Single inference

944 -

945 -You can pass a string to both `queries` and `documents`, forming a single sentence pair.

946 -

947 -```bash

948 -curl -X 'POST' \

949 - 'http://127.0.0.1:8000/score' \

950 - -H 'accept: application/json' \

951 - -H 'Content-Type: application/json' \

952 - -d '{

953 - "model": "BAAI/bge-reranker-v2-m3",

954 - "encoding_format": "float",

955 - "queries": "What is the capital of France?",

956 - "documents": "The capital of France is Paris."

957 -}'

958 -```

959 -

960 -??? console "Response"

961 -

962 - ```json

963 - {

964 - "id": "score-request-id",

965 - "object": "list",

966 - "created": 693447,

967 - "model": "BAAI/bge-reranker-v2-m3",

968 - "data": [

969 - {

970 - "index": 0,

971 - "object": "score",

972 - "score": 1

973 - }

974 - ],

975 - "usage": {}

976 - }

977 - ```

978 -

979 -#### Batch inference

980 -

981 -You can pass a string to `queries` and a list to `documents`, forming multiple sentence pairs

982 -where each pair is built from `queries` and a string in `documents`.

983 -The total number of pairs is `len(documents)`.

984 -

985 -??? console "Request"

986 -

987 - ```bash

988 - curl -X 'POST' \

989 - 'http://127.0.0.1:8000/score' \

990 - -H 'accept: application/json' \

991 - -H 'Content-Type: application/json' \

992 - -d '{

993 - "model": "BAAI/bge-reranker-v2-m3",

994 - "queries": "What is the capital of France?",

995 - "documents": [

996 - "The capital of Brazil is Brasilia.",

997 - "The capital of France is Paris."

998 - ]

999 - }'

1000 - ```

1001 -

1002 -??? console "Response"

1003 -

1004 - ```json

1005 - {

1006 - "id": "score-request-id",

1007 - "object": "list",

1008 - "created": 693570,

1009 - "model": "BAAI/bge-reranker-v2-m3",

1010 - "data": [

1011 - {

1012 - "index": 0,

1013 - "object": "score",

1014 - "score": 0.001094818115234375

1015 - },

1016 - {

1017 - "index": 1,

1018 - "object": "score",

1019 - "score": 1

1020 - }

1021 - ],

1022 - "usage": {}

1023 - }

1024 - ```

1025 -

1026 -You can pass a list to both `queries` and `documents`, forming multiple sentence pairs

1027 -where each pair is built from a string in `queries` and the corresponding string in `documents` (similar to `zip()`).

1028 -The total number of pairs is `len(documents)`.

1029 -

1030 -??? console "Request"

1031 -

1032 - ```bash

1033 - curl -X 'POST' \

1034 - 'http://127.0.0.1:8000/score' \

1035 - -H 'accept: application/json' \

1036 - -H 'Content-Type: application/json' \

1037 - -d '{

1038 - "model": "BAAI/bge-reranker-v2-m3",

1039 - "encoding_format": "float",

1040 - "queries": [

1041 - "What is the capital of Brazil?",

1042 - "What is the capital of France?"

1043 - ],

1044 - "documents": [

1045 - "The capital of Brazil is Brasilia.",

1046 - "The capital of France is Paris."

1047 - ]

1048 - }'

1049 - ```

1050 -

1051 -??? console "Response"

1052 -

1053 - ```json

1054 - {

1055 - "id": "score-request-id",

1056 - "object": "list",

1057 - "created": 693447,

1058 - "model": "BAAI/bge-reranker-v2-m3",

1059 - "data": [

1060 - {

1061 - "index": 0,

1062 - "object": "score",

1063 - "score": 1

1064 - },

1065 - {

1066 - "index": 1,

1067 - "object": "score",

1068 - "score": 1

1069 - }

1070 - ],

1071 - "usage": {}

1072 - }

1073 - ```

1074 -

1075 -#### Multi-modal inputs

1076 -

1077 -You can pass multi-modal inputs to scoring models by passing `content` including a list of multi-modal input (image, etc.) in the request. Refer to the examples below for illustration.

1078 -

1079 -=== "JinaVL-Reranker"

1080 -

1081 - To serve the model:

1082 -

1083 - ```bash

1084 - vllm serve jinaai/jina-reranker-m0

1085 - ```

1086 -

1087 - Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

1088 -

1089 - ??? Code

1090 -

1091 - ```python

1092 - import requests

1093 -

1094 - response = requests.post(

1095 - "http://localhost:8000/v1/score",

1096 - json={

1097 - "model": "jinaai/jina-reranker-m0",

1098 - "queries": "slm markdown",

1099 - "documents": [

1100 - {

1101 - "content": [

1102 - {

1103 - "type": "image_url",

1104 - "image_url": {

1105 - "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"

1106 - },

1107 - }

1108 - ],

1109 - },

1110 - {

1111 - "content": [

1112 - {

1113 - "type": "image_url",

1114 - "image_url": {

1115 - "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"

1116 - },

1117 - }

1118 - ]

1119 - },

1120 - ],

1121 - },

1122 - )

1123 - response.raise_for_status()

1124 - response_json = response.json()

1125 - print("Scoring output:", response_json["data"][0]["score"])

1126 - print("Scoring output:", response_json["data"][1]["score"])

1127 - ```

1128 -Full example:

1129 -

1130 -- [examples/pooling/score/vision_score_api_online.py](../../examples/pooling/score/vision_score_api_online.py)

1131 -- [examples/pooling/score/vision_rerank_api_online.py](../../examples/pooling/score/vision_rerank_api_online.py)

1132 -

1133 -#### Extra parameters

1134 -

1135 -The following [pooling parameters][vllm.PoolingParams] are supported.

1136 -

1137 -```python

1138 ---8<-- "vllm/pooling_params.py:common-pooling-params"

1139 ---8<-- "vllm/pooling_params.py:classify-pooling-params"

1140 -```

1141 -

1142 -The following Score API parameters are supported:

1143 -

1144 -```python

1145 ---8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

1146 -```

1147 -

1148 -The following extra parameters are supported:

1149 -

1150 -```python

1151 ---8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

1152 ---8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

1153 -```

1154 -

1155 -### Re-rank API

1156 -

1157 -Our Re-rank API can apply an embedding model or a cross-encoder model to predict relevant scores between a single query, and

1158 -each of a list of documents. Usually, the score for a sentence pair refers to the similarity between two sentences or multi-modal inputs (image, etc.), on a scale of 0 to 1.

1159 -

1160 -You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).

1161 -

1162 -The rerank endpoints support popular re-rank models such as `BAAI/bge-reranker-base` and other models supporting the

1163 -`score` task. Additionally, `/rerank`, `/v1/rerank`, and `/v2/rerank`

1164 -endpoints are compatible with both [Jina AI's re-rank API interface](https://jina.ai/reranker/) and

1165 -[Cohere's re-rank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with

1166 -popular open-source tools.

1167 -

1168 -Code example: [examples/pooling/score/rerank_api_online.py](../../examples/pooling/score/rerank_api_online.py)

1169 -

1170 -#### Example Request

1171 -

1172 -Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.

1173 -Result documents will be sorted by relevance, and the `index` property can be used to determine original order.

1174 -

1175 -??? console "Request"

1176 -

1177 - ```bash

1178 - curl -X 'POST' \

1179 - 'http://127.0.0.1:8000/v1/rerank' \

1180 - -H 'accept: application/json' \

1181 - -H 'Content-Type: application/json' \

1182 - -d '{

1183 - "model": "BAAI/bge-reranker-base",

1184 - "query": "What is the capital of France?",

1185 - "documents": [

1186 - "The capital of Brazil is Brasilia.",

1187 - "The capital of France is Paris.",

1188 - "Horses and cows are both animals"

1189 - ]

1190 - }'

1191 - ```

1192 -

1193 -??? console "Response"

1194 -

1195 - ```json

1196 - {

1197 - "id": "rerank-fae51b2b664d4ed38f5969b612edff77",

1198 - "model": "BAAI/bge-reranker-base",

1199 - "usage": {

1200 - "total_tokens": 56

1201 - },

1202 - "results": [

1203 - {

1204 - "index": 1,

1205 - "document": {

1206 - "text": "The capital of France is Paris."

1207 - },

1208 - "relevance_score": 0.99853515625

1209 - },

1210 - {

1211 - "index": 0,

1212 - "document": {

1213 - "text": "The capital of Brazil is Brasilia."

1214 - },

1215 - "relevance_score": 0.0005860328674316406

1216 - }

1217 - ]

1218 - }

1219 - ```

1220 -

1221 -#### Extra parameters

1222 -

1223 -The following [pooling parameters][vllm.PoolingParams] are supported.

1224 -

1225 -```python

1226 ---8<-- "vllm/pooling_params.py:common-pooling-params"

1227 ---8<-- "vllm/pooling_params.py:classify-pooling-params"

1228 -```

1229 -

1230 -The following Re-rank API parameters are supported:

1231 -

1232 -```python

1233 ---8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"

1234 ---8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

1235 -```

1236 -

1237 -The following extra parameters are supported:

1238 -

1239 -```python

1240 ---8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"

1241 ---8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"

1242 -```

1243 -

484 ## Ray Serve LLM

485

486 Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure.