@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 463 lines 15.5 KB
Rendered Raw
1 # Specific Model Examples
2
3 ## ColBERT Late Interaction Models
4
5 [ColBERT](https://arxiv.org/abs/2004.12832) (Contextualized Late Interaction over BERT) is a retrieval model that uses per-token embeddings and MaxSim scoring for document ranking. Unlike single-vector embedding models, ColBERT retains token-level representations and computes relevance scores through late interaction, providing better accuracy while being more efficient than cross-encoders.
6
7 vLLM supports ColBERT models with multiple encoder backbones:
8
9 | Architecture | Backbone | Example HF Models |
10 | - | - | - |
11 | `HF_ColBERT` | BERT | `answerdotai/answerai-colbert-small-v1`, `colbert-ir/colbertv2.0` |
12 | `ColBERTModernBertModel` | ModernBERT | `lightonai/GTE-ModernColBERT-v1` |
13 | `ColBERTJinaRobertaModel` | Jina XLM-RoBERTa | `jinaai/jina-colbert-v2` |
14 | `ColBERTLfm2Model` | LFM2 | `LiquidAI/LFM2-ColBERT-350M` |
15
16 **BERT-based ColBERT** models work out of the box:
17
18 ```shell
19 vllm serve answerdotai/answerai-colbert-small-v1
20 ```
21
22 For **non-BERT backbones**, use `--hf-overrides` to set the correct architecture:
23
24 ```shell
25 # ModernBERT backbone
26 vllm serve lightonai/GTE-ModernColBERT-v1 \
27 --hf-overrides '{"architectures": ["ColBERTModernBertModel"]}'
28
29 # Jina XLM-RoBERTa backbone
30 vllm serve jinaai/jina-colbert-v2 \
31 --hf-overrides '{"architectures": ["ColBERTJinaRobertaModel"]}' \
32 --trust-remote-code
33
34 # LFM2 backbone
35 vllm serve LiquidAI/LFM2-ColBERT-350M \
36 --hf-overrides '{"architectures": ["ColBERTLfm2Model"]}'
37 ```
38
39 Then you can use the rerank API:
40
41 ```shell
42 curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{
43 "model": "answerdotai/answerai-colbert-small-v1",
44 "query": "What is machine learning?",
45 "documents": [
46 "Machine learning is a subset of artificial intelligence.",
47 "Python is a programming language.",
48 "Deep learning uses neural networks."
49 ]
50 }'
51 ```
52
53 Or the score API:
54
55 ```shell
56 curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{
57 "model": "answerdotai/answerai-colbert-small-v1",
58 "text_1": "What is machine learning?",
59 "text_2": ["Machine learning is a subset of AI.", "The weather is sunny."]
60 }'
61 ```
62
63 You can also get the raw token embeddings using the Pooling API with `token_embed` task:
64
65 ```shell
66 curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
67 "model": "answerdotai/answerai-colbert-small-v1",
68 "input": "What is machine learning?",
69 "task": "token_embed"
70 }'
71 ```
72
73 An example can be found here: [examples/pooling/score/colbert_rerank_online.py](../../../examples/pooling/score/colbert_rerank_online.py)
74
75 ## ColQwen3 Multi-Modal Late Interaction Models
76
77 ColQwen3 is based on [ColPali](https://arxiv.org/abs/2407.01449), which extends ColBERT's late interaction approach to **multi-modal** inputs. While ColBERT operates on text-only token embeddings, ColPali/ColQwen3 can embed both **text and images** (e.g. PDF pages, screenshots, diagrams) into per-token L2-normalized vectors and compute relevance via MaxSim scoring. ColQwen3 specifically uses Qwen3-VL as its vision-language backbone.
78
79 | Architecture | Backbone | Example HF Models |
80 | - | - | - |
81 | `ColQwen3` | Qwen3-VL | `TomoroAI/tomoro-colqwen3-embed-4b`, `TomoroAI/tomoro-colqwen3-embed-8b` |
82 | `OpsColQwen3Model` | Qwen3-VL | `OpenSearch-AI/Ops-Colqwen3-4B`, `OpenSearch-AI/Ops-Colqwen3-8B` |
83 | `Qwen3VLNemotronEmbedModel` | Qwen3-VL | `nvidia/nemotron-colembed-vl-4b-v2`, `nvidia/nemotron-colembed-vl-8b-v2` |
84
85 Start the server:
86
87 ```shell
88 vllm serve TomoroAI/tomoro-colqwen3-embed-4b --max-model-len 4096
89 ```
90
91 ### Text-only scoring and reranking
92
93 Use the `/rerank` API:
94
95 ```shell
96 curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{
97 "model": "TomoroAI/tomoro-colqwen3-embed-4b",
98 "query": "What is machine learning?",
99 "documents": [
100 "Machine learning is a subset of artificial intelligence.",
101 "Python is a programming language.",
102 "Deep learning uses neural networks."
103 ]
104 }'
105 ```
106
107 Or the `/score` API:
108
109 ```shell
110 curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{
111 "model": "TomoroAI/tomoro-colqwen3-embed-4b",
112 "text_1": "What is the capital of France?",
113 "text_2": ["The capital of France is Paris.", "Python is a programming language."]
114 }'
115 ```
116
117 ### Multi-modal scoring and reranking (text query × image documents)
118
119 The `/score` and `/rerank` APIs also accept multi-modal inputs directly.
120 Pass image documents using the `data_1`/`data_2` (for `/score`) or `documents` (for `/rerank`) fields
121 with a `content` list containing `image_url` and `text` parts — the same format used by the
122 OpenAI chat completion API:
123
124 Score a text query against image documents:
125
126 ```shell
127 curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{
128 "model": "TomoroAI/tomoro-colqwen3-embed-4b",
129 "data_1": "Retrieve the city of Beijing",
130 "data_2": [
131 {
132 "content": [
133 {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
134 {"type": "text", "text": "Describe the image."}
135 ]
136 }
137 ]
138 }'
139 ```
140
141 Rerank image documents by a text query:
142
143 ```shell
144 curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{
145 "model": "TomoroAI/tomoro-colqwen3-embed-4b",
146 "query": "Retrieve the city of Beijing",
147 "documents": [
148 {
149 "content": [
150 {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_1>"}},
151 {"type": "text", "text": "Describe the image."}
152 ]
153 },
154 {
155 "content": [
156 {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_2>"}},
157 {"type": "text", "text": "Describe the image."}
158 ]
159 }
160 ],
161 "top_n": 2
162 }'
163 ```
164
165 ### Raw token embeddings
166
167 You can also get the raw token embeddings using the `/pooling` API with `token_embed` task:
168
169 ```shell
170 curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
171 "model": "TomoroAI/tomoro-colqwen3-embed-4b",
172 "input": "What is machine learning?",
173 "task": "token_embed"
174 }'
175 ```
176
177 For **image inputs** via the Pooling API, use the chat-style `messages` field:
178
179 ```shell
180 curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
181 "model": "TomoroAI/tomoro-colqwen3-embed-4b",
182 "messages": [
183 {
184 "role": "user",
185 "content": [
186 {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
187 {"type": "text", "text": "Describe the image."}
188 ]
189 }
190 ]
191 }'
192 ```
193
194 ### Examples
195
196 - Multi-vector retrieval: [examples/pooling/token_embed/colqwen3_token_embed_online.py](../../../examples/pooling/token_embed/colqwen3_token_embed_online.py)
197 - Reranking (text + multi-modal): [examples/pooling/score/colqwen3_rerank_online.py](../../../examples/pooling/score/colqwen3_rerank_online.py)
198
199 ## ColQwen3.5 Multi-Modal Late Interaction Models
200
201 ColQwen3.5 is based on [ColPali](https://arxiv.org/abs/2407.01449), extending ColBERT's late interaction approach to **multi-modal** inputs. It uses the Qwen3.5 hybrid backbone (linear + full attention) and produces per-token L2-normalized vectors for MaxSim scoring.
202
203 | Architecture | Backbone | Example HF Models |
204 | - | - | - |
205 | `ColQwen3_5` | Qwen3.5 | `athrael-soju/colqwen3.5-4.5B` |
206
207 Start the server:
208
209 ```shell
210 vllm serve athrael-soju/colqwen3.5-4.5B --max-model-len 4096
211 ```
212
213 Then you can use the rerank endpoint:
214
215 ```shell
216 curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{
217 "model": "athrael-soju/colqwen3.5-4.5B",
218 "query": "What is machine learning?",
219 "documents": [
220 "Machine learning is a subset of artificial intelligence.",
221 "Python is a programming language.",
222 "Deep learning uses neural networks."
223 ]
224 }'
225 ```
226
227 Or the score endpoint:
228
229 ```shell
230 curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{
231 "model": "athrael-soju/colqwen3.5-4.5B",
232 "text_1": "What is the capital of France?",
233 "text_2": ["The capital of France is Paris.", "Python is a programming language."]
234 }'
235 ```
236
237 An example can be found here: [examples/pooling/score/colqwen3_5_rerank_online.py](../../../examples/pooling/score/colqwen3_5_rerank_online.py)
238
239 ## Llama Nemotron Multimodal
240
241 ### Embedding Model
242
243 Llama Nemotron VL Embedding models combine the bidirectional Llama embedding backbone
244 (from `nvidia/llama-nemotron-embed-1b-v2`) with SigLIP as the vision encoder to produce
245 single-vector embeddings from text and/or images.
246
247 | Architecture | Backbone | Example HF Models |
248 | - | - | - |
249 | `LlamaNemotronVLModel` | Bidirectional Llama + SigLIP | `nvidia/llama-nemotron-embed-vl-1b-v2` |
250
251 Start the server:
252
253 ```shell
254 vllm serve nvidia/llama-nemotron-embed-vl-1b-v2 \
255 --trust-remote-code \
256 --chat-template examples/pooling/embed/template/nemotron_embed_vl.jinja
257 ```
258
259 !!! note
260 The chat template bundled with this model's tokenizer is not suitable for
261 the embeddings API. Use the provided override template above when serving
262 with the `messages`-based (chat-style) embeddings API.
263
264 The override template uses the message `role` to automatically prepend the
265 appropriate prefix: set `role` to `"query"` for queries (prepends `query: `)
266 or `"document"` for passages (prepends `passage: `). Any other role omits
267 the prefix.
268
269 Embed text queries:
270
271 ```shell
272 curl -s http://localhost:8000/v1/embeddings -H "Content-Type: application/json" -d '{
273 "model": "nvidia/llama-nemotron-embed-vl-1b-v2",
274 "messages": [
275 {
276 "role": "query",
277 "content": [
278 {"type": "text", "text": "What is machine learning?"}
279 ]
280 }
281 ]
282 }'
283 ```
284
285 Embed images via the chat-style `messages` field:
286
287 ```shell
288 curl -s http://localhost:8000/v1/embeddings -H "Content-Type: application/json" -d '{
289 "model": "nvidia/llama-nemotron-embed-vl-1b-v2",
290 "messages": [
291 {
292 "role": "document",
293 "content": [
294 {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
295 {"type": "text", "text": "Describe the image."}
296 ]
297 }
298 ]
299 }'
300 ```
301
302 ### Reranker Model
303
304 Llama Nemotron VL reranker models combine the same bidirectional Llama + SigLIP
305 backbone with a sequence-classification head for cross-encoder scoring and reranking.
306
307 | Architecture | Backbone | Example HF Models |
308 | - | - | - |
309 | `LlamaNemotronVLForSequenceClassification` | Bidirectional Llama + SigLIP | `nvidia/llama-nemotron-rerank-vl-1b-v2` |
310
311 Start the server:
312
313 ```shell
314 vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \
315 --runner pooling \
316 --trust-remote-code \
317 --chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja
318 ```
319
320 !!! note
321 The chat template bundled with this checkpoint's tokenizer is not suitable
322 for the Score/Rerank APIs. Use the provided override template when serving:
323 `examples/pooling/score/template/nemotron-vl-rerank.jinja`.
324
325 Score a text query against an image document:
326
327 ```shell
328 curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{
329 "model": "nvidia/llama-nemotron-rerank-vl-1b-v2",
330 "data_1": "Find diagrams about autonomous robots",
331 "data_2": [
332 {
333 "content": [
334 {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
335 {"type": "text", "text": "Robotics workflow diagram."}
336 ]
337 }
338 ]
339 }'
340 ```
341
342 Rerank image documents by a text query:
343
344 ```shell
345 curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{
346 "model": "nvidia/llama-nemotron-rerank-vl-1b-v2",
347 "query": "Find diagrams about autonomous robots",
348 "documents": [
349 {
350 "content": [
351 {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_1>"}},
352 {"type": "text", "text": "Robotics workflow diagram."}
353 ]
354 },
355 {
356 "content": [
357 {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_2>"}},
358 {"type": "text", "text": "General skyline photo."}
359 ]
360 }
361 ],
362 "top_n": 2
363 }'
364 ```
365
366 ## BAAI/bge-m3
367
368 `BAAI/bge-m3` supports dense retrieval, lexical matching, and ColBERT-style
369 multi-vector retrieval. Its `config.json` declares `XLMRobertaModel`, so vLLM
370 otherwise loads it as a vanilla RoBERTa model without the extra sparse and
371 ColBERT weights. The examples below therefore override the architecture with
372 `BgeM3EmbeddingModel`.
373
374 The three retrieval modes map to concrete pooling tasks as follows:
375
376 | Retrieval mode | Pooling task | Output |
377 | -------------- | ------------ | ------ |
378 | Dense | `embed` | One embedding vector per input |
379 | Lexical/sparse | `token_classify` | One scalar weight per non-special token |
380 | ColBERT multi-vector | `token_embed` | One embedding vector per non-special token |
381
382 Serve one concrete mode by selecting its task at load time:
383
384 ```shell
385 vllm serve BAAI/bge-m3 \
386 --runner pooling \
387 --hf-overrides '{"architectures": ["BgeM3EmbeddingModel"]}' \
388 --pooler-config.task <task>
389 ```
390
391 For dense embeddings, replace `<task>` with `embed` and use the Embeddings API:
392
393 ```shell
394 curl -s http://localhost:8000/v1/embeddings \
395 -H "Content-Type: application/json" -d '{
396 "model": "BAAI/bge-m3",
397 "input": ["What is BGE M3?", "Definition of BM25"]
398 }'
399 ```
400
401 For lexical weights, replace `<task>` with `token_classify` and use the
402 Pooling API:
403
404 ```shell
405 curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
406 "model": "BAAI/bge-m3",
407 "task": "token_classify",
408 "input": ["What is BGE M3?", "Definition of BM25"]
409 }'
410 ```
411
412 Due to limitations in the output schema, the output consists of a list of
413 token scores for each input. Call `/tokenize` as well to pair token IDs with
414 their scores. See
415 [`test_bge_m3.py`](../../../tests/models/language/pooling/test_bge_m3.py) for a
416 complete example that also combines repeated token IDs.
417
418 For ColBERT vectors, replace `<task>` with `token_embed` and use the Pooling API:
419
420 ```shell
421 curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
422 "model": "BAAI/bge-m3",
423 "task": "token_embed",
424 "input": ["What is BGE M3?", "Definition of BM25"]
425 }'
426 ```
427
428 ### Dense and sparse output through an IO processor plugin
429
430 The source tree includes a reference
431 [BGE-M3 IO processor plugin](../../../tests/plugins/bge_m3_sparse_plugin) that
432 formats dense embeddings, sparse token weights, or both in one response. From
433 a source checkout, install it in the vLLM environment and load it as follows:
434
435 ```shell
436 uv pip install ./tests/plugins/bge_m3_sparse_plugin
437
438 vllm serve BAAI/bge-m3 \
439 --runner pooling \
440 --hf-overrides '{"architectures": ["BgeM3EmbeddingModel"]}' \
441 --io-processor-plugin bge_m3_sparse_plugin
442 ```
443
444 The plugin selects the internal `embed&token_classify` task so the model
445 computes dense and lexical outputs together. Public requests must use task
446 `plugin` and put the plugin-specific fields under `data`:
447
448 ```shell
449 curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
450 "model": "BAAI/bge-m3",
451 "task": "plugin",
452 "data": {
453 "input": ["What is BGE M3?", "Definition of BM25"],
454 "embed_task": "dense&sparse",
455 "return_tokens": true
456 }
457 }'
458 ```
459
460 `embed_task` accepts `dense`, `sparse`, or `dense&sparse`. The combined
461 `embed&token_classify` task is an internal execution contract for this plugin,
462 not a generic Pooling API response format. Without the plugin, select one of
463 the three concrete tasks above.