@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 5719a4e4e6

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 8, 00:00 UTC

[Frontend] Support multimodal inputs for late-interaction scoring (ColQwen3) + NewModel: nvidia/nemotron-colembed (#34574)

Signed-off-by: craftsangjae <craftsangjae@gmail.com>

Kata Coder committed Feb 21, 2026 at 13:01 UTC 5719a4e4e601fb91274294d25370b7aad656d629

10 files changed +532 -66

docs/models/pooling_models.md

+59 -6

@@ -382,6 +382,7 @@ ColQwen3 is based on [ColPali](https://arxiv.org/abs/2407.01449), which extends

382 |---|---|---|

383 | `ColQwen3` | Qwen3-VL | `TomoroAI/tomoro-colqwen3-embed-4b`, `TomoroAI/tomoro-colqwen3-embed-8b` |

384 | `OpsColQwen3Model` | Qwen3-VL | `OpenSearch-AI/Ops-Colqwen3-4B`, `OpenSearch-AI/Ops-Colqwen3-8B` |

385 +| `Qwen3VLNemotronEmbedModel` | Qwen3-VL | `nvidia/nemotron-colembed-vl-4b-v2`, `nvidia/nemotron-colembed-vl-8b-v2` |

386

387 Start the server:

388

@@ -389,7 +390,9 @@ Start the server:

390 vllm serve TomoroAI/tomoro-colqwen3-embed-4b --max-model-len 4096

391 ```

392

392 -Then you can use the rerank endpoint:

393 +#### Text-only scoring and reranking

394 +

395 +Use the `/rerank` endpoint:

396

397 ```shell

398 curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

@@ -403,7 +406,7 @@ curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

406 }'

407 ```

408

406 -Or the score endpoint:

409 +Or the `/score` endpoint:

410

411 ```shell

412 curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

@@ -413,7 +416,57 @@ curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

416 }'

417 ```

418

416 -You can also get the raw token embeddings using the pooling endpoint with `token_embed` task:

419 +#### Multi-modal scoring and reranking (text query × image documents)

420 +

421 +The `/score` and `/rerank` endpoints also accept multi-modal inputs directly.

422 +Pass image documents using the `data_1`/`data_2` (for `/score`) or `documents` (for `/rerank`) fields

423 +with a `content` list containing `image_url` and `text` parts — the same format used by the

424 +OpenAI chat completion API:

425 +

426 +Score a text query against image documents:

427 +

428 +```shell

429 +curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{

430 + "model": "TomoroAI/tomoro-colqwen3-embed-4b",

431 + "data_1": "Retrieve the city of Beijing",

432 + "data_2": [

433 + {

434 + "content": [

435 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},

436 + {"type": "text", "text": "Describe the image."}

437 + ]

438 + }

439 + ]

440 +}'

441 +```

442 +

443 +Rerank image documents by a text query:

444 +

445 +```shell

446 +curl -s http://localhost:8000/rerank -H "Content-Type: application/json" -d '{

447 + "model": "TomoroAI/tomoro-colqwen3-embed-4b",

448 + "query": "Retrieve the city of Beijing",

449 + "documents": [

450 + {

451 + "content": [

452 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_1>"}},

453 + {"type": "text", "text": "Describe the image."}

454 + ]

455 + },

456 + {

457 + "content": [

458 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64_2>"}},

459 + {"type": "text", "text": "Describe the image."}

460 + ]

461 + }

462 + ],

463 + "top_n": 2

464 +}'

465 +```

466 +

467 +#### Raw token embeddings

468 +

469 +You can also get the raw token embeddings using the `/pooling` endpoint with `token_embed` task:

470

471 ```shell

472 curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

@@ -423,7 +476,7 @@ curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

476 }'

477 ```

478

426 -For **image inputs**, use the chat-style `messages` field so that the vLLM multimodal processor handles them correctly:

479 +For **image inputs** via the pooling endpoint, use the chat-style `messages` field:

480

481 ```shell

482 curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

@@ -440,10 +493,10 @@ curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{

493 }'

494 ```

495

443 -Examples can be found here:

496 +#### Examples

497

498 - Multi-vector retrieval: [examples/pooling/token_embed/colqwen3_token_embed_online.py](../../examples/pooling/token_embed/colqwen3_token_embed_online.py)

446 -- Reranking: [examples/pooling/score/colqwen3_rerank_online.py](../../examples/pooling/score/colqwen3_rerank_online.py)

499 +- Reranking (text + multi-modal): [examples/pooling/score/colqwen3_rerank_online.py](../../examples/pooling/score/colqwen3_rerank_online.py)

500

501 ### BAAI/bge-m3

502

examples/pooling/score/colqwen3_rerank_online.py

+129 -1

@@ -1,7 +1,8 @@

1 # SPDX-License-Identifier: Apache-2.0

2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +# ruff: noqa: E501

4 """

4 -Example of using ColQwen3 late interaction model for reranking.

5 +Example of using ColQwen3 late interaction model for reranking and scoring.

7 ColQwen3 is a multi-modal ColBERT-style model based on Qwen3-VL.

8 It produces per-token embeddings and uses MaxSim scoring for retrieval

@@ -14,13 +15,65 @@ Then run this script:

15 python colqwen3_rerank_online.py

16 """

18 +import base64

19 +from io import BytesIO

20 +

21 import requests

22 +from PIL import Image

24 MODEL = "TomoroAI/tomoro-colqwen3-embed-4b"

25 BASE_URL = "http://127.0.0.1:8000"

27 headers = {"accept": "application/json", "Content-Type": "application/json"}

29 +# ── Image helpers ──────────────────────────────────────────

30 +

31 +

32 +def load_image(url: str) -> Image.Image:

33 + """Download an image from URL (handles Wikimedia 403)."""

34 + for hdrs in (

35 + {},

36 + {"User-Agent": "Mozilla/5.0 (compatible; ColQwen3-demo/1.0)"},

37 + ):

38 + resp = requests.get(url, headers=hdrs, timeout=15)

39 + if resp.status_code == 403:

40 + continue

41 + resp.raise_for_status()

42 + return Image.open(BytesIO(resp.content)).convert("RGB")

43 + raise RuntimeError(f"Could not fetch image from {url}")

44 +

45 +

46 +def encode_image_base64(image: Image.Image) -> str:

47 + """Encode a PIL image to a base64 data URI."""

48 + buf = BytesIO()

49 + image.save(buf, format="PNG")

50 + return "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode()

51 +

52 +

53 +def make_image_content(image_url: str, text: str = "Describe the image.") -> dict:

54 + """Build a ScoreMultiModalParam dict from an image URL."""

55 + image = load_image(image_url)

56 + return {

57 + "content": [

58 + {

59 + "type": "image_url",

60 + "image_url": {"url": encode_image_base64(image)},

61 + },

62 + {"type": "text", "text": text},

63 + ]

64 + }

65 +

66 +

67 +# ── Sample image URLs ─────────────────────────────────────

68 +

69 +IMAGE_URLS = {

70 + "beijing": "https://upload.wikimedia.org/wikipedia/commons/6/61/Beijing_skyline_at_night.JPG",

71 + "london": "https://upload.wikimedia.org/wikipedia/commons/4/49/London_skyline.jpg",

72 + "singapore": "https://upload.wikimedia.org/wikipedia/commons/2/27/Singapore_skyline_2022.jpg",

73 +}

74 +

75 +# ── Text-only examples ────────────────────────────────────

76 +

78 def rerank_text():

79 """Text-only reranking via /rerank endpoint."""

@@ -120,11 +173,86 @@ def score_text_top_n():

173 print(f" {response.text[:300]}")

174

175

176 +# ── Multi-modal examples (text query × image documents) ──

177 +

178 +

179 +def score_text_vs_images():

180 + """Score a text query against image documents via /score."""

181 + print()

182 + print("=" * 60)

183 + print("4. Multi-modal scoring: text query vs image docs (/score)")

184 + print("=" * 60)

185 +

186 + query = "Retrieve the city of Beijing"

187 + labels = list(IMAGE_URLS.keys())

188 + print(f"\n Loading {len(labels)} images...")

189 + image_contents = [make_image_content(IMAGE_URLS[name]) for name in labels]

190 +

191 + data = {

192 + "model": MODEL,

193 + "data_1": query,

194 + "data_2": image_contents,

195 + }

196 +

197 + response = requests.post(f"{BASE_URL}/score", headers=headers, json=data)

198 +

199 + if response.status_code == 200:

200 + result = response.json()

201 + print(f'\n Query: "{query}"\n')

202 + for item in result["data"]:

203 + idx = item["index"]

204 + print(f" Doc {idx} [{labels[idx]}] score={item['score']:.4f}")

205 + else:

206 + print(f" Request failed: {response.status_code}")

207 + print(f" {response.text[:300]}")

208 +

209 +

210 +def rerank_text_vs_images():

211 + """Rerank image documents by a text query via /rerank."""

212 + print()

213 + print("=" * 60)

214 + print("5. Multi-modal reranking: text query vs image docs (/rerank)")

215 + print("=" * 60)

216 +

217 + query = "Retrieve the city of London"

218 + labels = list(IMAGE_URLS.keys())

219 + print(f"\n Loading {len(labels)} images...")

220 + image_contents = [make_image_content(IMAGE_URLS[name]) for name in labels]

221 +

222 + data = {

223 + "model": MODEL,

224 + "query": query,

225 + "documents": image_contents,

226 + "top_n": 2,

227 + }

228 +

229 + response = requests.post(f"{BASE_URL}/rerank", headers=headers, json=data)

230 +

231 + if response.status_code == 200:

232 + result = response.json()

233 + print(f'\n Query: "{query}"')

234 + print(f" Top {data['top_n']} results:\n")

235 + for item in result["results"]:

236 + idx = item["index"]

237 + print(f" [{item['relevance_score']:.4f}] {labels[idx]}")

238 + else:

239 + print(f" Request failed: {response.status_code}")

240 + print(f" {response.text[:300]}")

241 +

242 +

243 +# ── Main ──────────────────────────────────────────────────

244 +

245 +

246 def main():

247 + # Text-only

248 rerank_text()

249 score_text()

250 score_text_top_n()

251

252 + # Multi-modal (text query × image documents)

253 + score_text_vs_images()

254 + rerank_text_vs_images()

255 +

256

257 if __name__ == "__main__":

258 main()

tests/models/multimodal/pooling/test_colqwen3.py

+191

@@ -7,19 +7,31 @@ ColBERT-style late interaction scoring (MaxSim). It produces per-token

7 embeddings for both text and image inputs.

8 """

10 +import base64

11 +from io import BytesIO

12 +

13 import pytest

14 import torch

15 +from PIL import Image

16 +

17 +from vllm.entrypoints.chat_utils import (

18 + ChatCompletionContentPartImageParam,

19 + ChatCompletionContentPartTextParam,

20 +)

21 +from vllm.entrypoints.pooling.score.utils import ScoreMultiModalParam

23 from ....conftest import VllmRunner

25 MODELS = [

26 "TomoroAI/tomoro-colqwen3-embed-4b",

27 "OpenSearch-AI/Ops-Colqwen3-4B",

28 + "nvidia/nemotron-colembed-vl-4b-v2",

29 ]

31 EMBED_DIMS = {

32 "TomoroAI/tomoro-colqwen3-embed-4b": 320,

33 "OpenSearch-AI/Ops-Colqwen3-4B": 2560,

34 + "nvidia/nemotron-colembed-vl-4b-v2": 2560,

35 }

37 TEXT_QUERIES = [

@@ -33,6 +45,43 @@ TEXT_DOCUMENTS = [

45 ]

47 DTYPE = "half"

48 +GPU_MEMORY_UTILIZATION = 0.7

49 +

50 +

51 +def _make_base64_image(

52 + width: int = 64, height: int = 64, color: tuple[int, int, int] = (255, 0, 0)

53 +) -> str:

54 + """Create a small solid-color PNG image and return its base64 data URI."""

55 + img = Image.new("RGB", (width, height), color)

56 + buf = BytesIO()

57 + img.save(buf, format="PNG")

58 + b64 = base64.b64encode(buf.getvalue()).decode()

59 + return f"data:image/png;base64,{b64}"

60 +

61 +

62 +def _make_image_mm_param(

63 + image_uri: str,

64 + text: str | None = None,

65 +) -> ScoreMultiModalParam:

66 + """Build a ScoreMultiModalParam containing an image (and optional text)."""

67 + content: list = [

68 + ChatCompletionContentPartImageParam(

69 + type="image_url",

70 + image_url={"url": image_uri},

71 + ),

72 + ]

73 + if text is not None:

74 + content.append(

75 + ChatCompletionContentPartTextParam(type="text", text=text),

76 + )

77 + return ScoreMultiModalParam(content=content)

78 +

79 +

80 +def _make_text_mm_param(text: str) -> ScoreMultiModalParam:

81 + """Build a ScoreMultiModalParam containing only text."""

82 + return ScoreMultiModalParam(

83 + content=[ChatCompletionContentPartTextParam(type="text", text=text)],

84 + )

87 def _run_token_embed_test(

@@ -48,6 +97,7 @@ def _run_token_embed_test(

97 dtype=dtype,

98 max_model_len=4096,

99 enforce_eager=True,

100 + gpu_memory_utilization=GPU_MEMORY_UTILIZATION,

101 ) as vllm_model:

102 outputs = vllm_model.token_embed([TEXT_QUERIES[0]])

103

@@ -83,6 +133,7 @@ def _run_late_interaction_test(

133 dtype=dtype,

134 max_model_len=4096,

135 enforce_eager=True,

136 + gpu_memory_utilization=GPU_MEMORY_UTILIZATION,

137 ) as vllm_model:

138 q_outputs = vllm_model.token_embed([TEXT_QUERIES[0]])

139 d_outputs = vllm_model.token_embed([TEXT_DOCUMENTS[0]])

@@ -118,6 +169,7 @@ def _run_relevance_test(

169 dtype=dtype,

170 max_model_len=4096,

171 enforce_eager=True,

172 + gpu_memory_utilization=GPU_MEMORY_UTILIZATION,

173 ) as vllm_model:

174 scores = vllm_model.score(query, documents)

175

@@ -154,3 +206,142 @@ def test_colqwen3_relevance_ordering(

206 dtype: str,

207 ) -> None:

208 _run_relevance_test(vllm_runner, model, dtype=dtype)

209 +

210 +

211 +# ── Multimodal scoring tests ────────────────────────────────

212 +

213 +

214 +def _run_multimodal_text_query_image_docs_test(

215 + vllm_runner: type[VllmRunner],

216 + model: str,

217 + *,

218 + dtype: str,

219 +) -> None:

220 + """Score a text query against image documents via the multimodal path.

221 +

222 + Verifies that score_data_to_prompts correctly handles image content

223 + and produces valid MaxSim scores.

224 + """

225 + red_image = _make_base64_image(64, 64, color=(255, 0, 0))

226 + blue_image = _make_base64_image(64, 64, color=(0, 0, 255))

227 +

228 + query = "Describe the red object"

229 + image_docs = [

230 + _make_image_mm_param(red_image),

231 + _make_image_mm_param(blue_image),

232 + ]

233 +

234 + with vllm_runner(

235 + model,

236 + runner="pooling",

237 + dtype=dtype,

238 + max_model_len=4096,

239 + enforce_eager=True,

240 + gpu_memory_utilization=GPU_MEMORY_UTILIZATION,

241 + ) as vllm_model:

242 + scores = vllm_model.llm.score(query, image_docs)

243 +

244 + assert len(scores) == 2

245 + for s in scores:

246 + assert isinstance(s.outputs.score, float)

247 +

248 +

249 +def _run_multimodal_mixed_docs_test(

250 + vllm_runner: type[VllmRunner],

251 + model: str,

252 + *,

253 + dtype: str,

254 +) -> None:

255 + """Score a text query against a mix of text and image documents.

256 +

257 + Ensures the late-interaction path handles heterogeneous document

258 + types (plain strings alongside ScoreMultiModalParam images) in

259 + a single call.

260 + """

261 + red_image = _make_base64_image(64, 64, color=(255, 0, 0))

262 +

263 + query = "What is the capital of France?"

264 + documents: list = [

265 + "The capital of France is Paris.",

266 + _make_image_mm_param(red_image),

267 + ]

268 +

269 + with vllm_runner(

270 + model,

271 + runner="pooling",

272 + dtype=dtype,

273 + max_model_len=4096,

274 + enforce_eager=True,

275 + gpu_memory_utilization=GPU_MEMORY_UTILIZATION,

276 + ) as vllm_model:

277 + scores = vllm_model.llm.score(query, documents)

278 +

279 + assert len(scores) == 2

280 + for s in scores:

281 + assert isinstance(s.outputs.score, float)

282 + # Text document about France should score higher than a random image

283 + assert scores[0].outputs.score > scores[1].outputs.score

284 +

285 +

286 +def _run_multimodal_image_query_text_docs_test(

287 + vllm_runner: type[VllmRunner],

288 + model: str,

289 + *,

290 + dtype: str,

291 +) -> None:

292 + """Score an image query against text documents.

293 +

294 + Verifies the reverse direction: multimodal query with text-only

295 + documents through the late-interaction scoring path.

296 + """

297 + red_image = _make_base64_image(64, 64, color=(255, 0, 0))

298 + image_query = _make_image_mm_param(red_image, text="red color")

299 +

300 + documents = [

301 + "A bright red sports car.",

302 + "The weather forecast shows rain tomorrow.",

303 + ]

304 +

305 + with vllm_runner(

306 + model,

307 + runner="pooling",

308 + dtype=dtype,

309 + max_model_len=4096,

310 + enforce_eager=True,

311 + gpu_memory_utilization=GPU_MEMORY_UTILIZATION,

312 + ) as vllm_model:

313 + scores = vllm_model.llm.score(image_query, documents)

314 +

315 + assert len(scores) == 2

316 + for s in scores:

317 + assert isinstance(s.outputs.score, float)

318 +

319 +

320 +@pytest.mark.parametrize("model", MODELS)

321 +@pytest.mark.parametrize("dtype", [DTYPE])

322 +def test_colqwen3_multimodal_text_query_image_docs(

323 + vllm_runner,

324 + model: str,

325 + dtype: str,

326 +) -> None:

327 + _run_multimodal_text_query_image_docs_test(vllm_runner, model, dtype=dtype)

328 +

329 +

330 +@pytest.mark.parametrize("model", MODELS)

331 +@pytest.mark.parametrize("dtype", [DTYPE])

332 +def test_colqwen3_multimodal_mixed_docs(

333 + vllm_runner,

334 + model: str,

335 + dtype: str,

336 +) -> None:

337 + _run_multimodal_mixed_docs_test(vllm_runner, model, dtype=dtype)

338 +

339 +

340 +@pytest.mark.parametrize("model", MODELS)

341 +@pytest.mark.parametrize("dtype", [DTYPE])

342 +def test_colqwen3_multimodal_image_query_text_docs(

343 + vllm_runner,

344 + model: str,

345 + dtype: str,

346 +) -> None:

347 + _run_multimodal_image_query_text_docs_test(vllm_runner, model, dtype=dtype)

tests/models/registry.py

@@ -603,6 +603,9 @@ _EMBEDDING_EXAMPLE_MODELS = {

603 "OpsColQwen3Model": _HfExamplesInfo(

604 "OpenSearch-AI/Ops-Colqwen3-4B", trust_remote_code=True

605 ),

606 + "Qwen3VLNemotronEmbedModel": _HfExamplesInfo(

607 + "nvidia/nemotron-colembed-vl-4b-v2",

608 + ),

609 "SiglipModel": _HfExamplesInfo("google/siglip-base-patch16-224"),

610 "PrithviGeoSpatialMAE": _HfExamplesInfo(

611 "ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11",

vllm/entrypoints/llm.py

+9 -20

@@ -50,6 +50,7 @@ from vllm.entrypoints.pooling.score.utils import (

50 compress_token_type_ids,

51 compute_maxsim_score,

52 get_score_prompt,

53 + score_data_to_prompts,

54 validate_score_input,

55 )

56 from vllm.entrypoints.utils import log_non_default_args

@@ -1395,25 +1396,13 @@ class LLM:

1396

1397 tokenizer = self.get_tokenizer()

1398

1398 - # Extract text from ScoreData

1399 - text_1: list[str] = []

1400 - for text in data_1:

1401 - if not isinstance(text, str):

1402 - raise NotImplementedError(

1403 - "Late interaction scores currently do not support multimodal input."

1404 - )

1405 - text_1.append(text)

1406 -

1407 - text_2: list[str] = []

1408 - for text in data_2:

1409 - if not isinstance(text, str):

1410 - raise NotImplementedError(

1411 - "Late interaction scores currently do not support multimodal input."

1412 - )

1413 - text_2.append(text)

1399 + # Convert ScoreData to PromptType (handles both text and multimodal)

1400 + model_config = self.model_config

1401 + prompts_1 = score_data_to_prompts(data_1, "query", model_config)

1402 + prompts_2 = score_data_to_prompts(data_2, "document", model_config)

1403

1415 - encoded_output = self.encode(

1416 - text_1 + text_2,

1404 + encoded_output: list[PoolingRequestOutput] = self.encode(

1405 + prompts_1 + prompts_2,

1406 use_tqdm=use_tqdm,

1407 lora_request=lora_request,

1408 pooling_params=pooling_params,

@@ -1421,8 +1410,8 @@ class LLM:

1410 tokenization_kwargs=tokenization_kwargs,

1411 )

1412

1424 - encoded_output_1 = encoded_output[0 : len(text_1)]

1425 - encoded_output_2 = encoded_output[len(text_1) :]

1413 + encoded_output_1: list[PoolingRequestOutput] = encoded_output[: len(prompts_1)]

1414 + encoded_output_2: list[PoolingRequestOutput] = encoded_output[len(prompts_1) :]

1415

1416 if len(encoded_output_1) == 1:

1417 encoded_output_1 = encoded_output_1 * len(encoded_output_2)

vllm/entrypoints/pooling/score/serving.py

+60 -23

@@ -33,6 +33,7 @@ from vllm.entrypoints.pooling.score.utils import (

33 compress_token_type_ids,

34 compute_maxsim_score,

35 get_score_prompt,

36 + parse_score_data_single,

37 validate_score_input,

38 )

39 from vllm.inputs.data import ProcessorInputs, TokensPrompt, token_inputs

@@ -174,6 +175,43 @@ class ServingScores(OpenAIServing):

175

176 return final_res_batch

177

178 + def _preprocess_late_interaction_item(

179 + self,

180 + data: ScoreData,

181 + role: str,

182 + request: RerankRequest | ScoreRequest,

183 + tokenizer: TokenizerLike,

184 + tokenization_kwargs: dict[str, Any],

185 + ) -> tuple[str, TokensPrompt]:

186 + """Parse a single ScoreData into a text + optional multimodal

187 + TokensPrompt for late-interaction encoding.

188 +

189 + For plain strings, tokenises directly.

190 + For multimodal content parts, extracts text and multi_modal_data.

191 + """

192 + model_config = self.model_config

193 +

194 + if isinstance(data, str):

195 + text, mm_data, mm_uuids = data, None, None

196 + else:

197 + text, mm_data, mm_uuids = parse_score_data_single(data, role, model_config)

198 +

199 + prompt_inputs = tokenizer(text, **tokenization_kwargs)

200 + self._validate_input(request, prompt_inputs["input_ids"], text)

201 +

202 + engine_prompt = TokensPrompt(

203 + prompt_token_ids=prompt_inputs["input_ids"],

204 + )

205 +

206 + if mm_data is not None:

207 + engine_prompt["multi_modal_data"] = mm_data

208 + if mm_uuids is not None:

209 + engine_prompt["multi_modal_uuids"] = mm_uuids

210 + if request.mm_processor_kwargs is not None:

211 + engine_prompt["mm_processor_kwargs"] = request.mm_processor_kwargs

212 +

213 + return text, engine_prompt

214 +

215 async def _late_interaction_score(

216 self,

217 data_1: list[ScoreData],

@@ -189,37 +227,36 @@ class ServingScores(OpenAIServing):

227 Encodes queries and documents into per-token embeddings, then computes

228 MaxSim: sum over query tokens of max similarity to any document token.

229 """

192 - input_texts: list[str] = []

193 - for text in data_1 + data_2:

194 - if not isinstance(text, str):

195 - raise NotImplementedError(

196 - "Late interaction scores currently do not support multimodal input."

197 - )

198 - input_texts.append(text)

199 -

230 model_config = self.model_config

231 tokenizer = self.renderer.get_tokenizer()

232 + tokenization_kwargs = request.build_tok_params(model_config).get_encode_kwargs()

233

203 - encode_async = make_async(

204 - tokenizer.encode,

205 - executor=self._tokenizer_executor,

206 - )

234 + all_data = data_1 + data_2

235 + roles = ["query"] * len(data_1) + ["document"] * len(data_2)

236

208 - tokenization_kwargs = request.build_tok_params(model_config).get_encode_kwargs()

209 - tokenized_prompts = await asyncio.gather(

210 - *(encode_async(t, **tokenization_kwargs) for t in input_texts)

237 + preprocess_async = make_async(

238 + self._preprocess_late_interaction_item,

239 + executor=self._tokenizer_executor,

240 )

241

213 - engine_prompts: list[ProcessorInputs] = []

214 - for tok_result, input_text in zip(tokenized_prompts, input_texts):

215 - text_token_prompt = self._validate_input(request, tok_result, input_text)

216 -

217 - engine_prompts.append(

218 - token_inputs(

219 - text_token_prompt["prompt_token_ids"],

220 - prompt=input_text,

242 + preprocessed = await asyncio.gather(

243 + *(

244 + preprocess_async(

245 + data=d,

246 + role=r,

247 + request=request,

248 + tokenizer=tokenizer,

249 + tokenization_kwargs=tokenization_kwargs,

250 )

251 + for d, r in zip(all_data, roles)

252 )

253 + )

254 +

255 + input_texts: list[str] = []

256 + engine_prompts: list[TokensPrompt] = []

257 + for text, engine_prompt in preprocessed:

258 + input_texts.append(text)

259 + engine_prompts.append(engine_prompt)

260

261 # Schedule the request and get the result generator.

262 generators: list[AsyncGenerator[PoolingRequestOutput, None]] = []

vllm/entrypoints/pooling/score/utils.py

+71 -10

@@ -21,6 +21,7 @@ from vllm.entrypoints.chat_utils import (

21 _parse_chat_message_content_parts,

22 )

23 from vllm.inputs import TokensPrompt

24 +from vllm.inputs.data import PromptType, TextPrompt

25 from vllm.model_executor.models.interfaces import supports_score_template

26 from vllm.multimodal.inputs import MultiModalDataDict, MultiModalUUIDDict

27 from vllm.outputs import PoolingRequestOutput

@@ -153,31 +154,91 @@ def validate_score_input(

154 return score_input_1, score_input_2

155

156

157 +def _ensure_str(content: list[ConversationMessage]) -> str:

158 + """Extract a single string prompt from parsed conversation content."""

159 + assert len(content) == 1

160 + prompt = content[0]["content"]

161 + if prompt is not None and isinstance(prompt, str):

162 + return cast(str, prompt)

163 + raise ValueError(f"Only string content is supported, but got {content}.")

164 +

165 +

166 def parse_score_data(

167 data_1: ScoreData,

168 data_2: ScoreData,

169 model_config: ModelConfig,

170 ) -> tuple[str, str, MultiModalDataDict | None, MultiModalUUIDDict | None]:

171 + """Parse a query-document pair into text prompts and shared multi-modal

172 + data.

173 +

174 + Uses a **single** :class:`MultiModalItemTracker` so that multi-modal

175 + items from both inputs are merged into one ``mm_data`` dict. This is

176 + the correct behaviour for cross-encoder scoring, where query and

177 + document are concatenated into a single model prompt.

178 + """

179 mm_tracker = MultiModalItemTracker(model_config)

180

181 content_1 = _parse_score_content("query", data_1, mm_tracker)

182 content_2 = _parse_score_content("document", data_2, mm_tracker)

183

166 - def ensure_str(content: list[ConversationMessage]) -> str:

167 - assert len(content) == 1

168 - prompt = content[0]["content"]

169 - if prompt is not None and isinstance(prompt, str):

170 - return cast(str, prompt)

171 - else:

172 - raise ValueError(f"Only string content is supported, but got {content}.")

173 -

174 - prompt_1 = ensure_str(content_1)

175 - prompt_2 = ensure_str(content_2)

184 + prompt_1 = _ensure_str(content_1)

185 + prompt_2 = _ensure_str(content_2)

186 mm_items, mm_uuids = mm_tracker.resolve_items()

187

188 return prompt_1, prompt_2, mm_items, mm_uuids

189

190

191 +def parse_score_data_single(

192 + data: ScoreData,

193 + role: str,

194 + model_config: ModelConfig,

195 +) -> tuple[str, MultiModalDataDict | None, MultiModalUUIDDict | None]:

196 + """Parse **one** ScoreData into a text prompt and its own multi-modal

197 + data.

198 +

199 + Unlike :func:`parse_score_data`, each call creates an **independent**

200 + :class:`MultiModalItemTracker` so multi-modal items are kept separate.

201 + This is the correct behaviour for late-interaction scoring, where

202 + query and document are encoded independently.

203 + """

204 + mm_tracker = MultiModalItemTracker(model_config)

205 + content = _parse_score_content(role, data, mm_tracker)

206 +

207 + prompt = _ensure_str(content)

208 + mm_items, mm_uuids = mm_tracker.resolve_items()

209 + return prompt, mm_items, mm_uuids

210 +

211 +

212 +def score_data_to_prompts(

213 + data_list: list[ScoreData],

214 + role: str,

215 + model_config: ModelConfig,

216 +) -> list[PromptType]:

217 + """Convert a list of ScoreData into PromptType objects.

218 +

219 + For plain text inputs, returns the string directly.

220 + For multimodal inputs (list of content parts), parses them into

221 + a :class:`TextPrompt` with attached ``multi_modal_data`` /

222 + ``multi_modal_uuids``.

223 +

224 + This is used by late-interaction scoring where each query/document

225 + is encoded independently.

226 + """

227 + prompts: list[PromptType] = []

228 + for data in data_list:

229 + if isinstance(data, str):

230 + prompts.append(data)

231 + else:

232 + text, mm_data, mm_uuids = parse_score_data_single(data, role, model_config)

233 + prompt: TextPrompt = TextPrompt(prompt=text)

234 + if mm_data is not None:

235 + prompt["multi_modal_data"] = mm_data

236 + if mm_uuids is not None:

237 + prompt["multi_modal_uuids"] = mm_uuids

238 + prompts.append(prompt)

239 + return prompts

240 +

241 +

242 def _parse_score_content(

243 role: str,

244 data: ScoreData,

vllm/model_executor/models/colqwen3.py

+8 -6

@@ -16,6 +16,7 @@ Based on: Qwen3-VL backbone with custom text projection

16 Target models:

17 - TomoroAI/tomoro-colqwen3-embed-8b

18 - OpenSearch-AI/Ops-Colqwen3-4B

19 +- nvidia/nemotron-colembed-vl-4b-v2

20 """

22 from collections.abc import Iterable, Mapping

@@ -229,13 +230,14 @@ class ColQwen3Model(

230 if not isinstance(hidden_states, torch.Tensor):

231 return hidden_states # type: ignore

232

232 - proj_dtype = self.custom_text_proj.weight.dtype # type: ignore

233 - if hidden_states.dtype != proj_dtype:

234 - hidden_states = hidden_states.to(proj_dtype)

233 + if self.custom_text_proj is not None:

234 + proj_dtype = self.custom_text_proj.weight.dtype

235 + if hidden_states.dtype != proj_dtype:

236 + hidden_states = hidden_states.to(proj_dtype)

237 + hidden_states = self.custom_text_proj(hidden_states)

238

236 - # Project to embedding dimension and L2 normalize

237 - proj = self.custom_text_proj(hidden_states) # type: ignore

238 - return torch.nn.functional.normalize(proj, p=2, dim=-1)

239 + # L2 normalize

240 + return torch.nn.functional.normalize(hidden_states, p=2, dim=-1)

241

242 # Names used for the projection layer across different ColQwen3 variants

243 _PROJ_LAYER_NAMES = {

vllm/model_executor/models/registry.py

@@ -256,6 +256,7 @@ _EMBEDDING_MODELS = {

256 "Qwen2VLForConditionalGeneration": ("qwen2_vl", "Qwen2VLForConditionalGeneration"), # noqa: E501

257 "ColQwen3": ("colqwen3", "ColQwen3Model"),

258 "OpsColQwen3Model": ("colqwen3", "ColQwen3Model"),

259 + "Qwen3VLNemotronEmbedModel": ("colqwen3", "ColQwen3Model"),

260 "SiglipModel": ("siglip", "SiglipEmbeddingModel"),

261 # Technically Terratorch models work on images, both in

262 # input and output. I am adding it here because it piggy-backs on embedding

vllm/transformers_utils/config.py

@@ -76,6 +76,7 @@ _CONFIG_REGISTRY: dict[str, type[PretrainedConfig]] = LazyConfigDict(

76 chatglm="ChatGLMConfig",

77 colqwen3="ColQwen3Config",

78 ops_colqwen3="OpsColQwen3Config",

79 + qwen3_vl_nemotron_embed="Qwen3VLNemotronEmbedConfig",

80 deepseek_vl_v2="DeepseekVLV2Config",

81 deepseek_v32="DeepseekV3Config",

82 flex_olmo="FlexOlmoConfig",