A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 04:45 UTC
[examples] Resettle pooling examples. (#29365)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
wang.yuqi committed
Dec 2, 2025 at 23:54 UTC
2eb4fe912916aea8998d085786df7abd7737e1f3
36 files changed
+111
-251
.buildkite/test-pipeline.yaml
+12
-8
@@ -390,20 +390,24 @@ steps:
390
- examples/
391
commands:
392
- pip install tensorizer # for tensorizer test
393
+ # for basic
394
+ - python3 offline_inference/basic/chat.py
395
- python3 offline_inference/basic/generate.py --model facebook/opt-125m
396
- python3 offline_inference/basic/generate.py --model meta-llama/Llama-2-13b-chat-hf --cpu-offload-gb 10
395
- - python3 offline_inference/basic/chat.py
396
- - python3 offline_inference/prefix_caching.py
397
- - python3 offline_inference/llm_engine_example.py
397
+ - python3 offline_inference/basic/classify.py
398
+ - python3 offline_inference/basic/embed.py
399
+ - python3 offline_inference/basic/score.py
400
+ # for multi-modal models
401
- python3 offline_inference/audio_language.py --seed 0
402
- python3 offline_inference/vision_language.py --seed 0
400
- - python3 offline_inference/vision_language_pooling.py --seed 0
403
- python3 offline_inference/vision_language_multi_image.py --seed 0
402
- - python3 others/tensorize_vllm_model.py --model facebook/opt-125m serialize --serialized-directory /tmp/ --suffix v1 && python3 others/tensorize_vllm_model.py --model facebook/opt-125m deserialize --path-to-tensors /tmp/vllm/facebook/opt-125m/v1/model.tensors
404
- python3 offline_inference/encoder_decoder_multimodal.py --model-type whisper --seed 0
404
- - python3 offline_inference/basic/classify.py
405
- - python3 offline_inference/basic/embed.py
406
- - python3 offline_inference/basic/score.py
405
+ # for pooling models
406
+ - python3 pooling/pooling/vision_language_pooling.py --seed 0
407
+ # for features demo
408
+ - python3 offline_inference/prefix_caching.py
409
+ - python3 offline_inference/llm_engine_example.py
410
+ - python3 others/tensorize_vllm_model.py --model facebook/opt-125m serialize --serialized-directory /tmp/ --suffix v1 && python3 others/tensorize_vllm_model.py --model facebook/opt-125m deserialize --path-to-tensors /tmp/vllm/facebook/opt-125m/v1/model.tensors
411
- python3 offline_inference/spec_decode.py --test --method eagle --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill --max-model-len 2048
412
# https://github.com/vllm-project/vllm/pull/26682 uses slightly more memory in PyTorch 2.9+ causing this test to OOM in 1xL4 GPU
413
- python3 offline_inference/spec_decode.py --test --method eagle3 --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill --max-model-len 1536
.github/CODEOWNERS
+2
-2
@@ -146,10 +146,10 @@ mkdocs.yaml @hmellor
146
/requirements/kv_connectors.txt @NickLucche
147
148
# Pooling models
149
-/examples/*/pooling/ @noooop
149
+/examples/pooling @noooop
150
/tests/models/*/pooling* @noooop
151
/tests/entrypoints/pooling @noooop
152
-/vllm/entrypoints/pooling @aarnphm @chaunceyjiang @noooop
152
+/vllm/entrypoints/pooling @noooop
153
/vllm/config/pooler.py @noooop
154
/vllm/pooling_params.py @noooop
155
/vllm/model_executor/layers/pooler.py @noooop
docs/.nav.yml
+1
-5
@@ -5,11 +5,7 @@ nav:
5
- Getting Started:
6
- getting_started/quickstart.md
7
- getting_started/installation
8
- - Examples:
9
- - examples/README.md
10
- - Offline Inference: examples/offline_inference
11
- - Online Serving: examples/online_serving
12
- - Others: examples/others
8
+ - Examples: examples
9
- General:
10
- usage/v1_guide.md
11
- usage/*
docs/design/io_processor_plugins.md
+1
-1
@@ -79,7 +79,7 @@ The `post_process*` methods take `PoolingRequestOutput` objects as input and gen
79
The `validate_or_generate_params` method is used for validating with the plugin any `SamplingParameters`/`PoolingParameters` received with the user request, or to generate new ones if none are specified. The function always returns the validated/generated parameters.
80
The `output_to_response` method is used only for online serving and converts the plugin output to the `IOProcessorResponse` type that is then returned by the API Server. The implementation of the `/pooling` serving endpoint is available here [vllm/entrypoints/openai/serving_pooling.py](../../vllm/entrypoints/pooling/pooling/serving.py).
81
82
-An example implementation of a plugin that enables generating geotiff images with the PrithviGeospatialMAE model is available [here](https://github.com/IBM/terratorch/tree/main/terratorch/vllm/plugins/segmentation). Please, also refer to our online ([examples/online_serving/pooling/prithvi_geospatial_mae.py](../../examples/online_serving/pooling/prithvi_geospatial_mae.py)) and offline ([examples/offline_inference/pooling/prithvi_geospatial_mae_io_processor.py](../../examples/offline_inference/pooling/prithvi_geospatial_mae_io_processor.py)) inference examples.
82
+An example implementation of a plugin that enables generating geotiff images with the PrithviGeospatialMAE model is available [here](https://github.com/IBM/terratorch/tree/main/terratorch/vllm/plugins/segmentation). Please, also refer to our online ([examples/pooling/plugin/prithvi_geospatial_mae_client.py](../../examples/pooling/plugin/prithvi_geospatial_mae_client.py)) and offline ([examples/pooling/plugin/prithvi_geospatial_mae_io_processor.py](../../examples/pooling/plugin/prithvi_geospatial_mae_io_processor.py)) inference examples.
83
84
## Using an IO Processor plugin
85
docs/mkdocs/hooks/generate_examples.py
+84
-70
@@ -2,7 +2,8 @@
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
import itertools
4
import logging
5
-from dataclasses import dataclass, field
5
+from dataclasses import dataclass
6
+from functools import cached_property
7
from pathlib import Path
8
from typing import Literal
9
@@ -16,13 +17,18 @@ EXAMPLE_DIR = ROOT_DIR / "examples"
17
EXAMPLE_DOC_DIR = ROOT_DIR / "docs/examples"
18
19
19
-def fix_case(text: str) -> str:
20
+def title(text: str) -> str:
21
+ # Default title case
22
+ text = text.replace("_", " ").replace("/", " - ").title()
23
+ # Custom substitutions
24
subs = {
25
+ "io": "IO",
26
"api": "API",
27
"cli": "CLI",
28
"cpu": "CPU",
29
"llm": "LLM",
30
"mae": "MAE",
31
+ "ner": "NER",
32
"tpu": "TPU",
33
"gguf": "GGUF",
34
"lora": "LoRA",
@@ -48,71 +54,65 @@ class Example:
54
Attributes:
55
path (Path): The path to the main directory or file.
56
category (str): The category of the document.
51
- main_file (Path): The main file in the directory.
52
- other_files (list[Path]): list of other files in the directory.
53
- title (str): The title of the document.
57
+
58
+ Properties::
59
+ main_file() -> Path | None: Determines the main file in the given path.
60
+ other_files() -> list[Path]: Determines other files in the directory excluding
61
+ the main file.
62
+ title() -> str: Determines the title of the document.
63
64
Methods:
56
- __post_init__(): Initializes the main_file, other_files, and title attributes.
57
- determine_main_file() -> Path: Determines the main file in the given path.
58
- determine_other_files() -> list[Path]: Determines other files in the directory excluding the main file.
59
- determine_title() -> str: Determines the title of the document.
65
generate() -> str: Generates the documentation content.
61
- """ # noqa: E501
66
+ """
67
68
path: Path
64
- category: str = None
65
- main_file: Path = field(init=False)
66
- other_files: list[Path] = field(init=False)
67
- title: str = field(init=False)
68
-
69
- def __post_init__(self):
70
- self.main_file = self.determine_main_file()
71
- self.other_files = self.determine_other_files()
72
- self.title = self.determine_title()
73
-
74
- @property
75
- def is_code(self) -> bool:
76
- return self.main_file.suffix != ".md"
77
-
78
- def determine_main_file(self) -> Path:
79
- """
80
- Determines the main file in the given path.
81
- If the path is a file, it returns the path itself. Otherwise, it searches
82
- for Markdown files (*.md) in the directory and returns the first one found.
83
- Returns:
84
- Path: The main file path, either the original path if it's a file or the first
85
- Markdown file found in the directory.
86
- Raises:
87
- IndexError: If no Markdown files are found in the directory.
88
- """ # noqa: E501
89
- return self.path if self.path.is_file() else list(self.path.glob("*.md")).pop()
90
-
91
- def determine_other_files(self) -> list[Path]:
92
- """
93
- Determine other files in the directory excluding the main file.
69
+ category: str
70
95
- This method checks if the given path is a file. If it is, it returns an empty list.
96
- Otherwise, it recursively searches through the directory and returns a list of all
97
- files that are not the main file.
71
+ @cached_property
72
+ def main_file(self) -> Path | None:
73
+ """Determines the main file in the given path.
74
99
- Returns:
100
- list[Path]: A list of Path objects representing the other files in the directory.
101
- """ # noqa: E501
75
+ If path is a file, it returns the path itself. If path is a directory, it
76
+ searches for Markdown files (*.md) in the directory and returns the first one
77
+ found. If no Markdown files are found, it returns None."""
78
+ # Single file example
79
+ if self.path.is_file():
80
+ return self.path
81
+ # Multi file example with a README
82
+ if md_paths := list(self.path.glob("*.md")):
83
+ return md_paths[0]
84
+ # Multi file example without a README
85
+ return None
86
+
87
+ @cached_property
88
+ def other_files(self) -> list[Path]:
89
+ """Determine other files in the directory excluding the main file.
90
+
91
+ If path is a file, it returns an empty list. Otherwise, it returns every file
92
+ in the directory except the main file in a list."""
93
+ # Single file example
94
if self.path.is_file():
95
return []
96
+ # Multi file example
97
is_other_file = lambda file: file.is_file() and file != self.main_file
105
- return [file for file in self.path.rglob("*") if is_other_file(file)]
106
-
107
- def determine_title(self) -> str:
108
- if not self.is_code:
109
- # Specify encoding for building on Windows
110
- with open(self.main_file, encoding="utf-8") as f:
111
- first_line = f.readline().strip()
112
- match = re.match(r"^#\s+(?P<title>.+)$", first_line)
113
- if match:
114
- return match.group("title")
115
- return fix_case(self.path.stem.replace("_", " ").title())
98
+ return sorted(file for file in self.path.rglob("*") if is_other_file(file))
99
+
100
+ @cached_property
101
+ def is_code(self) -> bool:
102
+ return self.main_file is not None and self.main_file.suffix != ".md"
103
+
104
+ @cached_property
105
+ def title(self) -> str:
106
+ # Generate title from filename if no main md file found
107
+ if self.main_file is None or self.is_code:
108
+ return title(self.path.stem)
109
+ # Specify encoding for building on Windows
110
+ with open(self.main_file, encoding="utf-8") as f:
111
+ first_line = f.readline().strip()
112
+ match = re.match(r"^#\s+(?P<title>.+)$", first_line)
113
+ if match:
114
+ return match.group("title")
115
+ raise ValueError(f"Title not found in {self.main_file}")
116
117
def fix_relative_links(self, content: str) -> str:
118
"""
@@ -156,24 +156,35 @@ class Example:
156
# included files containing code fences too
157
code_fence = "``````"
158
159
- if self.is_code:
160
- content += (
161
- f"{code_fence}{self.main_file.suffix[1:]}\n"
162
- f'--8<-- "{self.main_file}"\n'
163
- f"{code_fence}\n"
164
- )
159
+ if self.main_file is not None:
160
+ # Single file example or multi file example with a README
161
+ if self.is_code:
162
+ content += (
163
+ f"{code_fence}{self.main_file.suffix[1:]}\n"
164
+ f'--8<-- "{self.main_file}"\n'
165
+ f"{code_fence}\n"
166
+ )
167
+ else:
168
+ with open(self.main_file, encoding="utf-8") as f:
169
+ # Skip the title from md snippets as it's been included above
170
+ main_content = f.readlines()[1:]
171
+ content += self.fix_relative_links("".join(main_content))
172
+ content += "\n"
173
else:
166
- with open(self.main_file) as f:
167
- # Skip the title from md snippets as it's been included above
168
- main_content = f.readlines()[1:]
169
- content += self.fix_relative_links("".join(main_content))
170
- content += "\n"
174
+ # Multi file example without a README
175
+ for file in self.other_files:
176
+ file_title = title(str(file.relative_to(self.path).with_suffix("")))
177
+ content += f"## {file_title}\n\n"
178
+ content += (
179
+ f'{code_fence}{file.suffix[1:]}\n--8<-- "{file}"\n{code_fence}\n\n'
180
+ )
181
+ return content
182
183
if not self.other_files:
184
return content
185
186
content += "## Example materials\n\n"
176
- for file in sorted(self.other_files):
187
+ for file in self.other_files:
188
content += f'??? abstract "{file.relative_to(self.path)}"\n'
189
if file.suffix != ".md":
190
content += f" {code_fence}{file.suffix[1:]}\n"
@@ -200,11 +211,13 @@ def on_startup(command: Literal["build", "gh-deploy", "serve"], dirty: bool):
211
glob_patterns = ["*.py", "*.md", "*.sh"]
212
# Find categorised examples
213
for category in categories:
214
+ logger.info("Processing category: %s", category.stem)
215
globs = [category.glob(pattern) for pattern in glob_patterns]
216
for path in itertools.chain(*globs):
217
examples.append(Example(path, category.stem))
218
# Find examples in subdirectories
207
- for path in category.glob("*/*.md"):
219
+ globs = [category.glob(f"*/{pattern}") for pattern in glob_patterns]
220
+ for path in itertools.chain(*globs):
221
examples.append(Example(path.parent, category.stem))
222
223
# Generate the example documentation
@@ -217,3 +230,4 @@ def on_startup(command: Literal["build", "gh-deploy", "serve"], dirty: bool):
230
with open(doc_path, "w+", encoding="utf-8") as f:
231
f.write(example.generate())
232
logger.debug("Example generated: %s", doc_path.relative_to(ROOT_DIR))
233
+ logger.info("Total examples generated: %d", len(examples))
docs/models/pooling_models.md
+2
-2
@@ -274,7 +274,7 @@ outputs = llm.embed(
274
print(outputs[0].outputs)
275
```
276
277
-A code example can be found here: [examples/offline_inference/pooling/embed_matryoshka_fy.py](../../examples/offline_inference/pooling/embed_matryoshka_fy.py)
277
+A code example can be found here: [examples/pooling/embed/embed_matryoshka_fy.py](../../examples/pooling/embed/embed_matryoshka_fy.py)
278
279
### Online Inference
280
@@ -304,7 +304,7 @@ Expected output:
304
{"id":"embd-5c21fc9a5c9d4384a1b021daccaf9f64","object":"list","created":1745476417,"model":"jinaai/jina-embeddings-v3","data":[{"index":0,"object":"embedding","embedding":[-0.3828125,-0.1357421875,0.03759765625,0.125,0.21875,0.09521484375,-0.003662109375,0.1591796875,-0.130859375,-0.0869140625,-0.1982421875,0.1689453125,-0.220703125,0.1728515625,-0.2275390625,-0.0712890625,-0.162109375,-0.283203125,-0.055419921875,-0.0693359375,0.031982421875,-0.04052734375,-0.2734375,0.1826171875,-0.091796875,0.220703125,0.37890625,-0.0888671875,-0.12890625,-0.021484375,-0.0091552734375,0.23046875]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0,"prompt_tokens_details":null}}
305
```
306
307
-An OpenAI client example can be found here: [examples/online_serving/pooling/openai_embedding_matryoshka_fy.py](../../examples/online_serving/pooling/openai_embedding_matryoshka_fy.py)
307
+An OpenAI client example can be found here: [examples/pooling/embed/openai_embedding_matryoshka_fy.py](../../examples/pooling/embed/openai_embedding_matryoshka_fy.py)
308
309
## Deprecated Features
310
docs/models/supported_models.md
+2
-2
@@ -568,7 +568,7 @@ These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) A
568
```
569
570
!!! note
571
- Load the official original `Qwen3 Reranker` by using the following command. More information can be found at: [examples/offline_inference/pooling/qwen3_reranker.py](../../examples/offline_inference/pooling/qwen3_reranker.py).
571
+ Load the official original `Qwen3 Reranker` by using the following command. More information can be found at: [examples/pooling/score/qwen3_reranker.py](../../examples/pooling/score/qwen3_reranker.py).
572
573
```bash
574
vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
@@ -606,7 +606,7 @@ These models primarily support the [`LLM.encode`](./pooling_models.md#llmencode)
606
| `ModernBertForTokenClassification` | ModernBERT-based | `disham993/electrical-ner-ModernBERT-base` | | |
607
608
!!! note
609
- Named Entity Recognition (NER) usage, please refer to [examples/offline_inference/pooling/ner.py](../../examples/offline_inference/pooling/ner.py), [examples/online_serving/pooling/ner_client.py](../../examples/online_serving/pooling/ner_client.py).
609
+ Named Entity Recognition (NER) usage, please refer to [examples/pooling/token_classify/ner.py](../../examples/pooling/token_classify/ner.py), [examples/pooling/token_classify/ner_client.py](../../examples/pooling/token_classify/ner_client.py).
610
611
## List of Multimodal Language Models
612
docs/serving/openai_compatible_server.md
+7
-7
@@ -234,7 +234,7 @@ The following extra parameters are supported:
234
Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
235
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
236
237
-Code example: [examples/online_serving/pooling/openai_embedding_client.py](../../examples/online_serving/pooling/openai_embedding_client.py)
237
+Code example: [examples/pooling/embed/openai_embedding_client.py](../../examples/pooling/embed/openai_embedding_client.py)
238
239
If the model has a [chat template](../serving/openai_compatible_server.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))
240
which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations:
@@ -335,7 +335,7 @@ and passing a list of `messages` in the request. Refer to the examples below for
335
`MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code
336
example below for details.
337
338
-Full example: [examples/online_serving/pooling/openai_chat_embedding_client_for_multimodal.py](../../examples/online_serving/pooling/openai_chat_embedding_client_for_multimodal.py)
338
+Full example: [examples/pooling/embed/openai_chat_embedding_client_for_multimodal.py](../../examples/pooling/embed/openai_chat_embedding_client_for_multimodal.py)
339
340
#### Extra parameters
341
@@ -516,7 +516,7 @@ Our Pooling API encodes input prompts using a [pooling model](../models/pooling_
516
517
The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
518
519
-Code example: [examples/online_serving/pooling/openai_pooling_client.py](../../examples/online_serving/pooling/openai_pooling_client.py)
519
+Code example: [examples/pooling/pooling/openai_pooling_client.py](../../examples/pooling/pooling/openai_pooling_client.py)
520
521
### Classification API
522
@@ -524,7 +524,7 @@ Our Classification API directly supports Hugging Face sequence-classification mo
524
525
We automatically wrap any other transformer via `as_seq_cls_model()`, which pools on the last token, attaches a `RowParallelLinear` head, and applies a softmax to produce per-class probabilities.
526
527
-Code example: [examples/online_serving/pooling/openai_classification_client.py](../../examples/online_serving/pooling/openai_classification_client.py)
527
+Code example: [examples/pooling/classify/openai_classification_client.py](../../examples/pooling/classify/openai_classification_client.py)
528
529
#### Example Requests
530
@@ -640,7 +640,7 @@ Usually, the score for a sentence pair refers to the similarity between two sent
640
641
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
642
643
-Code example: [examples/online_serving/pooling/openai_cross_encoder_score.py](../../examples/online_serving/pooling/openai_cross_encoder_score.py)
643
+Code example: [examples/pooling/score/openai_cross_encoder_score.py](../../examples/pooling/score/openai_cross_encoder_score.py)
644
645
#### Single inference
646
@@ -821,7 +821,7 @@ You can pass multi-modal inputs to scoring models by passing `content` including
821
print("Scoring output:", response_json["data"][0]["score"])
822
print("Scoring output:", response_json["data"][1]["score"])
823
```
824
-Full example: [examples/online_serving/pooling/openai_cross_encoder_score_for_multimodal.py](../../examples/online_serving/pooling/openai_cross_encoder_score_for_multimodal.py)
824
+Full example: [examples/pooling/score/openai_cross_encoder_score_for_multimodal.py](../../examples/pooling/score/openai_cross_encoder_score_for_multimodal.py)
825
826
#### Extra parameters
827
@@ -851,7 +851,7 @@ endpoints are compatible with both [Jina AI's re-rank API interface](https://jin
851
[Cohere's re-rank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with
852
popular open-source tools.
853
854
-Code example: [examples/online_serving/pooling/jinaai_rerank_client.py](../../examples/online_serving/pooling/jinaai_rerank_client.py)
854
+Code example: [examples/pooling/score/jinaai_rerank_client.py](../../examples/pooling/score/jinaai_rerank_client.py)
855
856
#### Example Request
857
examples/offline_inference/pooling/README.md
deleted
-57
@@ -1,57 +0,0 @@
1
-# Pooling models
2
-
3
-## Convert llm model to seq cls
4
-
5
-```bash
6
-# for BAAI/bge-reranker-v2-gemma
7
-# Caution: "Yes" and "yes" are two different tokens
8
-python examples/offline_inference/pooling/convert_model_to_seq_cls.py --model_name BAAI/bge-reranker-v2-gemma --classifier_from_tokens '["Yes"]' --method no_post_processing --path ./bge-reranker-v2-gemma-seq-cls
9
-# for mxbai-rerank-v2
10
-python examples/offline_inference/pooling/convert_model_to_seq_cls.py --model_name mixedbread-ai/mxbai-rerank-base-v2 --classifier_from_tokens '["0", "1"]' --method from_2_way_softmax --path ./mxbai-rerank-base-v2-seq-cls
11
-# for Qwen3-Reranker
12
-python examples/offline_inference/pooling/convert_model_to_seq_cls.py --model_name Qwen/Qwen3-Reranker-0.6B --classifier_from_tokens '["no", "yes"]' --method from_2_way_softmax --path ./Qwen3-Reranker-0.6B-seq-cls
13
-```
14
-
15
-## Embed jina_embeddings_v3 usage
16
-
17
-Only text matching task is supported for now. See <https://github.com/vllm-project/vllm/pull/16120>
18
-
19
-```bash
20
-python examples/offline_inference/pooling/embed_jina_embeddings_v3.py
21
-```
22
-
23
-## Embed matryoshka dimensions usage
24
-
25
-```bash
26
-python examples/offline_inference/pooling/embed_matryoshka_fy.py
27
-```
28
-
29
-## Multi vector retrieval usage
30
-
31
-```bash
32
-python examples/offline_inference/pooling/multi_vector_retrieval.py
33
-```
34
-
35
-## Named Entity Recognition (NER) usage
36
-
37
-```bash
38
-python examples/offline_inference/pooling/ner.py
39
-```
40
-
41
-## Prithvi Geospatial MAE usage
42
-
43
-```bash
44
-python examples/offline_inference/pooling/prithvi_geospatial_mae.py
45
-```
46
-
47
-## IO Processor Plugins for Prithvi Geospatial MAE
48
-
49
-```bash
50
-python examples/offline_inference/pooling/prithvi_geospatial_mae_io_processor.py
51
-```
52
-
53
-## Qwen3 reranker usage
54
-
55
-```bash
56
-python examples/offline_inference/pooling/qwen3_reranker.py
57
-```
examples/online_serving/pooling/README.md
deleted
-97
@@ -1,97 +0,0 @@
1
-# Pooling models
2
-
3
-## Cohere rerank usage
4
-
5
-```bash
6
-# vllm serve BAAI/bge-reranker-base
7
-python examples/online_serving/pooling/cohere_rerank_client.py
8
-```
9
-
10
-## Embedding requests base64 encoding_format usage
11
-
12
-```bash
13
-# vllm serve intfloat/e5-small
14
-python examples/online_serving/pooling/embedding_requests_base64_client.py
15
-```
16
-
17
-## Embedding requests bytes encoding_format usage
18
-
19
-```bash
20
-# vllm serve intfloat/e5-small
21
-python examples/online_serving/pooling/embedding_requests_bytes_client.py
22
-```
23
-
24
-## Jinaai rerank usage
25
-
26
-```bash
27
-# vllm serve BAAI/bge-reranker-base
28
-python examples/online_serving/pooling/jinaai_rerank_client.py
29
-```
30
-
31
-## Multi vector retrieval usage
32
-
33
-```bash
34
-# vllm serve BAAI/bge-m3
35
-python examples/online_serving/pooling/multi_vector_retrieval_client.py
36
-```
37
-
38
-## Named Entity Recognition (NER) usage
39
-
40
-```bash
41
-# vllm serve boltuix/NeuroBERT-NER
42
-python examples/online_serving/pooling/ner_client.py
43
-```
44
-
45
-## OpenAI chat embedding for multimodal usage
46
-
47
-```bash
48
-python examples/online_serving/pooling/openai_chat_embedding_client_for_multimodal.py
49
-```
50
-
51
-## OpenAI classification usage
52
-
53
-```bash
54
-# vllm serve jason9693/Qwen2.5-1.5B-apeach
55
-python examples/online_serving/pooling/openai_classification_client.py
56
-```
57
-
58
-## OpenAI cross_encoder score usage
59
-
60
-```bash
61
-# vllm serve BAAI/bge-reranker-v2-m3
62
-python examples/online_serving/pooling/openai_cross_encoder_score.py
63
-```
64
-
65
-## OpenAI cross_encoder score for multimodal usage
66
-
67
-```bash
68
-# vllm serve jinaai/jina-reranker-m0
69
-python examples/online_serving/pooling/openai_cross_encoder_score_for_multimodal.py
70
-```
71
-
72
-## OpenAI embedding usage
73
-
74
-```bash
75
-# vllm serve intfloat/e5-small
76
-python examples/online_serving/pooling/openai_embedding_client.py
77
-```
78
-
79
-## OpenAI embedding matryoshka dimensions usage
80
-
81
-```bash
82
-# vllm serve jinaai/jina-embeddings-v3 --trust-remote-code
83
-python examples/online_serving/pooling/openai_embedding_matryoshka_fy.py
84
-```
85
-
86
-## OpenAI pooling usage
87
-
88
-```bash
89
-# vllm serve internlm/internlm2-1_8b-reward --trust-remote-code
90
-python examples/online_serving/pooling/openai_pooling_client.py
91
-```
92
-
93
-## Online Prithvi Geospatial MAE usage
94
-
95
-```bash
96
-python examples/online_serving/pooling/prithvi_geospatial_mae.py
97
-```