A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] Fix pooling-params doc references in openai_compatible_server.md (#24939)
Kay Yan committed
Sep 18, 2025 at 19:36 UTC
eaffe4486cb1d7edf884e6e254cab33fc397e308
3 files changed
+26
-15
docs/api/README.md
-1
@@ -46,7 +46,6 @@ Engine classes for offline and online inference.
46
Inference parameters for vLLM APIs.
47
48
[](){ #sampling-params }
49
-[](){ #pooling-params }
49
50
- [vllm.SamplingParams][]
51
- [vllm.PoolingParams][]
docs/serving/openai_compatible_server.md
+12
-8
@@ -317,10 +317,11 @@ Full example: <gh-file:examples/online_serving/pooling/openai_chat_embedding_cli
317
318
#### Extra parameters
319
320
-The following [pooling parameters][pooling-params] are supported.
320
+The following [pooling parameters][vllm.PoolingParams] are supported.
321
322
```python
323
---8<-- "vllm/entrypoints/openai/protocol.py:embedding-pooling-params"
323
+--8<-- "vllm/pooling_params.py:common-pooling-params"
324
+--8<-- "vllm/pooling_params.py:embedding-pooling-params"
325
```
326
327
The following extra parameters are supported by default:
@@ -527,10 +528,11 @@ curl -v "http://127.0.0.1:8000/classify" \
528
529
#### Extra parameters
530
530
-The following [pooling parameters][pooling-params] are supported.
531
+The following [pooling parameters][vllm.PoolingParams] are supported.
532
533
```python
533
---8<-- "vllm/entrypoints/openai/protocol.py:classification-pooling-params"
534
+--8<-- "vllm/pooling_params.py:common-pooling-params"
535
+--8<-- "vllm/pooling_params.py:classification-pooling-params"
536
```
537
538
The following extra parameters are supported:
@@ -733,10 +735,11 @@ Full example: <gh-file:examples/online_serving/openai_cross_encoder_score_for_mu
735
736
#### Extra parameters
737
736
-The following [pooling parameters][pooling-params] are supported.
738
+The following [pooling parameters][vllm.PoolingParams] are supported.
739
740
```python
739
---8<-- "vllm/entrypoints/openai/protocol.py:score-pooling-params"
741
+--8<-- "vllm/pooling_params.py:common-pooling-params"
742
+--8<-- "vllm/pooling_params.py:classification-pooling-params"
743
```
744
745
The following extra parameters are supported:
@@ -815,10 +818,11 @@ Result documents will be sorted by relevance, and the `index` property can be us
818
819
#### Extra parameters
820
818
-The following [pooling parameters][pooling-params] are supported.
821
+The following [pooling parameters][vllm.PoolingParams] are supported.
822
823
```python
821
---8<-- "vllm/entrypoints/openai/protocol.py:rerank-pooling-params"
824
+--8<-- "vllm/pooling_params.py:common-pooling-params"
825
+--8<-- "vllm/pooling_params.py:classification-pooling-params"
826
```
827
828
The following extra parameters are supported:
vllm/pooling_params.py
+14
-6
@@ -20,25 +20,33 @@ class PoolingParams(
20
"""API parameters for pooling models.
21
22
Attributes:
23
+ truncate_prompt_tokens: Controls prompt truncation.
24
+ Set to -1 to use the model's default truncation size.
25
+ Set to k to keep only the last k tokens (left truncation).
26
+ Set to None to disable truncation.
27
normalize: Whether to normalize the embeddings outputs.
28
dimensions: Reduce the dimensions of embeddings
25
- if model support matryoshka representation.
29
+ if model support matryoshka representation.
30
activation: Whether to apply activation function to
27
- the classification outputs.
31
+ the classification outputs.
32
softmax: Whether to apply softmax to the reward outputs.
33
"""
34
+
35
+ # --8<-- [start:common-pooling-params]
36
truncate_prompt_tokens: Optional[Annotated[int,
37
msgspec.Meta(ge=-1)]] = None
32
- """If set to -1, will use the truncation size supported by the model. If
33
- set to an integer k, will use only the last k tokens from the prompt
34
- (i.e., left truncation). If set to `None`, truncation is disabled."""
38
+ # --8<-- [end:common-pooling-params]
39
40
## for embeddings models
41
+ # --8<-- [start:embedding-pooling-params]
42
dimensions: Optional[int] = None
43
normalize: Optional[bool] = None
44
+ # --8<-- [end:embedding-pooling-params]
45
40
- ## for classification models
46
+ ## for classification, scoring and rerank
47
+ # --8<-- [start:classification-pooling-params]
48
activation: Optional[bool] = None
49
+ # --8<-- [end:classification-pooling-params]
50
51
## for reward models
52
softmax: Optional[bool] = None