A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 23:45 UTC
[Doc] Link to RFC for pooling optimizations (#21806)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jul 29, 2025 at 14:53 UTC
a2480251ec92ba2a849464dde48db8a2b7f6ef81
1 file changed
+3
-3
docs/models/pooling_models.md
+3
-3
@@ -7,9 +7,9 @@ These models use a [Pooler][vllm.model_executor.layers.pooler.Pooler] to extract
7
before returning them.
8
9
!!! note
10
- We currently support pooling models primarily as a matter of convenience.
11
- As shown in the [Compatibility Matrix](../features/compatibility_matrix.md), most vLLM features are not applicable to
12
- pooling models as they only work on the generation or decode stage, so performance may not improve as much.
10
+ We currently support pooling models primarily as a matter of convenience. This is not guaranteed to have any performance improvement over using HF Transformers / Sentence Transformers directly.
11
+
12
+ We are now planning to optimize pooling models in vLLM. Please comment on <gh-issue:21796> if you have any suggestions!
13
14
## Configuration
15