A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 13, 20:45 UTC
[Docs] add runai_streamer_sharded to LoadConfig (#27937)
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Ning Xie committed
Nov 4, 2025 at 04:35 UTC
786030721efb2b85a582d65f9bb5d7197de06f83
1 file changed
+2
vllm/config/load.py
+2
@@ -40,6 +40,8 @@ class LoadConfig:
40
more information.\n
41
- "runai_streamer" will load the Safetensors weights using Run:ai Model
42
Streamer.\n
43
+ - "runai_streamer_sharded" will load weights from pre-sharded checkpoint
44
+ files using Run:ai Model Streamer.\n
45
- "bitsandbytes" will load the weights using bitsandbytes quantization.\n
46
- "sharded_state" will load weights from pre-sharded checkpoint files,
47
supporting efficient loading of tensor-parallel models.\n