A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 06:00 UTC
[DOC] fix doc typos (#19600)
Signed-off-by: Di Liu <liu-di@sjtu.edu.cn>
Di Liu committed
Jun 17, 2025 at 11:34 UTC
5b3ad5ecf21ee265b621f9b8b372c1c9e34a279e
1 file changed
+4
-4
vllm/model_executor/layers/vocab_parallel_embedding.py
+4
-4
@@ -176,17 +176,17 @@ class VocabParallelEmbedding(torch.nn.Module):
176
Therefore, the tensor format looks like the following:
177
TP1, rank 0 (no sharding):
178
|< --------BASE-------- >|< -BASE PADDING-- >|< -----LORA------ >|< -LORA PADDING-- >|
179
- corresponding token_id: | 0 | 1 | ... | 1009 | -1 | ... | -1 | 1010 | ... | 1015 | -1 | ... | -1 |
179
+ corresponding token_id: | 0 | 1 | ... | 1009 | -1 | ... | -1 | 1010 | ... | 1025 | -1 | ... | -1 |
180
index: | 0 | 1 | ... | 1009 | 1010 | ... | 1023 | 1024 | ... | 1039 | 1040 | ... | 1087 |
181
182
TP2, rank 0:
183
|< --------------------BASE--------------------- >|< -----LORA------ >|< -LORA PADDING- >|
184
- corresponding token_id: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 1000 | ... | 1015 | -1 | ... | -1 |
185
- index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 527 | 520 | ... | 543 |
184
+ corresponding token_id: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 1010 | ... | 1025 | -1 | ... | -1 |
185
+ index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 527 | 528 | ... | 543 |
186
TP2, rank 1:
187
|< -----------BASE----------- >|< -BASE PADDING- >|< -----------LORA PADDING----------- >|
188
corresponding token_id: | 512 | 513 | 514 | ... | 1009 | -1 | ... | -1 | -1 | ... | -1 | -1 | ... | -1 |
189
- index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 519 | 520 | ... | 543 |
189
+ index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 527 | 528 | ... | 543 |
190
191
Args:
192
num_embeddings: vocabulary size.