A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
Normalize head weights for Baichuan 2 (#1876)
Woosuk Kwon committed
Nov 30, 2023 at 20:03 UTC
e5452ddfd6e9a08d5e15bd81a010934550b9b507
3 files changed
+13
-2
README.md
+1
-1
@@ -47,7 +47,7 @@ vLLM is flexible and easy to use with:
47
vLLM seamlessly supports many Hugging Face models, including the following architectures:
48
49
- Aquila & Aquila2 (`BAAI/AquilaChat2-7B`, `BAAI/AquilaChat2-34B`, `BAAI/Aquila-7B`, `BAAI/AquilaChat-7B`, etc.)
50
-- Baichuan (`baichuan-inc/Baichuan-7B`, `baichuan-inc/Baichuan-13B-Chat`, etc.)
50
+- Baichuan & Baichuan2 (`baichuan-inc/Baichuan2-13B-Chat`, `baichuan-inc/Baichuan-7B`, etc.)
51
- BLOOM (`bigscience/bloom`, `bigscience/bloomz`, etc.)
52
- ChatGLM (`THUDM/chatglm2-6b`, `THUDM/chatglm3-6b`, etc.)
53
- Falcon (`tiiuae/falcon-7b`, `tiiuae/falcon-40b`, `tiiuae/falcon-rw-7b`, etc.)
docs/source/models/supported_models.rst
+1
-1
@@ -19,7 +19,7 @@ Alongside each architecture, we include some popular models that use it.
19
- :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
20
* - :code:`BaiChuanForCausalLM`
21
- Baichuan
22
- - :code:`baichuan-inc/Baichuan-7B`, :code:`baichuan-inc/Baichuan-13B-Chat`, etc.
22
+ - :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
23
* - :code:`ChatGLMModel`
24
- ChatGLM
25
- :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
vllm/model_executor/models/baichuan.py
+11
@@ -341,6 +341,17 @@ class BaiChuanBaseForCausalLM(nn.Module):
341
model_name_or_path, cache_dir, load_format, revision):
342
if "rotary_emb.inv_freq" in name:
343
continue
344
+ if name == "lm_head.weight":
345
+ # Unlike Baichuan, Baichuan2 normalizes the head weights. Refer to:
346
+ # https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat/blob/84603cde5ebffb6084e476cfaeceaf0b8b91fe54/modeling_baichuan.py#L508
347
+ # Distinguish between Baichuan and Baichuan2 by checking the
348
+ # vocab size. This is suggested by
349
+ # https://github.com/vllm-project/vllm/pull/1022#discussion_r1325652704
350
+ is_baichuan2 = self.config.vocab_size == 125696
351
+ if is_baichuan2:
352
+ loaded_weight = torch.nn.functional.normalize(
353
+ loaded_weight)
354
+
355
for (param_name, weight_name, shard_id) in stacked_params_mapping:
356
if weight_name not in name:
357
continue