A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Doc] Add note to `gte-Qwen2` models (#11808)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jan 7, 2025 at 21:50 UTC
c0efe92d8b9ef968a5b796fd7d6ebc426d78e726
1 file changed
+3
docs/source/models/supported_models.md
+3
@@ -430,6 +430,9 @@ You can set `--hf-overrides '{"is_causal": false}'` to change the attention mask
430
431
On the other hand, its 1.5B variant (`Alibaba-NLP/gte-Qwen2-1.5B-instruct`) uses causal attention
432
despite being described otherwise on its model card.
433
+
434
+Regardless of the variant, you need to enable `--trust-remote-code` for the correct tokenizer to be
435
+loaded. See [relevant issue on HF Transformers](https://github.com/huggingface/transformers/issues/34882).
436
```
437
438
If your model is not in the above list, we will try to automatically convert the model using