@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / c0efe92d8b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

[Doc] Add note to `gte-Qwen2` models (#11808)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Jan 7, 2025 at 21:50 UTC c0efe92d8b9ef968a5b796fd7d6ebc426d78e726
1 file changed +3
docs/source/models/supported_models.md
+3
@@ -430,6 +430,9 @@ You can set `--hf-overrides '{"is_causal": false}'` to change the attention mask
430
431 On the other hand, its 1.5B variant (`Alibaba-NLP/gte-Qwen2-1.5B-instruct`) uses causal attention
432 despite being described otherwise on its model card.
433 +
434 +Regardless of the variant, you need to enable `--trust-remote-code` for the correct tokenizer to be
435 +loaded. See [relevant issue on HF Transformers](https://github.com/huggingface/transformers/issues/34882).
436 ```
437
438 If your model is not in the above list, we will try to automatically convert the model using