@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f7c1234990
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

[Fix] Fissertion on YaRN model len (#2984)

Woosuk Kwon committed Feb 23, 2024 at 12:57 UTC f7c1234990793008f3d44790fd274040f26c4ee4
1 file changed -1
vllm/model_executor/layers/rotary_embedding.py
-1
@@ -354,7 +354,6 @@ def get_rope(
354 elif scaling_type == "yarn":
355 original_max_position = rope_scaling[
356 "original_max_position_embeddings"]
357 - assert max_position == original_max_position * scaling_factor
357 extra_kwargs = {
358 k: v
359 for k, v in rope_scaling.items()