@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e11222333f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 08:00 UTC

fix: bug fix when penalties are negative (#913)

Co-authored-by: dongyong-lee <dongyong.lee@navercorp.com>

Dong-Yong Lee committed Sep 1, 2023 at 00:37 UTC e11222333f43c8466c57d0223380dcf297b02bac
1 file changed +1 -1
vllm/model_executor/layers/sampler.py
+1 -1
@@ -158,7 +158,7 @@ def _apply_penalties(
158 continue
159 p = presence_penalties[i]
160 f = frequency_penalties[i]
161 - if p < _SAMPLING_EPS and f < _SAMPLING_EPS:
161 + if abs(p) < _SAMPLING_EPS and abs(f) < _SAMPLING_EPS:
162 continue
163 indices.append(i)
164