@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 067e77f9a8
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 09:30 UTC

[Bugfix] Steaming continuous_usage_stats default to False (#9709)

Signed-off-by: Sam Stoelinga <sammiestoel@gmail.com>

Sam Stoelinga committed Oct 25, 2024 at 22:05 UTC 067e77f9a87c3466fce41c8fe8710fddc69ec26c
1 file changed +1 -1
vllm/entrypoints/openai/protocol.py
+1 -1
@@ -127,7 +127,7 @@ class ResponseFormat(OpenAIBaseModel):
127
128 class StreamOptions(OpenAIBaseModel):
129 include_usage: Optional[bool] = True
130 - continuous_usage_stats: Optional[bool] = True
130 + continuous_usage_stats: Optional[bool] = False
131
132
133 class FunctionDefinition(OpenAIBaseModel):