@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e4c4072c94
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 03:00 UTC

[Bugfix] Remove key sorting for `guided_json` parameter in OpenAi compatible Server (#3945)

Daniel E Marasco committed Apr 10, 2024 at 13:15 UTC e4c4072c94b346053768691451566c56664e26a7
1 file changed +1 -1
vllm/model_executor/guided_decoding.py
+1 -1
@@ -91,7 +91,7 @@ def _get_guide_and_mode(
91 json = request.guided_json
92 if isinstance(json, dict):
93 # turn dict into hashable string
94 - json = json_dumps(json, sort_keys=True)
94 + json = json_dumps(json)
95 elif isinstance(json, BaseModel):
96 # use pydantic signature so that different model classes
97 # with the same fields will get hashed the same