@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 541c1852d3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 19:30 UTC

[ BugFix ] Fix ZMQ when `VLLM_PORT` is set (#7205)

Robert Shaw committed Aug 6, 2024 at 12:26 UTC 541c1852d37b9502fbc06253def70e901ca0c352
1 file changed +1 -1
vllm/envs.py
+1 -1
@@ -145,7 +145,7 @@ environment_variables: Dict[str, Callable[[], Any]] = {
145 # used when the frontend api server is running in multi-processing mode,
146 # to communicate with the backend engine process over ZMQ.
147 'VLLM_RPC_PORT':
148 - lambda: int(os.getenv('VLLM_PORT', '5570')),
148 + lambda: int(os.getenv('VLLM_RPC_PORT', '5570')),
149
150 # If true, will load models from ModelScope instead of Hugging Face Hub.
151 # note that the value is true or false, not numbers