A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 12:15 UTC
[Security] Don't bind tcp zmq socket to all interfaces (#17197)
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Russell Bryant committed
Apr 28, 2025 at 13:08 UTC
a0304dc504c85f421d38ef47c64f83046a13641c
2 files changed
+5
-1
docs/source/serving/distributed_serving.md
+4
@@ -77,6 +77,10 @@ bash run_cluster.sh \
77
78
Then you get a ray cluster of **containers**. Note that you need to keep the shells running these commands alive to hold the cluster. Any shell disconnect will terminate the cluster. In addition, please note that the argument `ip_of_head_node` should be the IP address of the head node, which is accessible by all the worker nodes. The IP addresses of each worker node should be specified in the `VLLM_HOST_IP` environment variable, and should be different for each worker node. Please check the network configuration of your cluster to make sure the nodes can communicate with each other through the specified IP addresses.
79
80
+:::{warning}
81
+It is considered best practice to set `VLLM_HOST_IP` to an address on a private network segment for the vLLM cluster. The traffic sent here is not encrypted. The endpoints are also exchanging data in a format that could be exploited to execute arbitrary code should a malicious party gain access to the network. Please ensure that this network is not reachable by any untrusted parties.
82
+:::
83
+
84
:::{warning}
85
Since this is a ray cluster of **containers**, all the following commands should be executed in the **containers**, otherwise you are executing the commands on the host machine, which is not connected to the ray cluster. To enter the container, you can use `docker exec -it node /bin/bash`.
86
:::
vllm/distributed/device_communicators/shm_broadcast.py
+1
-1
@@ -241,7 +241,7 @@ class MessageQueue:
241
self.remote_socket.setsockopt(IPV6, 1)
242
remote_addr_ipv6 = True
243
connect_ip = f"[{connect_ip}]"
244
- socket_addr = f"tcp://*:{remote_subscribe_port}"
244
+ socket_addr = f"tcp://{connect_ip}:{remote_subscribe_port}"
245
self.remote_socket.bind(socket_addr)
246
remote_subscribe_addr = f"tcp://{connect_ip}:{remote_subscribe_port}"
247
else: