A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 21:00 UTC
[misc] Mention `ray list nodes` command to troubleshoot ray issues (#14318)
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Rui Qiao committed
Mar 5, 2025 at 18:00 UTC
abcc61e0af9722b5012f2521d8c0503fe4164435
2 files changed
+9
-8
docs/source/serving/distributed_serving.md
+2
-2
@@ -81,7 +81,7 @@ Then you get a ray cluster of **containers**. Note that you need to keep the she
81
Since this is a ray cluster of **containers**, all the following commands should be executed in the **containers**, otherwise you are executing the commands on the host machine, which is not connected to the ray cluster. To enter the container, you can use `docker exec -it node /bin/bash`.
82
:::
83
84
-Then, on any node, use `docker exec -it node /bin/bash` to enter the container, execute `ray status` to check the status of the Ray cluster. You should see the right number of nodes and GPUs.
84
+Then, on any node, use `docker exec -it node /bin/bash` to enter the container, execute `ray status` and `ray list nodes` to check the status of the Ray cluster. You should see the right number of nodes and GPUs.
85
86
After that, on any node, use `docker exec -it node /bin/bash` to enter the container again. **In the container**, you can use vLLM as usual, just as you have all the GPUs on one node. The common practice is to set the tensor parallel size to the number of GPUs in each node, and the pipeline parallel size to the number of nodes. For example, if you have 16 GPUs in 2 nodes (8 GPUs per node), you can set the tensor parallel size to 8 and the pipeline parallel size to 2:
87
@@ -111,5 +111,5 @@ When you use huggingface repo id to refer to the model, you should append your h
111
:::
112
113
:::{warning}
114
-If you keep receiving the error message `Error: No available node types can fulfill resource request` but you have enough GPUs in the cluster, chances are your nodes have multiple IP addresses and vLLM cannot find the right one, especially when you are using multi-node inference. Please make sure vLLM and ray use the same IP address. You can set the `VLLM_HOST_IP` environment variable to the right IP address in the `run_cluster.sh` script (different for each node!), and check `ray status` to see the IP address used by Ray. See <gh-issue:7815> for more information.
114
+If you keep receiving the error message `Error: No available node types can fulfill resource request` but you have enough GPUs in the cluster, chances are your nodes have multiple IP addresses and vLLM cannot find the right one, especially when you are using multi-node inference. Please make sure vLLM and ray use the same IP address. You can set the `VLLM_HOST_IP` environment variable to the right IP address in the `run_cluster.sh` script (different for each node!), and check `ray status` and `ray list nodes` to see the IP address used by Ray. See <gh-issue:7815> for more information.
115
:::
vllm/executor/ray_utils.py
+7
-6
@@ -184,8 +184,9 @@ def _verify_bundles(placement_group: "PlacementGroup",
184
f"group {placement_group.id}. Node id -> bundles "
185
f"{node_id_to_bundle}. "
186
"You don't have enough GPUs available in a current node. Check "
187
- "`ray status` to see if you have available GPUs in a node "
188
- f"{driver_node_id} before starting an vLLM engine.")
187
+ "`ray status` and `ray list nodes` to see if you have available "
188
+ "GPUs in a node `{driver_node_id}` before starting an vLLM engine."
189
+ )
190
191
for node_id, bundles in node_id_to_bundle.items():
192
if len(bundles) < parallel_config.tensor_parallel_size:
@@ -225,8 +226,8 @@ def _wait_until_pg_ready(current_placement_group: "PlacementGroup"):
226
wait_interval *= 2
227
logger.info(
228
"Waiting for creating a placement group of specs for "
228
- "%d seconds. specs=%s. Check "
229
- "`ray status` to see if you have enough resources,"
229
+ "%d seconds. specs=%s. Check `ray status` and "
230
+ "`ray list nodes` to see if you have enough resources,"
231
" and make sure the IP addresses used by ray cluster"
232
" are the same as VLLM_HOST_IP environment variable"
233
" specified in each node if you are running on a multi-node.",
@@ -238,8 +239,8 @@ def _wait_until_pg_ready(current_placement_group: "PlacementGroup"):
239
raise ValueError(
240
"Cannot provide a placement group of "
241
f"{placement_group_specs=} within {PG_WAIT_TIMEOUT} seconds. See "
241
- "`ray status` to make sure the cluster has enough resources."
242
- ) from None
242
+ "`ray status` and `ray list nodes` to make sure the cluster has "
243
+ "enough resources.") from None
244
245
246
def _wait_until_pg_removed(current_placement_group: "PlacementGroup"):