A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
correct LWS deployment yaml (#23104)
Signed-off-by: cberge908 <42270330+cberge908@users.noreply.github.com>
Christian Berge committed
Sep 2, 2025 at 14:04 UTC
8bd5844989373d6914f6dea94a56822e6d7380d4
2 files changed
+3
-5
docs/deployment/frameworks/lws.md
+2
-4
@@ -22,7 +22,7 @@ Deploy the following yaml file `lws.yaml`
22
metadata:
23
name: vllm
24
spec:
25
- replicas: 2
25
+ replicas: 1
26
leaderWorkerTemplate:
27
size: 2
28
restartPolicy: RecreateGroupOnPodRestart
@@ -41,7 +41,7 @@ Deploy the following yaml file `lws.yaml`
41
- sh
42
- -c
43
- "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
44
- python3 -m vllm.entrypoints.openai.api_server --port 8080 --model meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline_parallel_size 2"
44
+ vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8 --pipeline_parallel_size 2"
45
resources:
46
limits:
47
nvidia.com/gpu: "8"
@@ -126,8 +126,6 @@ Should get an output similar to this:
126
NAME READY STATUS RESTARTS AGE
127
vllm-0 1/1 Running 0 2s
128
vllm-0-1 1/1 Running 0 2s
129
-vllm-1 1/1 Running 0 2s
130
-vllm-1-1 1/1 Running 0 2s
129
```
130
131
Verify that the distributed tensor-parallel inference works:
examples/online_serving/multi-node-serving.sh
+1
-1
@@ -11,7 +11,7 @@
11
# Example usage:
12
# On the head node machine, start the Ray head node process and run a vLLM server.
13
# ./multi-node-serving.sh leader --ray_port=6379 --ray_cluster_size=<SIZE> [<extra ray args>] && \
14
-# python3 -m vllm.entrypoints.openai.api_server --port 8080 --model meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline_parallel_size 2
14
+# vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8 --pipeline_parallel_size 2
15
#
16
# On each worker node, start the Ray worker node process.
17
# ./multi-node-serving.sh worker --ray_address=<HEAD_NODE_IP> --ray_port=6379 [<extra ray args>]