A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 06:30 UTC
[Docs] Add RunPod GPU deployment guide for vLLM (#34531)
Signed-off-by: lisperz <zhuchen200245@163.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Chen committed
Mar 4, 2026 at 12:11 UTC
138c5fa1869188ddeffd060ee586ed915d996d70
1 file changed
+87
docs/deployment/frameworks/runpod.md
new
+87
@@ -0,0 +1,87 @@
1
+# RunPod
2
+
3
+vLLM can be deployed on [RunPod](https://www.runpod.io/), a cloud GPU platform that provides on-demand and serverless GPU instances for AI inference workloads.
4
+
5
+## Prerequisites
6
+
7
+- A RunPod account with GPU pod access
8
+- A GPU pod running a CUDA-compatible template (e.g., `runpod/pytorch`)
9
+
10
+## Starting the Server
11
+
12
+SSH into your RunPod pod and launch the vLLM OpenAI-compatible server:
13
+
14
+```bash
15
+python -m vllm.entrypoints.openai.api_server \
16
+ --model <model-name> \
17
+ --host 0.0.0.0 \
18
+ --port 8000
19
+```
20
+
21
+!!! note
22
+
23
+ Use `--host 0.0.0.0` to bind to all interfaces so the server is reachable from outside the container.
24
+
25
+## Exposing Port 8000
26
+
27
+RunPod exposes HTTP services through its proxy. To make port 8000 accessible:
28
+
29
+1. In the RunPod dashboard, navigate to your pod settings.
30
+2. Add `8000` to the list of exposed HTTP ports.
31
+3. After the pod restarts, RunPod provides a public URL in the format:
32
+
33
+ ```text
34
+ https://<pod-id>-8000.proxy.runpod.net
35
+ ```
36
+
37
+## Troubleshooting 502 Bad Gateway
38
+
39
+A `502 Bad Gateway` error from the RunPod proxy typically means the server is not yet listening. Common causes:
40
+
41
+- **Model still loading** — Large models take time to download and load into GPU memory. Check the pod logs for progress.
42
+- **Wrong host binding** — Ensure you passed `--host 0.0.0.0`. Binding to `127.0.0.1` (the default) makes the server unreachable from the proxy.
43
+- **Port mismatch** — Verify the `--port` value matches the port exposed in the RunPod dashboard.
44
+- **Out of GPU memory** — The model may be too large for the allocated GPU. Check logs for CUDA OOM errors and consider using a larger instance or adding `--tensor-parallel-size` for multi-GPU pods.
45
+
46
+## Verifying the Deployment
47
+
48
+Once the server is running, test it with a curl request:
49
+
50
+!!! console "Command"
51
+
52
+ ```bash
53
+ curl https://<pod-id>-8000.proxy.runpod.net/v1/chat/completions \
54
+ -H "Content-Type: application/json" \
55
+ -d '{
56
+ "model": "<model-name>",
57
+ "messages": [
58
+ {"role": "user", "content": "Hello, how are you?"}
59
+ ],
60
+ "max_tokens": 50
61
+ }'
62
+ ```
63
+
64
+!!! console "Response"
65
+
66
+ ```json
67
+ {
68
+ "id": "chat-abc123",
69
+ "object": "chat.completion",
70
+ "choices": [
71
+ {
72
+ "message": {
73
+ "role": "assistant",
74
+ "content": "I'm doing well, thank you for asking! How can I help you today?"
75
+ },
76
+ "index": 0,
77
+ "finish_reason": "stop"
78
+ }
79
+ ]
80
+ }
81
+ ```
82
+
83
+You can also check the server health endpoint:
84
+
85
+```bash
86
+curl https://<pod-id>-8000.proxy.runpod.net/health
87
+```