| 1 | # RunPod |
| 2 | |
| 3 | vLLM can be deployed on [RunPod](https://www.runpod.io/), a cloud GPU platform that provides on-demand and serverless GPU instances for AI inference workloads. |
| 4 | |
| 5 | ## Prerequisites |
| 6 | |
| 7 | - A RunPod account with GPU pod access |
| 8 | - A GPU pod running a CUDA-compatible template (e.g., `runpod/pytorch`) |
| 9 | |
| 10 | ## Starting the Server |
| 11 | |
| 12 | SSH into your RunPod pod and launch the vLLM OpenAI-compatible server: |
| 13 | |
| 14 | ```bash |
| 15 | vllm serve <model-name> \ |
| 16 | --host 0.0.0.0 \ |
| 17 | --port 8000 |
| 18 | ``` |
| 19 | |
| 20 | !!! note |
| 21 | |
| 22 | Use `--host 0.0.0.0` to bind to all interfaces so the server is reachable from outside the container. |
| 23 | |
| 24 | ## Exposing Port 8000 |
| 25 | |
| 26 | RunPod exposes HTTP services through its proxy. To make port 8000 accessible: |
| 27 | |
| 28 | 1. In the RunPod dashboard, navigate to your pod settings. |
| 29 | 2. Add `8000` to the list of exposed HTTP ports. |
| 30 | 3. After the pod restarts, RunPod provides a public URL in the format: |
| 31 | |
| 32 | ```text |
| 33 | https://<pod-id>-8000.proxy.runpod.net |
| 34 | ``` |
| 35 | |
| 36 | ## Troubleshooting 502 Bad Gateway |
| 37 | |
| 38 | A `502 Bad Gateway` error from the RunPod proxy typically means the server is not yet listening. Common causes: |
| 39 | |
| 40 | - **Model still loading** — Large models take time to download and load into GPU memory. Check the pod logs for progress. |
| 41 | - **Wrong host binding** — Ensure you passed `--host 0.0.0.0`. Binding to `127.0.0.1` (the default) makes the server unreachable from the proxy. |
| 42 | - **Port mismatch** — Verify the `--port` value matches the port exposed in the RunPod dashboard. |
| 43 | - **Out of GPU memory** — The model may be too large for the allocated GPU. Check logs for CUDA OOM errors and consider using a larger instance or adding `--tensor-parallel-size` for multi-GPU pods. |
| 44 | |
| 45 | ## Verifying the Deployment |
| 46 | |
| 47 | Once the server is running, test it with a curl request: |
| 48 | |
| 49 | !!! console "Command" |
| 50 | |
| 51 | ```bash |
| 52 | curl https://<pod-id>-8000.proxy.runpod.net/v1/chat/completions \ |
| 53 | -H "Content-Type: application/json" \ |
| 54 | -d '{ |
| 55 | "model": "<model-name>", |
| 56 | "messages": [ |
| 57 | {"role": "user", "content": "Hello, how are you?"} |
| 58 | ], |
| 59 | "max_tokens": 50 |
| 60 | }' |
| 61 | ``` |
| 62 | |
| 63 | !!! console "Response" |
| 64 | |
| 65 | ```json |
| 66 | { |
| 67 | "id": "chat-abc123", |
| 68 | "object": "chat.completion", |
| 69 | "choices": [ |
| 70 | { |
| 71 | "message": { |
| 72 | "role": "assistant", |
| 73 | "content": "I'm doing well, thank you for asking! How can I help you today?" |
| 74 | }, |
| 75 | "index": 0, |
| 76 | "finish_reason": "stop" |
| 77 | } |
| 78 | ] |
| 79 | } |
| 80 | ``` |
| 81 | |
| 82 | You can also check the server health endpoint: |
| 83 | |
| 84 | ```bash |
| 85 | curl https://<pod-id>-8000.proxy.runpod.net/health |
| 86 | ``` |