@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / cf0d279142
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 3, 22:45 UTC

[Docs] Add Apple Silicon documentation for vLLM-Metal GPU support (#41987)

Signed-off-by: alexagriffith <agriffith96@gmail.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

Alexa Griffith committed May 11, 2026 at 14:34 UTC cf0d2791420eefab4a3416551bb3f4a1288f0aea
4 files changed +176 -3
docs/getting_started/installation/README.md
+4 -3
@@ -3,9 +3,10 @@
3 vLLM supports the following hardware platforms:
4
5 - [GPU](gpu.md)
6 - - [NVIDIA CUDA](gpu.md#nvidia-cuda)
7 - - [AMD ROCm](gpu.md#amd-rocm)
8 - - [Intel XPU](gpu.md#intel-xpu)
6 + - [NVIDIA CUDA](gpu.md)
7 + - [AMD ROCm](gpu.md)
8 + - [Intel XPU](gpu.md)
9 + - [Apple Silicon](gpu.md) (via [vLLM-Metal](https://github.com/vllm-project/vllm-metal))
10 - [CPU](cpu.md)
11 - [Intel/AMD x86](cpu.md#intelamd-x86)
12 - [ARM AArch64](cpu.md#arm-aarch64)
docs/getting_started/installation/gpu.apple.inc.md new
+125
@@ -0,0 +1,125 @@
1 +<!-- markdownlint-disable MD041 -->
2 +--8<-- [start:installation]
3 +
4 +For GPU-accelerated inference on Apple Silicon, use [vLLM-Metal](https://github.com/vllm-project/vllm-metal), a community-maintained hardware plugin that uses MLX as the compute backend and provides native GPU acceleration via Apple's Metal framework.
5 +
6 +vLLM-Metal works with MLX-optimized models from the [mlx-community](https://huggingface.co/mlx-community) organization on Hugging Face, which provides quantized versions of popular models optimized for Apple Silicon.
7 +
8 +!!! tip
9 + For installation and usage instructions, see the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section below.
10 +
11 +--8<-- [end:installation]
12 +--8<-- [start:requirements]
13 +
14 +- OS: macOS Sonoma or later
15 +- Hardware: Apple Silicon
16 +- Metal support enabled
17 +
18 +!!! note
19 + See the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section below for installation instructions.
20 +
21 +--8<-- [end:requirements]
22 +--8<-- [start:set-up-using-python]
23 +
24 +## Set up using vLLM-Metal
25 +
26 +vLLM-Metal is distributed as a separate package that provides native GPU acceleration on Apple Silicon.
27 +
28 +To install vLLM-Metal, follow the installation instructions in the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
29 +
30 +The installation will:
31 +
32 +1. Set up the appropriate Python environment
33 +2. Install MLX and required dependencies
34 +3. Install the vLLM-Metal package
35 +
36 +After installation, you can start using vLLM with Metal GPU acceleration.
37 +
38 +!!! tip
39 + When using vLLM-Metal, use models from the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face for best performance. These models are optimized for MLX and often include quantized versions (4-bit, 8-bit) that run efficiently on Apple Silicon.
40 +
41 + Example model: `mlx-community/Qwen2.5-0.5B-Instruct-4bit`
42 +
43 +### Using vLLM-Metal
44 +
45 +After installation, vLLM-Metal provides an easy-to-use CLI for running an OpenAI-compatible API server:
46 +
47 +```bash
48 +# Activate the vLLM-Metal environment
49 +source ~/.venv-vllm-metal/bin/activate
50 +
51 +# Start the API server (specify your mlx-community model or it will use default)
52 +vllm serve
53 +```
54 +
55 +Once the server is running, you have multiple options to interact with it:
56 +
57 +#### Option 1: Interactive chat
58 +
59 +Open a new terminal and start an interactive chat session:
60 +
61 +```bash
62 +source ~/.venv-vllm-metal/bin/activate
63 +vllm chat
64 +```
65 +
66 +#### Option 2: API requests with curl
67 +
68 +```bash
69 +curl http://localhost:8000/v1/chat/completions \
70 + -H "Content-Type: application/json" \
71 + -d '{
72 + "messages": [{"role": "user", "content": "Hello!"}],
73 + "max_tokens": 50
74 + }'
75 +```
76 +
77 +#### Option 3: Python with OpenAI SDK
78 +
79 +```python
80 +from openai import OpenAI
81 +
82 +client = OpenAI(
83 + base_url="http://localhost:8000/v1",
84 + api_key="dummy" # No auth required for local server
85 +)
86 +
87 +response = client.chat.completions.create(
88 + model="mlx-community/Qwen2.5-0.5B-Instruct-4bit",
89 + messages=[{"role": "user", "content": "Hello!"}]
90 +)
91 +
92 +print(response.choices[0].message.content)
93 +```
94 +
95 +For more details on the `vllm` CLI commands, see the [OpenAI-compatible server documentation](../../serving/openai_compatible_server.md).
96 +
97 +--8<-- [end:set-up-using-python]
98 +--8<-- [start:pre-built-wheels]
99 +
100 +vLLM-Metal is installed via the vLLM-Metal package. See the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section above.
101 +
102 +--8<-- [end:pre-built-wheels]
103 +--8<-- [start:build-wheel-from-source]
104 +
105 +For build instructions from source, refer to the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
106 +
107 +--8<-- [end:build-wheel-from-source]
108 +--8<-- [start:pre-built-images]
109 +
110 +--8<-- [end:pre-built-images]
111 +--8<-- [start:build-image-from-source]
112 +
113 +--8<-- [end:build-image-from-source]
114 +--8<-- [start:supported-features]
115 +
116 +vLLM-Metal provides:
117 +
118 +- Native GPU acceleration using Metal
119 +- MLX-based compute backend optimized for Apple Silicon
120 +- OpenAI-compatible API server
121 +- Support for popular model architectures
122 +
123 +For specific feature support and limitations, refer to the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal).
124 +
125 +--8<-- [end:supported-features]
docs/getting_started/installation/gpu.md
+32
@@ -18,6 +18,10 @@ vLLM is a Python library that supports the following GPU variants. Select your G
18
19 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:installation"
20
21 +=== "Apple Silicon"
22 +
23 + --8<-- "docs/getting_started/installation/gpu.apple.inc.md:installation"
24 +
25 ## Requirements
26
27 - OS: Linux
@@ -38,6 +42,10 @@ vLLM is a Python library that supports the following GPU variants. Select your G
42
43 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:requirements"
44
45 +=== "Apple Silicon"
46 +
47 + --8<-- "docs/getting_started/installation/gpu.apple.inc.md:requirements"
48 +
49 ## Set up using Python
50
51 ### Create a new Python environment
@@ -56,6 +64,10 @@ vLLM is a Python library that supports the following GPU variants. Select your G
64
65 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:set-up-using-python"
66
67 +=== "Apple Silicon"
68 +
69 + --8<-- "docs/getting_started/installation/gpu.apple.inc.md:set-up-using-python"
70 +
71 ### Pre-built wheels {#pre-built-wheels}
72
73 === "NVIDIA CUDA"
@@ -70,6 +82,10 @@ vLLM is a Python library that supports the following GPU variants. Select your G
82
83 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:pre-built-wheels"
84
85 +=== "Apple Silicon"
86 +
87 + --8<-- "docs/getting_started/installation/gpu.apple.inc.md:pre-built-wheels"
88 +
89 ### Build wheel from source
90
91 === "NVIDIA CUDA"
@@ -84,6 +100,10 @@ vLLM is a Python library that supports the following GPU variants. Select your G
100
101 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:build-wheel-from-source"
102
103 +=== "Apple Silicon"
104 +
105 + --8<-- "docs/getting_started/installation/gpu.apple.inc.md:build-wheel-from-source"
106 +
107 ## Set up using Docker
108
109 ### Pre-built images
@@ -102,6 +122,10 @@ vLLM is a Python library that supports the following GPU variants. Select your G
122
123 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:pre-built-images"
124
125 +=== "Apple Silicon"
126 +
127 + --8<-- "docs/getting_started/installation/gpu.apple.inc.md:pre-built-images"
128 +
129 --8<-- [end:pre-built-images]
130
131 ### Build image from source
@@ -120,6 +144,10 @@ vLLM is a Python library that supports the following GPU variants. Select your G
144
145 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:build-image-from-source"
146
147 +=== "Apple Silicon"
148 +
149 + --8<-- "docs/getting_started/installation/gpu.apple.inc.md:build-image-from-source"
150 +
151 --8<-- [end:build-image-from-source]
152
153 ## Supported features
@@ -135,3 +163,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
163 === "Intel XPU"
164
165 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:supported-features"
166 +
167 +=== "Apple Silicon"
168 +
169 + --8<-- "docs/getting_started/installation/gpu.apple.inc.md:supported-features"
docs/getting_started/quickstart.md
+15
@@ -10,6 +10,9 @@ This guide will help you quickly get started with vLLM to perform:
10 - OS: Linux
11 - Python: 3.10 -- 3.13
12
13 +!!! note
14 + vLLM also works on macOS with [vLLM-Metal](https://github.com/vllm-project/vllm-metal) for Apple Silicon GPU acceleration. See the [GPU installation guide](installation/gpu.md) and select the "Apple Silicon" tab.
15 +
16 ## Installation
17
18 === "NVIDIA CUDA"
@@ -73,6 +76,18 @@ This guide will help you quickly get started with vLLM to perform:
76 !!! note
77 For more detailed instructions, including Docker, installing from source, and troubleshooting, please refer to the [vLLM on TPU documentation](https://docs.vllm.ai/projects/tpu/en/latest/).
78
79 +=== "Apple Silicon (Mac)"
80 +
81 + If you are using Apple Silicon Macs, you can use vLLM-Metal for GPU-accelerated inference via Apple's Metal framework.
82 +
83 + Follow the installation instructions in the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
84 +
85 + !!! note
86 + vLLM-Metal uses MLX instead of PyTorch as the compute backend and requires MLX-optimized models from the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face.
87 +
88 + !!! tip
89 + For more detailed instructions, please refer to the [GPU installation guide](installation/gpu.md) and select the "Apple Silicon" tab.
90 +
91 !!! note
92 For more detail and non-CUDA platforms, please refer to the [installation guide](installation/README.md) for specific instructions on how to install vLLM.
93