@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 128 lines 4.12 KB
Rendered Raw
1 <!-- markdownlint-disable MD041 -->
2 --8<-- [start:installation]
3
4 For GPU-accelerated inference on Apple Silicon, use [vLLM-Metal](https://github.com/vllm-project/vllm-metal), a community-maintained hardware plugin that uses MLX as the compute backend and provides native GPU acceleration via Apple's Metal framework.
5
6 vLLM-Metal works with MLX-optimized models from the [mlx-community](https://huggingface.co/mlx-community) organization on Hugging Face, which provides quantized versions of popular models optimized for Apple Silicon.
7
8 !!! tip
9 For installation and usage instructions, see the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section below.
10
11 --8<-- [end:installation]
12 --8<-- [start:requirements]
13
14 - OS: macOS Sonoma or later
15 - Hardware: Apple Silicon
16 - Metal support enabled
17
18 !!! note
19 See the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section below for installation instructions.
20
21 --8<-- [end:requirements]
22 --8<-- [start:set-up-using-python]
23
24 ## Set up using vLLM-Metal
25
26 vLLM-Metal is distributed as a separate package that provides native GPU acceleration on Apple Silicon.
27
28 To install vLLM-Metal, follow the installation instructions in the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
29
30 The installation will:
31
32 1. Set up the appropriate Python environment
33 2. Install MLX and required dependencies
34 3. Install the vLLM-Metal package
35
36 After installation, you can start using vLLM with Metal GPU acceleration.
37
38 !!! tip
39 When using vLLM-Metal, use models from the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face for best performance. These models are optimized for MLX and often include quantized versions (4-bit, 8-bit) that run efficiently on Apple Silicon.
40
41 Example model: `mlx-community/Qwen2.5-0.5B-Instruct-4bit`
42
43 ### Using vLLM-Metal
44
45 After installation, vLLM-Metal provides an easy-to-use CLI for running an OpenAI-compatible API server:
46
47 ```bash
48 # Activate the vLLM-Metal environment
49 source ~/.venv-vllm-metal/bin/activate
50
51 # Start the API server (specify your mlx-community model or it will use default)
52 vllm serve
53 ```
54
55 Wait for 2-3 minutes until you see application startup complete:
56 > INFO: Application startup complete.
57
58 Once the server is running, you have multiple options to interact with it:
59
60 #### Option 1: Interactive chat
61
62 Open a new terminal and start an interactive chat session:
63
64 ```bash
65 source ~/.venv-vllm-metal/bin/activate
66 vllm chat
67 ```
68
69 #### Option 2: API requests with curl
70
71 ```bash
72 curl http://localhost:8000/v1/chat/completions \
73 -H "Content-Type: application/json" \
74 -d '{
75 "messages": [{"role": "user", "content": "Hello!"}],
76 "max_tokens": 50
77 }'
78 ```
79
80 #### Option 3: Python with OpenAI SDK
81
82 ```python
83 from openai import OpenAI
84
85 client = OpenAI(
86 base_url="http://localhost:8000/v1",
87 api_key="dummy" # No auth required for local server
88 )
89
90 response = client.chat.completions.create(
91 model="mlx-community/Qwen2.5-0.5B-Instruct-4bit",
92 messages=[{"role": "user", "content": "Hello!"}]
93 )
94
95 print(response.choices[0].message.content)
96 ```
97
98 For more details on the `vllm` CLI commands, see the [OpenAI-compatible server documentation](../../serving/online_serving/openai_compatible_server.md).
99
100 --8<-- [end:set-up-using-python]
101 --8<-- [start:pre-built-wheels]
102
103 vLLM-Metal is installed via the vLLM-Metal package. See the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section above.
104
105 --8<-- [end:pre-built-wheels]
106 --8<-- [start:build-wheel-from-source]
107
108 For build instructions from source, refer to the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
109
110 --8<-- [end:build-wheel-from-source]
111 --8<-- [start:pre-built-images]
112
113 --8<-- [end:pre-built-images]
114 --8<-- [start:build-image-from-source]
115
116 --8<-- [end:build-image-from-source]
117 --8<-- [start:supported-features]
118
119 vLLM-Metal provides:
120
121 - Native GPU acceleration using Metal
122 - MLX-based compute backend optimized for Apple Silicon
123 - OpenAI-compatible API server
124 - Support for popular model architectures
125
126 For specific feature support and limitations, refer to the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal).
127
128 --8<-- [end:supported-features]