@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 90 lines 3.11 KB
Rendered Raw
1 <!-- markdownlint-disable MD041 -->
2 --8<-- [start:installation]
3
4 vLLM has experimental support for macOS with Apple Silicon. For now, users must build from source to natively run on macOS.
5
6 Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.
7
8 !!! tip "GPU-Accelerated Inference with vLLM-Metal"
9 For GPU-accelerated inference on Apple Silicon using Metal, check out [vllm-metal](https://github.com/vllm-project/vllm-metal), a community-maintained hardware plugin that uses MLX as the compute backend.
10
11 --8<-- [end:installation]
12 --8<-- [start:requirements]
13
14 - OS: `macOS Sonoma` or later
15 - SDK: `XCode 15.4` or later with Command Line Tools
16 - Compiler: `Apple Clang >= 15.0.0`
17
18 !!! note
19 The macOS CPU build is smoke-tested in CI on the latest GA Apple Silicon
20 runner; other macOS or Apple Clang versions are best-effort.
21
22 --8<-- [end:requirements]
23 --8<-- [start:set-up-using-python]
24
25 --8<-- [end:set-up-using-python]
26 --8<-- [start:pre-built-wheels]
27
28 Currently, there are no pre-built Apple silicon CPU wheels.
29
30 --8<-- [end:pre-built-wheels]
31 --8<-- [start:build-wheel-from-source]
32
33 After installation of XCode and the Command Line Tools, which include Apple Clang, execute the following commands to build and install vLLM from source.
34
35 ```bash
36 git clone https://github.com/vllm-project/vllm.git
37 cd vllm
38 uv pip install -r requirements/cpu.txt
39 uv pip install -e .
40 ```
41
42 !!! note
43 On macOS the `VLLM_TARGET_DEVICE` is automatically set to `cpu`, which is currently the only supported device.
44
45 !!! example "Troubleshooting"
46 If the build fails with errors like the following where standard C++ headers cannot be found, try to remove and reinstall your
47 [Command Line Tools for Xcode](https://developer.apple.com/download/all/).
48
49 ```text
50 [...] fatal error: 'map' file not found
51 1 | #include <map>
52 | ^~~~~
53 1 error generated.
54 [2/8] Building CXX object CMakeFiles/_C.dir/csrc/cpu/pos_encoding.cpp.o
55
56 [...] fatal error: 'cstddef' file not found
57 10 | #include <cstddef>
58 | ^~~~~~~~~
59 1 error generated.
60 ```
61
62 ---
63
64 If the build fails with C++11/C++17 compatibility errors like the following, the issue is that the build system is defaulting to an older C++ standard:
65
66 ```text
67 [...] error: 'constexpr' is not a type
68 [...] error: expected ';' before 'constexpr'
69 [...] error: 'constexpr' does not name a type
70 ```
71
72 **Solution**: Your compiler might be using an older C++ standard. Edit `cmake/cpu_extension.cmake` and add `set(CMAKE_CXX_STANDARD 17)` before `set(CMAKE_CXX_STANDARD_REQUIRED ON)`.
73
74 To check your compiler's C++ standard support:
75 ```bash
76 clang++ -std=c++17 -pedantic -dM -E -x c++ /dev/null | grep __cplusplus
77 ```
78 On Apple Clang 16 you should see: `#define __cplusplus 201703L`
79
80 --8<-- [end:build-wheel-from-source]
81 --8<-- [start:pre-built-images]
82
83 Currently, there are no pre-built Arm silicon CPU images.
84
85 --8<-- [end:pre-built-images]
86 --8<-- [start:build-image-from-source]
87
88 --8<-- [end:build-image-from-source]
89 --8<-- [start:extra-information]
90 --8<-- [end:extra-information]