@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 180 lines 7.26 KB
Rendered Raw
1 # vllm-frontend-rs
2
3 This is a Rust drop-in alternative frontend for vLLM. The current goal is to rebuild the northbound serving layer in Rust while still talking to the core Python vLLM engine process(es) via ZMQ over the existing engine boundary.
4
5 It should still be considered experimental, and is not feature-complete. We are working to add more functionality from the python front-end.
6
7 See <https://github.com/Inferact/vllm-frontend-rs> for the original commit history before it was moved into the main vllm repo.
8
9 ## Architecture
10
11 The component is organized as a Cargo workspace with several crates, layered bottom-up:
12
13 ```text
14 ┌─────────────────────────────────┐
15 │ vllm-cmd / vllm-rs │ CLI entrypoint:
16 │ │ Python vLLM frontend subprocess
17 │ │ Rust managed-engine serve mode
18 │ │ Engine-free render mode
19 ├─────────────────────────────────┤
20 │ vllm-server │ OpenAI-compatible HTTP API (axum)
21 ├─────────────────────────────────┤
22 │ vllm-chat │ Chat completions: template rendering,
23 │ │ structured assistant events,
24 │ │ reasoning & tool parsing
25 ├─────────────────────────────────┤
26 │ vllm-text │ Tokenizer & incremental detokenizer
27 ├─────────────────────────────────┤
28 │ vllm-llm │ Thin token-in/token-out facade over
29 │ │ the engine client
30 ├─────────────────────────────────┤
31 │ vllm-engine-core-client │ ZMQ transport + MessagePack protocol
32 │ │ for the headless vLLM engine
33 └─────────────────────────────────┘
34 ```
35
36 `vllm-rs` integrates into Python `vllm` as a Rust frontend subprocess.
37 Python owns process startup and launches the Rust API server as a Python-supervised worker, while
38 passing the inherited listening socket and transport addresses into `vllm-rs`.
39
40 For example:
41
42 ```bash
43 VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3-0.6B
44 ```
45
46 To also expose the Rust frontend's gRPC inference and control services, add
47 `--grpc-port`. The HTTP listener remains on `--port`:
48
49 ```bash
50 VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3-0.6B \
51 --host 127.0.0.1 --port 8000 --grpc-port 50051
52 ```
53
54 The gRPC listener binds on `--host` like HTTP, or on `127.0.0.1` when HTTP uses
55 `--uds`. When HTTP uses TCP, its port and the gRPC port must differ unless
56 requesting an automatically assigned port with `0`.
57
58 This also works with Python-supervised hybrid data parallelism. It requires a
59 frontend process and cannot be combined with `--headless` or
60 `--data-parallel-multi-port-external-lb`.
61
62 The separate `vllm serve MODEL --grpc --port 50051` command launches the Python
63 SMG `VllmEngine` gRPC server in place of HTTP. The Rust listener exposes the
64 `vllm.Inference` and `vllm.Control` services alongside HTTP. These are different
65 gRPC APIs; `--grpc` and `--grpc-port` are mutually exclusive.
66
67 ### RL weight synchronization
68
69 With `VLLM_SERVER_DEV_MODE=1`, the Rust frontend supports the HTTP weight-transfer
70 lifecycle used by `HTTPVLLMWeightSyncClient`: initialization, starting an update,
71 transferring weights, and finishing the update. It also supports draft-model
72 updates and `/update_weight_version` and `/weight_info` for version tracking.
73
74 For trainer-side usage, see the [IPC](../examples/rl/rlhf_http_ipc.py) and
75 [NCCL](../examples/rl/rlhf_http_nccl.py) examples. Configure the weight-transfer
76 backend on the server and pause generation during updates, as shown there.
77
78 ### External Engine
79
80 `vllm-rs serve` can be run standalone with `--data-parallel-size-local 0` when the Python engines
81 are started elsewhere and this node should run only the Rust frontend. The frontend still uses
82 the global `--data-parallel-size` to determine how many engines it expects to join the shared handshake.
83
84 ```bash
85 vllm serve Qwen/Qwen3-0.6B \
86 --headless \
87 --data-parallel-address 127.0.0.1 \
88 --data-parallel-rpc-port 62100 \
89 --data-parallel-size 1 \
90 --data-parallel-size-local 1
91 ```
92
93 Then start the Rust frontend-only server:
94
95 ```bash
96 vllm-rs serve Qwen/Qwen3-0.6B \
97 --data-parallel-address 127.0.0.1 \
98 --data-parallel-rpc-port 62100 \
99 --data-parallel-size 1 \
100 --data-parallel-size-local 0
101 ```
102
103 To build the `vllm-rs` in isolation:
104
105 ```bash
106 # from the local checkout
107 ./tools/build_rust.sh
108 ```
109
110 ### Engine-free renderer
111
112 `vllm-rs render` serves text-only request preprocessing without starting or
113 connecting to a Python inference engine:
114
115 ```bash
116 cargo run --manifest-path rust/Cargo.toml -p vllm-cmd --release -- \
117 render Qwen/Qwen3-32B \
118 --host 127.0.0.1 --max-model-len 32768
119 ```
120
121 It exposes `/v1/chat/completions/render` and `/v1/completions/render`. Only
122 tokenizer and model configuration files are loaded; model weights, PyTorch,
123 and vLLM kernels are not required.
124
125 To serve HTTPS, pass `--ssl-certfile`. If the certificate and private key are
126 in separate files, pass `--ssl-keyfile` for the key; otherwise the key is read from
127 the certificate file. For mTLS, pass `--ssl-ca-certs` with a CA
128 bundle and set `--ssl-cert-reqs` to `1` (optional) or `2` (required) to
129 verify client certificates. Use `--ssl-ciphers` to override the default
130 OpenSSL cipher list.
131
132 ```bash
133 # Combined PEM (cert + key in one file):
134 vllm-rs \
135 render Qwen/Qwen3-32B \
136 --host 127.0.0.1 --max-model-len 32768 \
137 --ssl-certfile /path/to/combined.pem
138
139 # Separate cert and key files:
140 vllm-rs \
141 render Qwen/Qwen3-32B \
142 --host 127.0.0.1 --max-model-len 32768 \
143 --ssl-certfile /path/to/cert.pem --ssl-keyfile /path/to/key.pem
144 ```
145
146 The render endpoints return the public token-in `GenerateRequest` consumed by
147 the Rust `/inference/v1/generate` endpoint. A chat render response, or one item
148 from a completion render response, can be submitted to that endpoint without
149 changing its fields.
150
151 The render and inference paths use the same `vllm-chat` and `vllm-text`
152 request-preparation logic; render mode stops before engine submission.
153 Tool-call and reasoning parsers use model-based auto-detection by default. Use
154 `--tool-call-parser` and `--reasoning-parser` to override either selection;
155 unified parsers require the same selection for both options.
156
157 ```bash
158 curl http://127.0.0.1:8000/v1/chat/completions/render \
159 -H "Content-Type: application/json" \
160 -d '{
161 "model": "Qwen/Qwen3-32B",
162 "messages": [{"role": "user", "content": "Hello"}],
163 "max_completion_tokens": 16
164 }'
165 ```
166
167 ### Example Request
168
169 After either full-frontend startup path, you can use any OpenAI-compatible
170 client against the inference endpoints:
171
172 ```bash
173 curl http://127.0.0.1:8000/v1/chat/completions \
174 -H "Content-Type: application/json" \
175 -d '{
176 "model": "Qwen/Qwen3-0.6B",
177 "messages": [{"role": "user", "content": "What is the capital of France?"}],
178 "stream": true
179 }'
180 ```