| 1 | # `vllm-rs` CLI Quick Start |
| 2 | |
| 3 | Start Qwen3 with one managed `vllm-rs serve` command from the repo root: |
| 4 | |
| 5 | ```bash |
| 6 | HF_HUB_OFFLINE=1 \ |
| 7 | VLLM_CPU_KVCACHE_SPACE=2 \ |
| 8 | VLLM_HOST_IP=127.0.0.1 \ |
| 9 | VLLM_LOOPBACK_IP=127.0.0.1 \ |
| 10 | cargo run --bin vllm-rs -- serve \ |
| 11 | Qwen/Qwen3-0.6B \ |
| 12 | --python ../vllm/.venv/bin/python \ |
| 13 | --max-model-len 512 \ |
| 14 | -- \ |
| 15 | --dtype float16 |
| 16 | ``` |
| 17 | |
| 18 | This launches: |
| 19 | |
| 20 | - a managed headless Python `vllm` engine |
| 21 | - the Rust OpenAI-compatible frontend on `127.0.0.1:8000` |
| 22 | |
| 23 | All Python engine arguments must be placed after `--`. Arguments before `--` are parsed by the Rust |
| 24 | frontend itself. |
| 25 | |
| 26 | You can then send OpenAI-style requests to the Rust frontend: |
| 27 | |
| 28 | ```bash |
| 29 | curl http://127.0.0.1:8000/v1/chat/completions \ |
| 30 | -H "Content-Type: application/json" \ |
| 31 | -d '{ |
| 32 | "model": "Qwen/Qwen3-0.6B", |
| 33 | "messages": [{"role": "user", "content": "What is the capital of France?"}], |
| 34 | "stream": true |
| 35 | }' |
| 36 | ``` |
| 37 | |
| 38 | If you already started headless `vllm` yourself, use `frontend` instead: |
| 39 | |
| 40 | ```bash |
| 41 | cargo run --bin vllm-rs -- frontend \ |
| 42 | --handshake-address tcp://127.0.0.1:62100 \ |
| 43 | Qwen/Qwen3-0.6B |
| 44 | ``` |