| 1 | # Chat Smoke Test |
| 2 | |
| 3 | Start a fresh headless `vllm` engine: |
| 4 | |
| 5 | ```bash |
| 6 | source ../vllm/.venv/bin/activate |
| 7 | HF_HUB_OFFLINE=1 \ |
| 8 | VLLM_LOGGING_LEVEL=DEBUG \ |
| 9 | VLLM_CPU_KVCACHE_SPACE=2 \ |
| 10 | VLLM_HOST_IP=127.0.0.1 \ |
| 11 | VLLM_LOOPBACK_IP=127.0.0.1 \ |
| 12 | python3 -m vllm.entrypoints.cli.main serve Qwen/Qwen3-0.6B \ |
| 13 | --headless \ |
| 14 | --data-parallel-address 127.0.0.1 \ |
| 15 | --data-parallel-rpc-port 62100 \ |
| 16 | --data-parallel-size-local 1 \ |
| 17 | --max-model-len 512 \ |
| 18 | --dtype float16 |
| 19 | ``` |
| 20 | |
| 21 | Run the Rust chat smoke test through the `vllm-chat` interface: |
| 22 | |
| 23 | ```bash |
| 24 | cargo run -p vllm-chat --example external_engine_chat_qwen -- \ |
| 25 | --handshake-address tcp://127.0.0.1:62100 \ |
| 26 | --host 127.0.0.1 \ |
| 27 | --prompt 'What is the capital of France? Answer with one word.' |
| 28 | ``` |
| 29 | |
| 30 | The example now defaults to `Qwen/Qwen3-0.6B`. The current `vllm-chat` |
| 31 | request model stays text-first and supports either plain string content or |
| 32 | OpenAI-style text blocks, while the output side now emits structured assistant |
| 33 | events and automatically separates reasoning blocks for supported models. Tool |
| 34 | use and multimodal inputs are still out of scope. It uses the Rust |
| 35 | `tokenizers` library for the tokenizer itself, plus standard Hugging Face |
| 36 | config files to load the chat template and EOS metadata. |
| 37 | |
| 38 | IMPORTANT: Restart `vllm` each time you run the smoke test. The current headless |
| 39 | engine cannot safely handle frontend reconnects after the client shuts down. |