@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 53 lines 1.75 KB
Rendered Raw
1 # Engine-Core Smoke Tests
2
3 Start headless `vllm`:
4
5 ```bash
6 source ../vllm/.venv/bin/activate
7 HF_HUB_OFFLINE=1 \
8 VLLM_LOGGING_LEVEL=DEBUG \
9 VLLM_CPU_KVCACHE_SPACE=2 \
10 VLLM_HOST_IP=127.0.0.1 \
11 VLLM_LOOPBACK_IP=127.0.0.1 \
12 python3 -m vllm.entrypoints.cli.main serve Qwen/Qwen3-0.6B \
13 --headless \
14 --enable-sleep-mode \
15 --data-parallel-address 127.0.0.1 \
16 --data-parallel-rpc-port 62100 \
17 --data-parallel-size-local 1 \
18 --max-model-len 512 \
19 --dtype float16
20 ```
21
22 Run the Rust smoke test through the `vllm-engine-core-client` utility interface:
23
24 ```bash
25 cargo run -p vllm-engine-core-client --example external_engine_utility_call -- \
26 --handshake-address tcp://127.0.0.1:62100 \
27 --host 127.0.0.1
28 ```
29
30 If your current engine setup does not support sleep mode, skip the `sleep` / `wake_up` part of the
31 smoke:
32
33 ```bash
34 cargo run -p vllm-engine-core-client --example external_engine_utility_call -- \
35 --handshake-address tcp://127.0.0.1:62100 \
36 --host 127.0.0.1 \
37 --skip-sleep-wake
38 ```
39
40 Run the Rust smoke test for sample logprobs decoding through the raw engine-core request path:
41
42 ```bash
43 cargo run -p vllm-engine-core-client --example external_engine_logprobs -- \
44 --handshake-address tcp://127.0.0.1:62100 \
45 --host 127.0.0.1
46 ```
47
48 This smoke requests a small generated-token `logprobs` payload plus prompt logprobs over a much
49 longer prompt, so it exercises both the inline and aux-frame decode paths against a real engine.
50 The Rust client decodes those payloads into semantic per-position records rather than exposing the
51 raw ndarray/tensor wire shape.
52
53 IMPORTANT: You must restart `vllm` each time you run the smoke test, as the vLLM engine cannot manage frontend closures and subsequent reconnects. In other words, do not reuse existing `vllm` instances, if any.