A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 4, 06:15 UTC
[ROCm][CI] Use vLLM generation defaults for DeepSeek prefetch-offload eval (#41575)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Andreas Karatzas committed
May 5, 2026 at 20:08 UTC
4a8ae26e532c4b36acc1dd2f5e7c672d73832b4e
1 file changed
+1
.buildkite/scripts/scheduled_integration_test/deepseek_v2_lite_prefetch_offload.sh
+1
@@ -51,6 +51,7 @@ vllm serve "$MODEL" \
51
--offload-num-in-group 2 \
52
--offload-prefetch-step 1 \
53
--offload-params w13_weight w2_weight \
54
+ --generation-config vllm \
55
--port "$PORT" \
56
${EXTRA_ARGS+"${EXTRA_ARGS[@]}"} &
57
SERVER_PID=$!