A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 04:45 UTC
Update test case parameter to have the throughput above 8.0 (#19994)
Signed-off-by: Qiliang Cui <derrhein@gmail.com>
QiliangCui committed
Jun 23, 2025 at 17:18 UTC
a738dbb2a1238a5e4aafb92c629af3bce4213024
1 file changed
+2
-2
.buildkite/scripts/tpu/config_v6e_1.env
+2
-2
@@ -4,8 +4,8 @@ CONTAINER_NAME=vllm-tpu
4
5
# vllm config
6
MODEL=meta-llama/Llama-3.1-8B-Instruct
7
-MAX_NUM_SEQS=512
8
-MAX_NUM_BATCHED_TOKENS=512
7
+MAX_NUM_SEQS=256
8
+MAX_NUM_BATCHED_TOKENS=1024
9
TENSOR_PARALLEL_SIZE=1
10
MAX_MODEL_LEN=2048
11
DOWNLOAD_DIR=/mnt/disks/persist