A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 03:30 UTC
[TPU] Add a case to cover RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 (#20385)
Signed-off-by: Qiliang Cui <derrhein@gmail.com>
QiliangCui committed
Jul 2, 2025 at 23:46 UTC
4ff61ababa25f4a519185013c9cce00142341f04
1 file changed
+14
.buildkite/scripts/tpu/quantized_v6e_1.env
new
+14
@@ -0,0 +1,14 @@
1
+# Environment config
2
+TEST_NAME=llama8bw8a8
3
+CONTAINER_NAME=vllm-tpu
4
+
5
+# vllm config
6
+MODEL=RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8
7
+MAX_NUM_SEQS=128
8
+MAX_NUM_BATCHED_TOKENS=1024
9
+TENSOR_PARALLEL_SIZE=1
10
+MAX_MODEL_LEN=2048
11
+DOWNLOAD_DIR=/mnt/disks/persist
12
+EXPECTED_THROUGHPUT=10.0
13
+INPUT_LEN=1800
14
+OUTPUT_LEN=128