A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 02:15 UTC
[docs] convert supported configs to table (#20858)
Signed-off-by: reidliu41 <reid201711@gmail.com>
Reid committed
Jul 12, 2025 at 21:54 UTC
a86754a12bcaea314efc13000463befde0c9cd1c
1 file changed
+14
-30
docs/getting_started/installation/intel_gaudi.md
+14
-30
@@ -133,36 +133,20 @@ docker run \
133
The following configurations have been validated to function with
134
Gaudi2 devices. Configurations that are not listed may or may not work.
135
136
-- [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)
137
- on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
138
- datatype with random or greedy sampling
139
-- [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)
140
- on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
141
- datatype with random or greedy sampling
142
-- [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
143
- on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
144
- datatype with random or greedy sampling
145
-- [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
146
- on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
147
- datatype with random or greedy sampling
148
-- [meta-llama/Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B)
149
- on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
150
- datatype with random or greedy sampling
151
-- [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
152
- on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
153
- datatype with random or greedy sampling
154
-- [meta-llama/Llama-2-70b](https://huggingface.co/meta-llama/Llama-2-70b)
155
- with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
156
-- [meta-llama/Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf)
157
- with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
158
-- [meta-llama/Meta-Llama-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B)
159
- with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
160
-- [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)
161
- with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
162
-- [meta-llama/Meta-Llama-3.1-70B](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B)
163
- with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
164
-- [meta-llama/Meta-Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct)
165
- with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
136
+| Model | TP Size| dtype | Sampling |
137
+|-------|--------|--------|----------|
138
+| [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b) | 1, 2, 8 | BF16 | Random / Greedy |
139
+| [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | 1, 2, 8 | BF16 | Random / Greedy |
140
+| [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) | 1, 2, 8 | BF16 | Random / Greedy |
141
+| [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | 1, 2, 8 | BF16 | Random / Greedy |
142
+| [meta-llama/Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) | 1, 2, 8 | BF16 | Random / Greedy |
143
+| [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) | 1, 2, 8 | BF16 | Random / Greedy |
144
+| [meta-llama/Llama-2-70b](https://huggingface.co/meta-llama/Llama-2-70b) | 8 | BF16 | Random / Greedy |
145
+| [meta-llama/Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | 8 | BF16 | Random / Greedy |
146
+| [meta-llama/Meta-Llama-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B) | 8 | BF16 | Random / Greedy |
147
+| [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) | 8 | BF16 | Random / Greedy |
148
+| [meta-llama/Meta-Llama-3.1-70B](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B) | 8 | BF16 | Random / Greedy |
149
+| [meta-llama/Meta-Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | 8 | BF16 | Random / Greedy |
150
151
## Performance tuning
152