@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 2358ca527b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 23:30 UTC

[Doc]: Improve feature tables (#13224)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Feb 18, 2025 at 10:52 UTC 2358ca527bd0d2f8a99ced33586cc0d72ed51092
5 files changed +142 -128
docs/source/_static/custom.css new
+8
@@ -0,0 +1,8 @@
1 +.vertical-table-header th.head:not(.stub) {
2 + writing-mode: sideways-lr;
3 + white-space: nowrap;
4 + max-width: 0;
5 + p {
6 + margin: 0;
7 + }
8 +}
docs/source/conf.py
+4 -5
@@ -78,8 +78,12 @@ html_theme_options = {
78 'use_repository_button': True,
79 'use_edit_page_button': True,
80 }
81 +# Add any paths that contain custom static files (such as style sheets) here,
82 +# relative to this directory. They are copied after the builtin static files,
83 +# so a file named "default.css" will overwrite the builtin "default.css".
84 html_static_path = ["_static"]
85 html_js_files = ["custom.js"]
86 +html_css_files = ["custom.css"]
87
88 myst_url_schemes = {
89 'http': None,
@@ -121,11 +125,6 @@ if READTHEDOCS_VERSION_TYPE == "tag":
125 if os.path.exists(header_file):
126 os.remove(header_file)
127
124 -# Add any paths that contain custom static files (such as style sheets) here,
125 -# relative to this directory. They are copied after the builtin static files,
126 -# so a file named "default.css" will overwrite the builtin "default.css".
127 -# html_static_path = ['_static']
128 -
128
129 # Generate additional rst documentation here.
130 def setup(app):
docs/source/features/compatibility_matrix.md
+79 -72
@@ -4,8 +4,14 @@
4
5 The tables below show mutually exclusive features and the support on some hardware.
6
7 +The symbols used have the following meanings:
8 +
9 +- ✅ = Full compatibility
10 +- 🟠 = Partial compatibility
11 +- ❌ = No compatibility
12 +
13 :::{note}
8 -Check the '✗' with links to see tracking issue for unsupported feature/hardware combination.
14 +Check the ❌ or 🟠 with links to see tracking issue for unsupported feature/hardware combination.
15 :::
16
17 ## Feature x Feature
@@ -29,6 +35,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
35 :header-rows: 1
36 :stub-columns: 1
37 :widths: auto
38 +:class: vertical-table-header
39
40 - * Feature
41 * [CP](#chunked-prefill)
@@ -48,7 +55,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
55 * beam-search
56 * <abbr title="Guided Decoding">guided dec</abbr>
57 - * [CP](#chunked-prefill)
51 - *
58 + * ✅
59 *
60 *
61 *
@@ -66,7 +73,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
73 *
74 - * [APC](#automatic-prefix-caching)
75 * ✅
69 - *
76 + * ✅
77 *
78 *
79 *
@@ -82,9 +89,9 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
89 *
90 *
91 - * [LoRA](#lora-adapter)
85 - * [✗](gh-pr:9057)
92 * ✅
87 - *
93 + * ✅
94 + * ✅
95 *
96 *
97 *
@@ -102,7 +109,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
109 * ✅
110 * ✅
111 * ✅
105 - *
112 + * ✅
113 *
114 *
115 *
@@ -118,9 +125,9 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
125 - * [SD](#spec_decode)
126 * ✅
127 * ✅
121 - * ✗
128 + * ❌
129 + * ✅
130 * ✅
123 - *
131 *
132 *
133 *
@@ -138,7 +145,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
145 * ✅
146 * ✅
147 * ✅
141 - *
148 + * ✅
149 *
150 *
151 *
@@ -150,13 +157,13 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
157 *
158 *
159 - * <abbr title="Pooling Models">pooling</abbr>
153 - * ✗
154 - * ✗
155 - * ✗
156 - * ✗
157 - * ✗
158 - * ✗
159 - *
160 + * ❌
161 + * ❌
162 + * ❌
163 + * ❌
164 + * ❌
165 + * ❌
166 + * ✅
167 *
168 *
169 *
@@ -167,14 +174,14 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
174 *
175 *
176 - * <abbr title="Encoder-Decoder Models">enc-dec</abbr>
170 - * ✗
171 - * [✗](gh-issue:7366)
172 - * ✗
173 - * ✗
174 - * [✗](gh-issue:7366)
177 + * ❌
178 + * [❌](gh-issue:7366)
179 + * ❌
180 + * ❌
181 + * [❌](gh-issue:7366)
182 + * ✅
183 * ✅
184 * ✅
177 - *
185 *
186 *
187 *
@@ -190,9 +197,9 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
197 * ✅
198 * ✅
199 * ✅
193 - * ✗
200 + * ❌
201 + * ✅
202 * ✅
195 - *
203 *
204 *
205 *
@@ -205,12 +212,12 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
212 * ✅
213 * ✅
214 * ✅
208 - * [✗](gh-pr:8199)
215 * ✅
210 - * ✗
216 + * ✅
217 + * ❌
218 + * ✅
219 * ✅
220 * ✅
213 - *
221 *
222 *
223 *
@@ -222,49 +229,49 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
229 * ✅
230 * ✅
231 * ✅
225 - * ✗
232 + * ❌
233 + * ✅
234 + * ❌
235 + * ❌
236 * ✅
227 - * ✗
228 - * ✗
237 * ✅
238 * ✅
231 - *
239 *
240 *
241 *
242 *
243 *
244 - * multi-step
238 - * ✗
245 + * ❌
246 * ✅
240 - * ✗
247 + * ❌
248 + * ✅
249 + * ❌
250 + * ✅
251 + * ❌
252 + * ❌
253 * ✅
242 - * ✗
254 * ✅
244 - * ✗
245 - * ✗
255 * ✅
247 - * [✗](gh-issue:8198)
256 * ✅
249 - *
257 *
258 *
259 *
260 *
261 - * <abbr title="Multimodal Inputs">mm</abbr>
262 * ✅
256 - * [✗](gh-pr:8348)
257 - * [✗](gh-pr:7199)
258 - * ?
259 - * ?
263 + * [🟠](gh-pr:8348)
264 + * [🟠](gh-pr:4194)
265 + * ❔
266 + * ❔
267 * ✅
268 * ✅
269 * ✅
270 * ✅
271 * ✅
272 * ✅
266 - * ?
267 - *
273 + * ❔
274 + * ✅
275 *
276 *
277 *
@@ -273,16 +280,16 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
280 * ✅
281 * ✅
282 * ✅
276 - * [✗](gh-issue:6137)
283 + * [❌](gh-issue:6137)
284 * ✅
278 - * ✗
285 + * ❌
286 * ✅
287 * ✅
288 * ✅
282 - * ?
283 - * [✗](gh-issue:7968)
289 + * ❔
290 + * [❌](gh-issue:7968)
291 + * ✅
292 * ✅
285 - *
293 *
294 *
295 - * beam-search
@@ -290,35 +297,35 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
297 * ✅
298 * ✅
299 * ✅
293 - * [✗](gh-issue:6137)
300 + * [❌](gh-issue:6137)
301 * ✅
295 - * ✗
302 + * ❌
303 * ✅
304 * ✅
305 * ✅
299 - * ?
300 - * [✗](gh-issue:7968)
301 - * ?
306 + * ❔
307 + * [❌](gh-issue:7968)
308 + * ❔
309 + * ✅
310 * ✅
303 - *
311 *
312 - * <abbr title="Guided Decoding">guided dec</abbr>
313 * ✅
314 * ✅
308 - * ?
309 - * ?
310 - * [✗](gh-issue:11484)
315 + * ❔
316 + * ❔
317 + * [❌](gh-issue:11484)
318 * ✅
312 - * ✗
313 - * ?
319 + * ❌
320 + * ❔
321 * ✅
322 * ✅
323 * ✅
317 - * [✗](gh-issue:9893)
318 - * ?
324 + * [❌](gh-issue:9893)
325 + * ❔
326 + * ✅
327 * ✅
328 * ✅
321 - *
329 :::
330
331 (feature-x-hardware)=
@@ -339,7 +346,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
346 * CPU
347 * AMD
348 - * [CP](#chunked-prefill)
342 - * [✗](gh-issue:2729)
349 + * [❌](gh-issue:2729)
350 * ✅
351 * ✅
352 * ✅
@@ -347,7 +354,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
354 * ✅
355 * ✅
356 - * [APC](#automatic-prefix-caching)
350 - * [✗](gh-issue:3687)
357 + * [❌](gh-issue:3687)
358 * ✅
359 * ✅
360 * ✅
@@ -368,7 +375,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
375 * ✅
376 * ✅
377 * ✅
371 - * [✗](gh-issue:8475)
378 + * [❌](gh-issue:8475)
379 * ✅
380 - * [SD](#spec_decode)
381 * ✅
@@ -384,7 +391,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
391 * ✅
392 * ✅
393 * ✅
387 - * ✗
394 + * ❌
395 * ✅
396 - * <abbr title="Pooling Models">pooling</abbr>
397 * ✅
@@ -393,7 +400,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
400 * ✅
401 * ✅
402 * ✅
396 - * ?
403 + * ❔
404 - * <abbr title="Encoder-Decoder Models">enc-dec</abbr>
405 * ✅
406 * ✅
@@ -401,7 +408,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
408 * ✅
409 * ✅
410 * ✅
404 - * ✗
411 + * ❌
412 - * <abbr title="Multimodal Inputs">mm</abbr>
413 * ✅
414 * ✅
@@ -432,15 +439,15 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
439 * ✅
440 * ✅
441 * ✅
435 - * ✗
436 - * ✗
442 + * ❌
443 + * ❌
444 - * multi-step
445 * ✅
446 * ✅
447 * ✅
448 * ✅
449 * ✅
443 - * [✗](gh-issue:8477)
450 + * [❌](gh-issue:8477)
451 * ✅
452 - * best-of
453 * ✅
docs/source/features/quantization/supported_hardware.md
+47 -47
@@ -20,93 +20,93 @@ The table below shows the compatibility of various quantization implementations
20 * AWS Inferentia
21 * Google TPU
22 - * AWQ
23 - * ✗
23 + * ❌
24 * ✅︎
25 * ✅︎
26 * ✅︎
27 * ✅︎
28 - * ✗
28 + * ❌
29 * ✅︎
30 * ✅︎
31 - * ✗
32 - * ✗
31 + * ❌
32 + * ❌
33 - * GPTQ
34 * ✅︎
35 * ✅︎
36 * ✅︎
37 * ✅︎
38 * ✅︎
39 - * ✗
39 + * ❌
40 * ✅︎
41 * ✅︎
42 - * ✗
43 - * ✗
42 + * ❌
43 + * ❌
44 - * Marlin (GPTQ/AWQ/FP8)
45 - * ✗
46 - * ✗
45 + * ❌
46 + * ❌
47 * ✅︎
48 * ✅︎
49 * ✅︎
50 - * ✗
51 - * ✗
52 - * ✗
53 - * ✗
54 - * ✗
50 + * ❌
51 + * ❌
52 + * ❌
53 + * ❌
54 + * ❌
55 - * INT8 (W8A8)
56 - * ✗
56 + * ❌
57 * ✅︎
58 * ✅︎
59 * ✅︎
60 * ✅︎
61 - * ✗
62 - * ✗
61 + * ❌
62 + * ❌
63 * ✅︎
64 - * ✗
65 - * ✗
64 + * ❌
65 + * ❌
66 - * FP8 (W8A8)
67 - * ✗
68 - * ✗
69 - * ✗
67 + * ❌
68 + * ❌
69 + * ❌
70 * ✅︎
71 * ✅︎
72 * ✅︎
73 - * ✗
74 - * ✗
75 - * ✗
76 - * ✗
73 + * ❌
74 + * ❌
75 + * ❌
76 + * ❌
77 - * AQLM
78 * ✅︎
79 * ✅︎
80 * ✅︎
81 * ✅︎
82 * ✅︎
83 - * ✗
84 - * ✗
85 - * ✗
86 - * ✗
87 - * ✗
83 + * ❌
84 + * ❌
85 + * ❌
86 + * ❌
87 + * ❌
88 - * bitsandbytes
89 * ✅︎
90 * ✅︎
91 * ✅︎
92 * ✅︎
93 * ✅︎
94 - * ✗
95 - * ✗
96 - * ✗
97 - * ✗
98 - * ✗
94 + * ❌
95 + * ❌
96 + * ❌
97 + * ❌
98 + * ❌
99 - * DeepSpeedFP
100 * ✅︎
101 * ✅︎
102 * ✅︎
103 * ✅︎
104 * ✅︎
105 - * ✗
106 - * ✗
107 - * ✗
108 - * ✗
109 - * ✗
105 + * ❌
106 + * ❌
107 + * ❌
108 + * ❌
109 + * ❌
110 - * GGUF
111 * ✅︎
112 * ✅︎
@@ -114,16 +114,16 @@ The table below shows the compatibility of various quantization implementations
114 * ✅︎
115 * ✅︎
116 * ✅︎
117 - * ✗
118 - * ✗
119 - * ✗
120 - * ✗
117 + * ❌
118 + * ❌
119 + * ❌
120 + * ❌
121
122 :::
123
124 - Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
125 -- "✅︎" indicates that the quantization method is supported on the specified hardware.
126 -- "✗" indicates that the quantization method is not supported on the specified hardware.
125 +- ✅︎ indicates that the quantization method is supported on the specified hardware.
126 +- ❌ indicates that the quantization method is not supported on the specified hardware.
127
128 :::{note}
129 This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
docs/source/models/pooling_models.md
+4 -4
@@ -28,10 +28,10 @@ The selected option sets the default pooler used to extract the final hidden sta
28 - * Embedding (`embed`)
29 * `LAST`
30 * ✅︎
31 - * ✗
31 + * ❌
32 - * Classification (`classify`)
33 * `LAST`
34 - * ✗
34 + * ❌
35 * ✅︎
36 - * Sentence Pair Scoring (`score`)
37 * \*
@@ -39,8 +39,8 @@ The selected option sets the default pooler used to extract the final hidden sta
39 * \*
40 - * Reward Modeling (`reward`)
41 * `ALL`
42 - * ✗
43 - * ✗
42 + * ❌
43 + * ❌
44 :::
45
46 \*The default pooler is always defined by the model.