@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 379da6dcb5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 01:45 UTC

[Kernel] [FP8] Improve FP8 linear layer performance (#4691)

This PR improves the FP8 performance of linear layers, which had been lacking before (#4118 (comment) and #4118 (comment)). We noticed that CUBLASLt can find a better algorithm if the first dimension of the matrix is greater than 16. So this PR enlarges matrices appropriately during quantization. This improves FP8 performance and removes the performance regression vs. FP16, in many cases exceeding FP16 performance. Here are benchmarks on llama3 70b (ITL numbers for 1000 input and 50 output tokens at fixed qps and at TP 4), all FP8 measurements are for dynamic quantization: qps = 1: 24 ms (FP8, this PR), 32 ms (FP8, previous main), 26 ms (FP16) qps = 2: 26 ms (FP8, this PR), 34ms (FP8, previous main), 28 ms (FP16) qps = 4: 33 ms (FP8, this PR), 44 ms (FP8, previous main), 36 ms (FP16) qps = 6: 46 ms (FP8, this PR), 56 ms (FP8, previous main), 54 ms (FP16) qps = 8: 85 ms (FP8, this PR), 85 ms (FP8, previous main), 138 ms (FP16)

Philipp Moritz committed May 9, 2024 at 16:38 UTC 379da6dcb5f5d062d0452b2fc23291e5113dcf04
2 files changed +36 -5
vllm/_custom_ops.py
+27 -1
@@ -189,8 +189,34 @@ def gptq_marlin_gemm(a: torch.Tensor, b_q_weight: torch.Tensor,
189 def scaled_fp8_quant(
190 input: torch.Tensor,
191 scale: Optional[torch.Tensor] = None,
192 + batch_dim_padding: Optional[int] = None,
193 ) -> Tuple[torch.Tensor, torch.Tensor]:
193 - output = torch.empty_like(input, dtype=torch.float8_e4m3fn)
194 + """
195 + Quantize input tensor to FP8 and return quantized tensor and scale.
196 +
197 + This function supports both static and dynamic quantization: If you
198 + provide the scale, it will use static scaling and if you omit it,
199 + the scale will be determined dynamically. The function also allows
200 + optional padding of the output tensor for downstream kernels that
201 + will benefit from padding.
202 +
203 + Args:
204 + input: The input tensor to be quantized to FP8
205 + scale: Optional scaling factor for the FP8 quantization
206 + batch_dim_padding: If specified, pad the first dimension
207 + of the output to at least this value.
208 +
209 + Returns:
210 + Tuple[torch.Tensor, torch.Tensor]: The output tensor in FP8 and
211 + scaling factor.
212 + """
213 + if batch_dim_padding:
214 + shape = (max(batch_dim_padding, input.shape[0]), *input.shape[1:])
215 + output = torch.empty(shape,
216 + device=input.device,
217 + dtype=torch.float8_e4m3fn)
218 + else:
219 + output = torch.empty_like(input, dtype=torch.float8_e4m3fn)
220 if scale is None:
221 scale = torch.zeros(1, device=input.device, dtype=torch.float32)
222 vllm_ops.dynamic_scaled_fp8_quant(output, input, scale)
vllm/model_executor/layers/quantization/fp8.py
+9 -4
@@ -231,9 +231,14 @@ class Fp8LinearMethod(LinearMethodBase):
231 # ops.scaled_fp8_quant supports both dynamic and static quant.
232 # If dynamic, layer.act_scale is None and x_scale computed from x.
233 # If static, layer.act_scale is scalar and x_scale set to act_scale.
234 - qinput, x_scale = ops.scaled_fp8_quant(x, layer.act_scale)
235 -
236 - # Fused GEMM_DQ
234 + qinput, x_scale = ops.scaled_fp8_quant(x,
235 + layer.act_scale,
236 + batch_dim_padding=17)
237 +
238 + # Fused GEMM_DQ -- note we padded the input above because
239 + # torch._scaled_mm is more performant for matrices with
240 + # batch dimension > 16. Note that this could change
241 + # in the future.
242 output, _ = torch._scaled_mm(
243 qinput,
244 layer.weight,
@@ -243,7 +248,7 @@ class Fp8LinearMethod(LinearMethodBase):
248 bias=bias,
249 )
250
246 - return output
251 + return torch.narrow(output, 0, 0, x.shape[0])
252
253
254 def all_close_1d(x: torch.Tensor) -> bool: