A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 23:30 UTC
[ROCm] [Feature] [Doc] [Dockerfile] [BugFix] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing (#12501)
TJian committed
Feb 8, 2025 at 00:13 UTC
eaa92d443743830f9efd35320cf6d440e49283e3
8 files changed
+295
-32
Dockerfile.rocm_base
+1
-1
@@ -6,7 +6,7 @@ ARG RCCL_BRANCH="648a58d"
6
ARG RCCL_REPO="https://github.com/ROCm/rccl"
7
ARG TRITON_BRANCH="e5be006"
8
ARG TRITON_REPO="https://github.com/triton-lang/triton.git"
9
-ARG PYTORCH_BRANCH="8d4926e"
9
+ARG PYTORCH_BRANCH="3a585126"
10
ARG PYTORCH_VISION_BRANCH="v0.19.1"
11
ARG PYTORCH_REPO="https://github.com/pytorch/pytorch.git"
12
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
docs/source/getting_started/installation/gpu/rocm.inc.md
+45
-19
@@ -1,6 +1,6 @@
1
# Installation
2
3
-vLLM supports AMD GPUs with ROCm 6.2.
3
+vLLM supports AMD GPUs with ROCm 6.3.
4
5
:::{attention}
6
There are no pre-built wheels for this device, so you must either use the pre-built Docker image or build vLLM from source.
@@ -9,7 +9,7 @@ There are no pre-built wheels for this device, so you must either use the pre-bu
9
## Requirements
10
11
- GPU: MI200s (gfx90a), MI300 (gfx942), Radeon RX 7900 series (gfx1100)
12
-- ROCm 6.2
12
+- ROCm 6.3
13
14
## Set up using Python
15
@@ -24,9 +24,15 @@ Currently, there are no pre-built ROCm wheels.
24
- [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html)
25
- [PyTorch](https://pytorch.org/)
26
27
- For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm6.2_ubuntu20.04_py3.9_pytorch_release_2.3.0`, `rocm/pytorch-nightly`.
27
+ For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm6.3_ubuntu24.04_py3.12_pytorch_release_2.4.0`, `rocm/pytorch-nightly`. If you are using docker image, you can skip to Step 3.
28
29
- Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/)
29
+ Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:
30
+
31
+ ```console
32
+ # Install PyTorch
33
+ $ pip uninstall torch -y
34
+ $ pip install --no-cache-dir --pre torch --index-url https://download.pytorch.org/whl/rocm6.3
35
+ ```
36
37
1. Install [Triton flash attention for ROCm](https://github.com/ROCm/triton)
38
@@ -37,7 +43,7 @@ Currently, there are no pre-built ROCm wheels.
43
pip uninstall -y triton
44
git clone https://github.com/OpenAI/triton.git
45
cd triton
40
- git checkout e192dba
46
+ git checkout e5be006
47
cd python
48
pip3 install .
49
cd ../..
@@ -49,15 +55,15 @@ Currently, there are no pre-built ROCm wheels.
55
56
2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/ROCm/flash-attention/tree/ck_tile)
57
52
- Install ROCm's flash attention (v2.5.9.post1) following the instructions from [ROCm/flash-attention](https://github.com/ROCm/flash-attention/tree/ck_tile#amd-gpurocm-support)
58
+ Install ROCm's flash attention (v2.7.2) following the instructions from [ROCm/flash-attention](https://github.com/ROCm/flash-attention/tree/ck_tile#amd-gpurocm-support)
59
Alternatively, wheels intended for vLLM use can be accessed under the releases.
60
55
- For example, for ROCm 6.2, suppose your gfx arch is `gfx90a`. To get your gfx architecture, run `rocminfo |grep gfx`.
61
+ For example, for ROCm 6.3, suppose your gfx arch is `gfx90a`. To get your gfx architecture, run `rocminfo |grep gfx`.
62
63
```console
64
git clone https://github.com/ROCm/flash-attention.git
65
cd flash-attention
60
- git checkout 3cea2fb
66
+ git checkout b7d29fb
67
git submodule update --init
68
GPU_ARCHS="gfx90a" python3 setup.py install
69
cd ..
@@ -67,20 +73,16 @@ Currently, there are no pre-built ROCm wheels.
73
You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
74
:::
75
70
-3. Build vLLM. For example, vLLM on ROCM 6.2 can be built with the following steps:
76
+3. Build vLLM. For example, vLLM on ROCM 6.3 can be built with the following steps:
77
78
```bash
79
$ pip install --upgrade pip
80
75
- # Install PyTorch
76
- $ pip uninstall torch -y
77
- $ pip install --no-cache-dir --pre torch --index-url https://download.pytorch.org/whl/rocm6.2
78
-
81
# Build & install AMD SMI
82
$ pip install /opt/rocm/share/amd_smi
83
84
# Install dependencies
83
- $ pip install --upgrade numba scipy huggingface-hub[cli]
85
+ $ pip install --upgrade numba scipy huggingface-hub[cli,hf_transfer] setuptools_scm
86
$ pip install "numpy<2"
87
$ pip install -r requirements-rocm.txt
88
@@ -104,7 +106,7 @@ Currently, there are no pre-built ROCm wheels.
106
For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/workload.html#vllm-performance-optimization).
107
:::
108
107
-## Set up using Docker
109
+## Set up using Docker (Recommended)
110
111
### Pre-built images
112
@@ -120,7 +122,12 @@ for instructions on how to use this prebuilt docker image.
122
123
Building the Docker image from source is the recommended way to use vLLM with ROCm.
124
123
-First, build a docker image from <gh-file:Dockerfile.rocm> and launch a docker container from the image.
125
+#### (Optional) Build an image with ROCm software stack
126
+
127
+Build a docker image from <gh-file:Dockerfile.rocm_base> which setup ROCm software stack needed by the vLLM.
128
+**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
129
+If you choose to build this rocm_base image yourself, the steps are as follows.
130
+
131
It is important that the user kicks off the docker build using buildkit. Either the user put DOCKER_BUILDKIT=1 as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
132
133
```console
@@ -131,7 +138,26 @@ It is important that the user kicks off the docker build using buildkit. Either
138
}
139
```
140
134
-<gh-file:Dockerfile.rocm> uses ROCm 6.2 by default, but also supports ROCm 5.7, 6.0 and 6.1 in older vLLM branches.
141
+To build vllm on ROCm 6.3 for MI200 and MI300 series, you can use the default:
142
+
143
+```console
144
+DOCKER_BUILDKIT=1 docker build -f Dockerfile.rocm_base -t rocm/vllm-dev:base .
145
+```
146
+
147
+#### Build an image with vLLM
148
+
149
+First, build a docker image from <gh-file:Dockerfile.rocm> and launch a docker container from the image.
150
+It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
151
+
152
+```console
153
+{
154
+ "features": {
155
+ "buildkit": true
156
+ }
157
+}
158
+```
159
+
160
+<gh-file:Dockerfile.rocm> uses ROCm 6.3 by default, but also supports ROCm 5.7, 6.0, 6.1, and 6.2, in older vLLM branches.
161
It provides flexibility to customize the build of docker image using the following arguments:
162
163
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using <gh-file:Dockerfile.rocm_base>
@@ -141,13 +167,13 @@ It provides flexibility to customize the build of docker image using the followi
167
168
Their values can be passed in when running `docker build` with `--build-arg` options.
169
144
-To build vllm on ROCm 6.2 for MI200 and MI300 series, you can use the default:
170
+To build vllm on ROCm 6.3 for MI200 and MI300 series, you can use the default:
171
172
```console
173
DOCKER_BUILDKIT=1 docker build -f Dockerfile.rocm -t vllm-rocm .
174
```
175
150
-To build vllm on ROCm 6.2 for Radeon RX7900 series (gfx1100), you should pick the alternative base image:
176
+To build vllm on ROCm 6.3 for Radeon RX7900 series (gfx1100), you should pick the alternative base image:
177
178
```console
179
DOCKER_BUILDKIT=1 docker build --build-arg BASE_IMAGE="rocm/vllm-dev:navi_base" -f Dockerfile.rocm -t vllm-rocm .
tests/quantization/test_fp8.py
+38
-11
@@ -55,10 +55,21 @@ def test_kv_cache_model_load_and_run(vllm_runner, model_id: str):
55
56
assert isinstance(attn.quant_method, Fp8KVCacheMethod)
57
58
- # NOTE: it is valid for scales to be 1.0 (default value), but
59
- # we know these checkpoints have scales < 1.0
60
- assert 0.0 < attn._k_scale < 1.0
61
- assert 0.0 < attn._v_scale < 1.0
58
+ if not current_platform.is_rocm():
59
+ # NOTE: This code path requires validation on Non-CUDA platform
60
+ # NOTE: it is valid for scales to be 1.0 (default value), but
61
+ # we know these checkpoints have scales < 1.0
62
+ assert 0.0 < attn._k_scale < 1.0
63
+ assert 0.0 < attn._v_scale < 1.0
64
+ else:
65
+ # NOTE: This code path is for ROCm platform
66
+ # NOTE: it is valid for scales to be 1.0 (default value), but
67
+ # we know these checkpoints have scales < 1.0
68
+ # However on ROCm platform, the _k_scale and _v_scale will be
69
+ # scaled by a factor of 2 as described in
70
+ # vllm/model_executor/layers/quantization/kv_cache.py
71
+ assert 0.0 < attn._k_scale < (1.0 * 2.0)
72
+ assert 0.0 < attn._v_scale < (1.0 * 2.0)
73
74
llm.apply_model(check_model)
75
@@ -91,13 +102,29 @@ def test_load_fp16_model(vllm_runner, kv_cache_dtype: str, force_marlin: bool,
102
assert attn._k_scale == 1.0
103
assert attn._v_scale == 1.0
104
94
- if current_platform.has_device_capability(89) and not force_marlin:
95
- # For GPUs with hardware support, we keep weights in fp8
96
- assert fc1.weight.dtype == torch.float8_e4m3fn
97
- else:
98
- # For GPUs without hardware support, we pack the fp8 weights
99
- # for weight-only quantization using Marlin kernels
100
- assert fc1.weight.dtype == torch.int32
105
+ if current_platform.is_cuda():
106
+ if current_platform.has_device_capability(
107
+ 89) and not force_marlin:
108
+ # For GPUs with hardware support, we keep weights in fp8
109
+ assert fc1.weight.dtype == torch.float8_e4m3fn
110
+ else:
111
+ # For GPUs without hardware support, we pack the fp8 weights
112
+ # for weight-only quantization using Marlin kernels
113
+ assert fc1.weight.dtype == torch.int32
114
+ elif current_platform.is_rocm():
115
+ # Only MI300 and above support quantization='fp8'
116
+ if current_platform.has_device_capability(
117
+ 94) and not force_marlin:
118
+ # For GPUs with hardware support, we keep weights in fp8
119
+ assert fc1.weight.dtype == torch.float8_e4m3fnuz
120
+ else: # unsupported ROCm platform
121
+ pytest.skip(
122
+ "Skip `test_load_fp16_model`. "
123
+ "It only runs on ROCm platform with FP8 compute."
124
+ " e.g. MI300X and above.")
125
+ else: # unsupported platform
126
+ pytest.skip("Skip `test_load_fp16_model`. "
127
+ "It only runs on CUDA and ROCm platform.")
128
129
llm.apply_model(check_model)
130
tests/quantization/test_ptpc_fp8.py
new
+55
@@ -0,0 +1,55 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+"""Tests whether PTPC w8a8 FP8 computation is enabled correctly.
3
+
4
+Run `pytest tests/quantization/test_ptpc_fp8.py --forked`.
5
+"""
6
+import pytest
7
+import torch
8
+
9
+from tests.quantization.utils import is_quant_method_supported
10
+from vllm.model_executor.layers.quantization.fp8 import Fp8KVCacheMethod
11
+from vllm.model_executor.layers.quantization.ptpc_fp8 import (
12
+ PTPCFp8LinearMethod)
13
+from vllm.platforms import current_platform
14
+
15
+
16
+@pytest.mark.skipif(not is_quant_method_supported("ptpc_fp8"),
17
+ reason="PTPC FP8 is not supported on this GPU type.")
18
+@pytest.mark.skipif(not current_platform.is_rocm(),
19
+ reason="This test is for ROCm GPU.")
20
+@pytest.mark.parametrize("dtype", ["auto", "bfloat16", "float16"])
21
+@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8", "fp8_e4m3"])
22
+def test_ptpc_fp8_rocm(vllm_runner, dtype: str, kv_cache_dtype: str) -> None:
23
+
24
+ try:
25
+ with vllm_runner("facebook/opt-125m",
26
+ dtype=dtype,
27
+ quantization="ptpc_fp8",
28
+ kv_cache_dtype=kv_cache_dtype) as llm:
29
+
30
+ model = llm.model.llm_engine.model_executor.driver_worker.model_runner.model # noqa: E501
31
+ fc1 = model.model.decoder.layers[0].fc1
32
+ assert isinstance(fc1.quant_method, PTPCFp8LinearMethod)
33
+ if kv_cache_dtype == "ptpc_fp8":
34
+ attn = model.model.decoder.layers[0].self_attn.attn
35
+ assert isinstance(attn.quant_method, Fp8KVCacheMethod)
36
+ assert attn._k_scale == 1.0
37
+ assert attn._v_scale == 1.0
38
+
39
+ if current_platform.has_device_capability(94):
40
+ # For GPUs with hardware support, we keep weights in fp8
41
+ assert fc1.weight.dtype == torch.float8_e4m3fnuz
42
+ else:
43
+ pytest.skip()
44
+
45
+ output = llm.generate_greedy("Hello my name is", max_tokens=20)
46
+ assert output
47
+ except AssertionError as e:
48
+ if str(
49
+ e
50
+ ) == "Currently torch._scaled_mm (hipBLASLt) rowwise gemm only support output dtype of bfloat16. torch.float16 is specified.": # noqa: E501
51
+ # If the error message matches, the test passes
52
+ pass
53
+ else:
54
+ # If the error message does not match, re-raise the exception
55
+ raise
vllm/model_executor/layers/quantization/__init__.py
+3
@@ -11,6 +11,7 @@ QUANTIZATION_METHODS: List[str] = [
11
"deepspeedfp",
12
"tpu_int8",
13
"fp8",
14
+ "ptpc_fp8",
15
"fbgemm_fp8",
16
"modelopt",
17
# The order of gptq methods is important for config.py iteration over
@@ -99,6 +100,7 @@ def get_quantization_config(quantization: str) -> Type[QuantizationConfig]:
100
from .modelopt import ModelOptFp8Config
101
from .moe_wna16 import MoeWNA16Config
102
from .neuron_quant import NeuronQuantConfig
103
+ from .ptpc_fp8 import PTPCFp8Config
104
from .qqq import QQQConfig
105
from .tpu_int8 import Int8TpuConfig
106
@@ -120,6 +122,7 @@ def get_quantization_config(quantization: str) -> Type[QuantizationConfig]:
122
"gptq": GPTQConfig,
123
"compressed-tensors": CompressedTensorsConfig,
124
"bitsandbytes": BitsAndBytesConfig,
125
+ "ptpc_fp8": PTPCFp8Config,
126
"qqq": QQQConfig,
127
"hqq": HQQMarlinConfig,
128
"experts_int8": ExpertsInt8Config,
vllm/model_executor/layers/quantization/ptpc_fp8.py
new
+125
@@ -0,0 +1,125 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+
3
+from typing import Any, Dict, List, Optional
4
+
5
+import torch
6
+from torch.nn.parameter import Parameter
7
+
8
+from vllm import _custom_ops as ops
9
+from vllm.logger import init_logger
10
+from vllm.model_executor.layers.linear import (LinearBase,
11
+ UnquantizedLinearMethod)
12
+from vllm.model_executor.layers.quantization.base_config import (
13
+ QuantizeMethodBase)
14
+from vllm.model_executor.layers.quantization.fp8 import (Fp8Config,
15
+ Fp8KVCacheMethod,
16
+ Fp8LinearMethod)
17
+from vllm.model_executor.layers.quantization.utils.quant_utils import (
18
+ is_layer_skipped)
19
+from vllm.model_executor.layers.quantization.utils.w8a8_utils import (
20
+ apply_fp8_linear)
21
+from vllm.platforms import current_platform
22
+
23
+ACTIVATION_SCHEMES = ["static", "dynamic"]
24
+
25
+logger = init_logger(__name__)
26
+
27
+
28
+class PTPCFp8Config(Fp8Config):
29
+ """Config class for Per-Token-Per-Channel Dynamic Quantization Fp8."""
30
+
31
+ def __init__(
32
+ self,
33
+ activation_scheme: str = "dynamic",
34
+ ignored_layers: Optional[List[str]] = None,
35
+ ) -> None:
36
+ if not current_platform.is_rocm():
37
+ raise ValueError(
38
+ "ptpc_fp8 quantization is supported only on ROCm.")
39
+
40
+ if not current_platform.has_device_capability(94):
41
+ raise ValueError(
42
+ "ptpc_fp8 quantization is supported only on AMD Instinct MI300 GPUs and newer." # noqa: E501
43
+ )
44
+ if activation_scheme == "static":
45
+ raise ValueError(
46
+ "ptpc_fp8 as of now only support dynamic quantization.")
47
+
48
+ super().__init__(is_checkpoint_fp8_serialized=False,
49
+ activation_scheme=activation_scheme,
50
+ ignored_layers=ignored_layers)
51
+
52
+ @classmethod
53
+ def get_name(cls) -> str:
54
+ return "ptpc_fp8"
55
+
56
+ @classmethod
57
+ def from_config(cls, config: Dict[str, Any]) -> "PTPCFp8Config":
58
+ activation_scheme = cls.get_from_keys(config, ["activation_scheme"])
59
+ ignored_layers = cls.get_from_keys_or(config, ["ignored_layers"], None)
60
+ return cls(activation_scheme=activation_scheme,
61
+ ignored_layers=ignored_layers)
62
+
63
+ def get_quant_method(self, layer: torch.nn.Module,
64
+ prefix: str) -> Optional["QuantizeMethodBase"]:
65
+ from vllm.attention.layer import Attention # Avoid circular import
66
+
67
+ if isinstance(layer, LinearBase):
68
+ if is_layer_skipped(prefix, self.ignored_layers):
69
+ return UnquantizedLinearMethod()
70
+ return PTPCFp8LinearMethod(self)
71
+ elif isinstance(layer, Attention):
72
+ return Fp8KVCacheMethod(self)
73
+ return None
74
+
75
+
76
+class PTPCFp8LinearMethod(Fp8LinearMethod):
77
+ """Linear method for Per-Token and Per-Channel FP8 Quantization.
78
+ Only supports loading quantized BF16 model checkpoints with dynamic
79
+ activation scaling. To load FP16 model checkpoints, user must specify
80
+ to convert the FP16 model weight loading into BF16.
81
+ The weight scaling factor will be initialized after
82
+ the model weights are loaded.
83
+
84
+ Limitations:
85
+ 1. Only support float8_e4m3fnuz data type due to the limitation of
86
+ torch._scaled_mm (https://github.com/ROCm/pytorch/blob/8c0504d7f3fb0ee4c278c096a5c3caedb01129fa/aten/src/ATen/native/cuda/Blas.cpp#L1041)
87
+
88
+ Args:
89
+ quant_config: The quantization config.
90
+ """
91
+
92
+ def __init__(self, quant_config: PTPCFp8Config):
93
+ super().__init__(quant_config=quant_config)
94
+ # Force weight quantization
95
+ self.quant_config.is_checkpoint_fp8_serialized = False
96
+
97
+ def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
98
+ layer.weight = torch.nn.Parameter(layer.weight.data,
99
+ requires_grad=False)
100
+
101
+ assert layer.weight.data.dtype == torch.bfloat16, \
102
+ f"Currently torch._scaled_mm (hipBLASLt) rowwise gemm only support output dtype of bfloat16. {str(layer.weight.data.dtype)} is specified." # noqa: E501
103
+ # Quantize the weights.
104
+ qweight, weight_scale = ops.scaled_fp8_quant(
105
+ layer.weight, scale=None, use_per_token_if_dynamic=True)
106
+
107
+ # Update the layer with the new values.
108
+ layer.weight = Parameter(
109
+ qweight.t(), requires_grad=False) # Pretranspose the weight
110
+ layer.weight_scale = Parameter(weight_scale, requires_grad=False)
111
+ layer.input_scale = None
112
+
113
+ def apply(self,
114
+ layer: torch.nn.Module,
115
+ x: torch.Tensor,
116
+ bias: Optional[torch.Tensor] = None) -> torch.Tensor:
117
+
118
+ return apply_fp8_linear(input=x,
119
+ weight=layer.weight,
120
+ weight_scale=layer.weight_scale,
121
+ input_scale=None,
122
+ input_scale_ub=None,
123
+ bias=bias,
124
+ cutlass_fp8_supported=False,
125
+ use_per_token_if_dynamic=True)
vllm/model_executor/layers/quantization/utils/w8a8_utils.py
+27
@@ -11,6 +11,13 @@ from vllm.platforms import current_platform
11
# from pytorch 2.5. Allocating a dummy tensor to pass as input_scale
12
TORCH_DEVICE_IDENTITY = torch.ones(1, dtype=torch.float32)
13
14
+# The condition to determine if it is on a platform that supports
15
+# torch._scaled_mm rowwise feature.
16
+# The condition is determined once as the operations
17
+# are time consuming.
18
+USE_ROWWISE_TORCH_SCALED_MM = (current_platform.is_rocm()
19
+ and current_platform.has_device_capability(94))
20
+
21
22
def sparse_cutlass_supported() -> bool:
23
if not current_platform.is_cuda():
@@ -172,6 +179,26 @@ def apply_fp8_linear(
179
return torch.narrow(output, 0, 0,
180
input_2d.shape[0]).view(*output_shape)
181
182
+ elif (use_per_token_if_dynamic and not per_tensor_weights
183
+ and not per_tensor_activations and USE_ROWWISE_TORCH_SCALED_MM):
184
+ # For now validated on ROCm platform
185
+ # fp8 rowwise scaling in torch._scaled_mm is introduced in
186
+ # https://github.com/pytorch/pytorch/pull/144432 using
187
+ # hipBLASLt and ROCm 6.3, which only exists in torch 2.7 and above.
188
+ # For CUDA platform please validate if the
189
+ # torch._scaled_mm support rowwise scaled GEMM
190
+ # Fused GEMM_DQ Rowwise GEMM
191
+ output = torch._scaled_mm(qinput,
192
+ weight,
193
+ out_dtype=input.dtype,
194
+ scale_a=x_scale,
195
+ scale_b=weight_scale.t(),
196
+ bias=bias)
197
+
198
+ output = torch.narrow(output, 0, 0, input_2d.shape[0])
199
+ output = output.view(*output_shape)
200
+ return output
201
+
202
else:
203
# Fallback for channelwise case, where we use unfused DQ
204
# due to limitations with scaled_mm
vllm/platforms/rocm.py
+1
-1
@@ -72,7 +72,7 @@ class RocmPlatform(Platform):
72
73
supported_quantization: list[str] = [
74
"awq", "gptq", "fp8", "compressed_tensors", "compressed-tensors",
75
- "fbgemm_fp8", "gguf", "quark"
75
+ "fbgemm_fp8", "gguf", "quark", "ptpc_fp8"
76
]
77
78
@classmethod