A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 06:00 UTC
[doc] split "Other AI Accelerators" tabs (#19708)
David Xia committed
Jun 17, 2025 at 09:05 UTC
93aee29fdb81fa51d5d44f34ea84df80d0764235
6 files changed
+69
-217
docs/getting_started/installation/.nav.yml
+3
-1
@@ -2,4 +2,6 @@ nav:
2
- README.md
3
- gpu.md
4
- cpu.md
5
- - ai_accelerator.md
\ No newline at end of file
5
+ - google_tpu.md
6
+ - intel_gaudi.md
7
+ - aws_neuron.md
docs/getting_started/installation/README.md
+3
-4
@@ -14,7 +14,6 @@ vLLM supports the following hardware platforms:
14
- [ARM AArch64](cpu.md#arm-aarch64)
15
- [Apple silicon](cpu.md#apple-silicon)
16
- [IBM Z (S390X)](cpu.md#ibm-z-s390x)
17
-- [Other AI accelerators](ai_accelerator.md)
18
- - [Google TPU](ai_accelerator.md#google-tpu)
19
- - [Intel Gaudi](ai_accelerator.md#intel-gaudi)
20
- - [AWS Neuron](ai_accelerator.md#aws-neuron)
17
+- [Google TPU](google_tpu.md)
18
+- [Intel Gaudi](intel_gaudi.md)
19
+- [AWS Neuron](aws_neuron.md)
docs/getting_started/installation/ai_accelerator.md
deleted
-117
@@ -1,117 +0,0 @@
1
-# Other AI accelerators
2
-
3
-vLLM is a Python library that supports the following AI accelerators. Select your AI accelerator type to see vendor specific instructions:
4
-
5
-=== "Google TPU"
6
-
7
- --8<-- "docs/getting_started/installation/ai_accelerator/tpu.inc.md:installation"
8
-
9
-=== "Intel Gaudi"
10
-
11
- --8<-- "docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md:installation"
12
-
13
-=== "AWS Neuron"
14
-
15
- --8<-- "docs/getting_started/installation/ai_accelerator/neuron.inc.md:installation"
16
-
17
-## Requirements
18
-
19
-=== "Google TPU"
20
-
21
- --8<-- "docs/getting_started/installation/ai_accelerator/tpu.inc.md:requirements"
22
-
23
-=== "Intel Gaudi"
24
-
25
- --8<-- "docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md:requirements"
26
-
27
-=== "AWS Neuron"
28
-
29
- --8<-- "docs/getting_started/installation/ai_accelerator/neuron.inc.md:requirements"
30
-
31
-## Configure a new environment
32
-
33
-=== "Google TPU"
34
-
35
- --8<-- "docs/getting_started/installation/ai_accelerator/tpu.inc.md:configure-a-new-environment"
36
-
37
-=== "Intel Gaudi"
38
-
39
- --8<-- "docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md:configure-a-new-environment"
40
-
41
-=== "AWS Neuron"
42
-
43
- --8<-- "docs/getting_started/installation/ai_accelerator/neuron.inc.md:configure-a-new-environment"
44
-
45
-## Set up using Python
46
-
47
-### Pre-built wheels
48
-
49
-=== "Google TPU"
50
-
51
- --8<-- "docs/getting_started/installation/ai_accelerator/tpu.inc.md:pre-built-wheels"
52
-
53
-=== "Intel Gaudi"
54
-
55
- --8<-- "docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md:pre-built-wheels"
56
-
57
-=== "AWS Neuron"
58
-
59
- --8<-- "docs/getting_started/installation/ai_accelerator/neuron.inc.md:pre-built-wheels"
60
-
61
-### Build wheel from source
62
-
63
-=== "Google TPU"
64
-
65
- --8<-- "docs/getting_started/installation/ai_accelerator/tpu.inc.md:build-wheel-from-source"
66
-
67
-=== "Intel Gaudi"
68
-
69
- --8<-- "docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md:build-wheel-from-source"
70
-
71
-=== "AWS Neuron"
72
-
73
- --8<-- "docs/getting_started/installation/ai_accelerator/neuron.inc.md:build-wheel-from-source"
74
-
75
-## Set up using Docker
76
-
77
-### Pre-built images
78
-
79
-=== "Google TPU"
80
-
81
- --8<-- "docs/getting_started/installation/ai_accelerator/tpu.inc.md:pre-built-images"
82
-
83
-=== "Intel Gaudi"
84
-
85
- --8<-- "docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md:pre-built-images"
86
-
87
-=== "AWS Neuron"
88
-
89
- --8<-- "docs/getting_started/installation/ai_accelerator/neuron.inc.md:pre-built-images"
90
-
91
-### Build image from source
92
-
93
-=== "Google TPU"
94
-
95
- --8<-- "docs/getting_started/installation/ai_accelerator/tpu.inc.md:build-image-from-source"
96
-
97
-=== "Intel Gaudi"
98
-
99
- --8<-- "docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md:build-image-from-source"
100
-
101
-=== "AWS Neuron"
102
-
103
- --8<-- "docs/getting_started/installation/ai_accelerator/neuron.inc.md:build-image-from-source"
104
-
105
-## Extra information
106
-
107
-=== "Google TPU"
108
-
109
- --8<-- "docs/getting_started/installation/ai_accelerator/tpu.inc.md:extra-information"
110
-
111
-=== "Intel Gaudi"
112
-
113
- --8<-- "docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md:extra-information"
114
-
115
-=== "AWS Neuron"
116
-
117
- --8<-- "docs/getting_started/installation/ai_accelerator/neuron.inc.md:extra-information"
docs/getting_started/installation/aws_neuron.md
renamed
+38
-46
@@ -1,15 +1,14 @@
1
-# --8<-- [start:installation]
1
+# AWS Neuron
2
3
[AWS Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/) is the software development kit (SDK) used to run deep learning and
4
- generative AI workloads on AWS Inferentia and AWS Trainium powered Amazon EC2 instances and UltraServers (Inf1, Inf2, Trn1, Trn2,
5
- and Trn2 UltraServer). Both Trainium and Inferentia are powered by fully-independent heterogeneous compute-units called NeuronCores.
6
- This tab describes how to set up your environment to run vLLM on Neuron.
4
+generative AI workloads on AWS Inferentia and AWS Trainium powered Amazon EC2 instances and UltraServers (Inf1, Inf2, Trn1, Trn2,
5
+and Trn2 UltraServer). Both Trainium and Inferentia are powered by fully-independent heterogeneous compute-units called NeuronCores.
6
+This describes how to set up your environment to run vLLM on Neuron.
7
8
!!! warning
9
There are no pre-built wheels or images for this device, so you must build vLLM from source.
10
11
-# --8<-- [end:installation]
12
-# --8<-- [start:requirements]
11
+## Requirements
12
13
- OS: Linux
14
- Python: 3.9 or newer
@@ -17,8 +16,7 @@
16
- Accelerator: NeuronCore-v2 (in trn1/inf2 chips) or NeuronCore-v3 (in trn2 chips)
17
- AWS Neuron SDK 2.23
18
20
-# --8<-- [end:requirements]
21
-# --8<-- [start:configure-a-new-environment]
19
+## Configure a new environment
20
21
### Launch a Trn1/Trn2/Inf2 instance and verify Neuron dependencies
22
@@ -27,6 +25,7 @@ The easiest way to launch a Trainium or Inferentia instance with pre-installed N
25
26
- After launching the instance, follow the instructions in [Connect to your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html) to connect to the instance
27
- Once inside your instance, activate the pre-installed virtual environment for inference by running
28
+
29
```console
30
source /opt/aws_neuronx_venv_pytorch_2_6_nxd_inference/bin/activate
31
```
@@ -38,20 +37,15 @@ for alternative setup instructions including using Docker and manually installin
37
NxD Inference is the default recommended backend to run inference on Neuron. If you are looking to use the legacy [transformers-neuronx](https://github.com/aws-neuron/transformers-neuronx)
38
library, refer to [Transformers NeuronX Setup](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/transformers-neuronx/setup/index.html).
39
41
-# --8<-- [end:configure-a-new-environment]
42
-# --8<-- [start:set-up-using-python]
40
+## Set up using Python
41
44
-# --8<-- [end:set-up-using-python]
45
-# --8<-- [start:pre-built-wheels]
42
+### Pre-built wheels
43
44
Currently, there are no pre-built Neuron wheels.
45
49
-# --8<-- [end:pre-built-wheels]
50
-# --8<-- [start:build-wheel-from-source]
51
-
52
-#### Install vLLM from source
46
+### Build wheel from source
47
54
-Install vllm as follows:
48
+To build and install vLLM from source, run:
49
50
```console
51
git clone https://github.com/vllm-project/vllm.git
@@ -61,8 +55,8 @@ VLLM_TARGET_DEVICE="neuron" pip install -e .
55
```
56
57
AWS Neuron maintains a [Github fork of vLLM](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2) at
64
- [https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2), which contains several features in addition to what's
65
- available on vLLM V0. Please utilize the AWS Fork for the following features:
58
+<https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2>, which contains several features in addition to what's
59
+available on vLLM V0. Please utilize the AWS Fork for the following features:
60
61
- Llama-3.2 multi-modal support
62
- Multi-node distributed inference
@@ -81,25 +75,22 @@ VLLM_TARGET_DEVICE="neuron" pip install -e .
75
76
Note that the AWS Neuron fork is only intended to support Neuron hardware; compatibility with other hardwares is not tested.
77
84
-# --8<-- [end:build-wheel-from-source]
85
-# --8<-- [start:set-up-using-docker]
78
+## Set up using Docker
79
87
-# --8<-- [end:set-up-using-docker]
88
-# --8<-- [start:pre-built-images]
80
+### Pre-built images
81
82
Currently, there are no pre-built Neuron images.
83
92
-# --8<-- [end:pre-built-images]
93
-# --8<-- [start:build-image-from-source]
84
+### Build image from source
85
86
See [deployment-docker-build-image-from-source][deployment-docker-build-image-from-source] for instructions on building the Docker image.
87
88
Make sure to use <gh-file:docker/Dockerfile.neuron> in place of the default Dockerfile.
89
99
-# --8<-- [end:build-image-from-source]
100
-# --8<-- [start:extra-information]
90
+## Extra information
91
92
[](){ #feature-support-through-nxd-inference-backend }
93
+
94
### Feature support through NxD Inference backend
95
96
The current vLLM and Neuron integration relies on either the `neuronx-distributed-inference` (preferred) or `transformers-neuronx` backend
@@ -108,12 +99,15 @@ to perform most of the heavy lifting which includes PyTorch model initialization
99
100
To configure NxD Inference features through the vLLM entrypoint, use the `override_neuron_config` setting. Provide the configs you want to override
101
as a dictionary (or JSON object when starting vLLM from the CLI). For example, to disable auto bucketing, include
102
+
103
```console
104
override_neuron_config={
105
"enable_bucketing":False,
106
}
107
```
108
+
109
or when launching vLLM from the CLI, pass
110
+
111
```console
112
--override-neuron-config "{\"enable_bucketing\":false}"
113
```
@@ -124,32 +118,30 @@ Alternatively, users can directly call the NxDI library to trace and compile you
118
### Known limitations
119
120
- EAGLE speculative decoding: NxD Inference requires the EAGLE draft checkpoint to include the LM head weights from the target model. Refer to this
127
- [guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/feature-guide.html#eagle-checkpoint-compatibility)
128
- for how to convert pretrained EAGLE model checkpoints to be compatible for NxDI.
121
+ [guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/feature-guide.html#eagle-checkpoint-compatibility)
122
+ for how to convert pretrained EAGLE model checkpoints to be compatible for NxDI.
123
- Quantization: the native quantization flow in vLLM is not well supported on NxD Inference. It is recommended to follow this
130
- [Neuron quantization guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/custom-quantization.html)
131
- to quantize and compile your model using NxD Inference, and then load the compiled artifacts into vLLM.
124
+ [Neuron quantization guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/custom-quantization.html)
125
+ to quantize and compile your model using NxD Inference, and then load the compiled artifacts into vLLM.
126
- Multi-LoRA serving: NxD Inference only supports loading of LoRA adapters at server startup. Dynamic loading of LoRA adapters at
133
- runtime is not currently supported. Refer to [multi-lora example](https://github.com/aws-neuron/upstreaming-to-vllm/blob/neuron-2.23-vllm-v0.7.2/examples/offline_inference/neuron_multi_lora.py)
127
+ runtime is not currently supported. Refer to [multi-lora example](https://github.com/aws-neuron/upstreaming-to-vllm/blob/neuron-2.23-vllm-v0.7.2/examples/offline_inference/neuron_multi_lora.py)
128
- Multi-modal support: multi-modal support is only available through the AWS Neuron fork. This feature has not been upstreamed
135
- to vLLM main because NxD Inference currently relies on certain adaptations to the core vLLM logic to support this feature.
129
+ to vLLM main because NxD Inference currently relies on certain adaptations to the core vLLM logic to support this feature.
130
- Multi-node support: distributed inference across multiple Trainium/Inferentia instances is only supported on the AWS Neuron fork. Refer
137
- to this [multi-node example](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2/examples/neuron/multi_node)
138
- to run. Note that tensor parallelism (distributed inference across NeuronCores) is available in vLLM main.
131
+ to this [multi-node example](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2/examples/neuron/multi_node)
132
+ to run. Note that tensor parallelism (distributed inference across NeuronCores) is available in vLLM main.
133
- Known edge case bug in speculative decoding: An edge case failure may occur in speculative decoding when sequence length approaches
140
- max model length (e.g. when requesting max tokens up to the max model length and ignoring eos). In this scenario, vLLM may attempt
141
- to allocate an additional block to ensure there is enough memory for number of lookahead slots, but since we do not have good support
142
- for paged attention, there isn't another Neuron block for vLLM to allocate. A workaround fix (to terminate 1 iteration early) is
143
- implemented in the AWS Neuron fork but is not upstreamed to vLLM main as it modifies core vLLM logic.
144
-
134
+ max model length (e.g. when requesting max tokens up to the max model length and ignoring eos). In this scenario, vLLM may attempt
135
+ to allocate an additional block to ensure there is enough memory for number of lookahead slots, but since we do not have good support
136
+ for paged attention, there isn't another Neuron block for vLLM to allocate. A workaround fix (to terminate 1 iteration early) is
137
+ implemented in the AWS Neuron fork but is not upstreamed to vLLM main as it modifies core vLLM logic.
138
139
### Environment variables
140
+
141
- `NEURON_COMPILED_ARTIFACTS`: set this environment variable to point to your pre-compiled model artifacts directory to avoid
148
- compilation time upon server initialization. If this variable is not set, the Neuron module will perform compilation and save the
149
- artifacts under `neuron-compiled-artifacts/{unique_hash}/` sub-directory in the model path. If this environment variable is set,
150
- but the directory does not exist, or the contents are invalid, Neuron will also fallback to a new compilation and store the artifacts
151
- under this specified path.
142
+ compilation time upon server initialization. If this variable is not set, the Neuron module will perform compilation and save the
143
+ artifacts under `neuron-compiled-artifacts/{unique_hash}/` sub-directory in the model path. If this environment variable is set,
144
+ but the directory does not exist, or the contents are invalid, Neuron will also fallback to a new compilation and store the artifacts
145
+ under this specified path.
146
- `NEURON_CONTEXT_LENGTH_BUCKETS`: Bucket sizes for context encoding. (Only applicable to `transformers-neuronx` backend).
147
- `NEURON_TOKEN_GEN_BUCKETS`: Bucket sizes for token generation. (Only applicable to `transformers-neuronx` backend).
154
-
155
-# --8<-- [end:extra-information]
docs/getting_started/installation/google_tpu.md
renamed
+11
-25
@@ -1,4 +1,4 @@
1
-# --8<-- [start:installation]
1
+# Google TPU
2
3
Tensor Processing Units (TPUs) are Google's custom-developed application-specific
4
integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs
@@ -33,8 +33,7 @@ information, see [Storage options for Cloud TPU data](https://cloud.devsite.corp
33
!!! warning
34
There are no pre-built wheels for this device, so you must either use the pre-built Docker image or build vLLM from source.
35
36
-# --8<-- [end:installation]
37
-# --8<-- [start:requirements]
36
+## Requirements
37
38
- Google Cloud TPU VM
39
- TPU versions: v6e, v5e, v5p, v4
@@ -63,8 +62,7 @@ For more information about using TPUs with GKE, see:
62
- <https://cloud.google.com/kubernetes-engine/docs/concepts/tpus>
63
- <https://cloud.google.com/kubernetes-engine/docs/concepts/plan-tpus>
64
66
-# --8<-- [end:requirements]
67
-# --8<-- [start:configure-a-new-environment]
65
+## Configure a new environment
66
67
### Provision a Cloud TPU with the queued resource API
68
@@ -100,16 +98,13 @@ gcloud compute tpus tpu-vm ssh TPU_NAME --project PROJECT_ID --zone ZONE
98
[TPU VM images]: https://cloud.google.com/tpu/docs/runtimes
99
[TPU regions and zones]: https://cloud.google.com/tpu/docs/regions-zones
100
103
-# --8<-- [end:configure-a-new-environment]
104
-# --8<-- [start:set-up-using-python]
101
+## Set up using Python
102
106
-# --8<-- [end:set-up-using-python]
107
-# --8<-- [start:pre-built-wheels]
103
+### Pre-built wheels
104
105
Currently, there are no pre-built TPU wheels.
106
111
-# --8<-- [end:pre-built-wheels]
112
-# --8<-- [start:build-wheel-from-source]
107
+### Build wheel from source
108
109
Install Miniconda:
110
@@ -142,7 +137,7 @@ Install build dependencies:
137
138
```bash
139
pip install -r requirements/tpu.txt
145
-sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev
140
+sudo apt-get install --no-install-recommends --yes libopenblas-base libopenmpi-dev libomp-dev
141
```
142
143
Run the setup script:
@@ -151,16 +146,13 @@ Run the setup script:
146
VLLM_TARGET_DEVICE="tpu" python -m pip install -e .
147
```
148
154
-# --8<-- [end:build-wheel-from-source]
155
-# --8<-- [start:set-up-using-docker]
149
+## Set up using Docker
150
157
-# --8<-- [end:set-up-using-docker]
158
-# --8<-- [start:pre-built-images]
151
+### Pre-built images
152
153
See [deployment-docker-pre-built-image][deployment-docker-pre-built-image] for instructions on using the official Docker image, making sure to substitute the image name `vllm/vllm-openai` with `vllm/vllm-tpu`.
154
162
-# --8<-- [end:pre-built-images]
163
-# --8<-- [start:build-image-from-source]
155
+### Build image from source
156
157
You can use <gh-file:docker/Dockerfile.tpu> to build a Docker image with TPU support.
158
@@ -194,11 +186,5 @@ docker run --privileged --net host --shm-size=16G -it vllm-tpu
186
Install OpenBLAS with the following command:
187
188
```console
197
- sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev
189
+ sudo apt-get install --no-install-recommends --yes libopenblas-base libopenmpi-dev libomp-dev
190
```
199
-
200
-# --8<-- [end:build-image-from-source]
201
-# --8<-- [start:extra-information]
202
-
203
-There is no extra information for this device.
204
-# --8<-- [end:extra-information]
docs/getting_started/installation/intel_gaudi.md
renamed
+14
-24
@@ -1,12 +1,11 @@
1
-# --8<-- [start:installation]
1
+# Intel Gaudi
2
3
-This tab provides instructions on running vLLM with Intel Gaudi devices.
3
+This page provides instructions on running vLLM with Intel Gaudi devices.
4
5
!!! warning
6
There are no pre-built wheels or images for this device, so you must build vLLM from source.
7
8
-# --8<-- [end:installation]
9
-# --8<-- [start:requirements]
8
+## Requirements
9
10
- OS: Ubuntu 22.04 LTS
11
- Python: 3.10
@@ -19,8 +18,7 @@ to set up the execution environment. To achieve the best performance,
18
please follow the methods outlined in the
19
[Optimizing Training Platform Guide](https://docs.habana.ai/en/latest/PyTorch/Model_Optimization_PyTorch/Optimization_in_Training_Platform.html).
20
22
-# --8<-- [end:requirements]
23
-# --8<-- [start:configure-a-new-environment]
21
+## Configure a new environment
22
23
### Environment verification
24
@@ -57,16 +55,13 @@ docker run \
55
vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
56
```
57
60
-# --8<-- [end:configure-a-new-environment]
61
-# --8<-- [start:set-up-using-python]
58
+## Set up using Python
59
63
-# --8<-- [end:set-up-using-python]
64
-# --8<-- [start:pre-built-wheels]
60
+### Pre-built wheels
61
62
Currently, there are no pre-built Intel Gaudi wheels.
63
68
-# --8<-- [end:pre-built-wheels]
69
-# --8<-- [start:build-wheel-from-source]
64
+### Build wheel from source
65
66
To build and install vLLM from source, run:
67
@@ -87,16 +82,13 @@ pip install -r requirements/hpu.txt
82
python setup.py develop
83
```
84
90
-# --8<-- [end:build-wheel-from-source]
91
-# --8<-- [start:set-up-using-docker]
85
+## Set up using Docker
86
93
-# --8<-- [end:set-up-using-docker]
94
-# --8<-- [start:pre-built-images]
87
+### Pre-built images
88
89
Currently, there are no pre-built Intel Gaudi images.
90
98
-# --8<-- [end:pre-built-images]
99
-# --8<-- [start:build-image-from-source]
91
+### Build image from source
92
93
```console
94
docker build -f docker/Dockerfile.hpu -t vllm-hpu-env .
@@ -113,10 +105,9 @@ docker run \
105
!!! tip
106
If you're observing the following error: `docker: Error response from daemon: Unknown runtime specified habana.`, please refer to "Install Using Containers" section of [Intel Gaudi Software Stack and Driver Installation](https://docs.habana.ai/en/v1.18.0/Installation_Guide/Bare_Metal_Fresh_OS.html). Make sure you have `habana-container-runtime` package installed and that `habana` container runtime is registered.
107
116
-# --8<-- [end:build-image-from-source]
117
-# --8<-- [start:extra-information]
108
+## Extra information
109
119
-## Supported features
110
+### Supported features
111
112
- [Offline inference][offline-inference]
113
- Online serving via [OpenAI-Compatible Server][openai-compatible-server]
@@ -130,14 +121,14 @@ docker run \
121
for accelerating low-batch latency and throughput
122
- Attention with Linear Biases (ALiBi)
123
133
-## Unsupported features
124
+### Unsupported features
125
126
- Beam search
127
- LoRA adapters
128
- Quantization
129
- Prefill chunking (mixed-batch inferencing)
130
140
-## Supported configurations
131
+### Supported configurations
132
133
The following configurations have been validated to function with
134
Gaudi2 devices. Configurations that are not listed may or may not work.
@@ -401,4 +392,3 @@ the below:
392
higher batches. You can do that by adding `--enforce-eager` flag to
393
server (for online serving), or by passing `enforce_eager=True`
394
argument to LLM constructor (for offline inference).
404
-# --8<-- [end:extra-information]