@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 70bfbd7b16

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 13, 23:15 UTC

Docs update tpu install instructions (#27824)

Signed-off-by: Rob Mulla <rob.mulla@gmail.com> Signed-off-by: Rob Mulla <RobMulla@users.noreply.github.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Rob Mulla committed Oct 31, 2025 at 13:29 UTC 70bfbd7b168a216c6b5cb4db678a48b1e3f9c720

5 files changed +14 -196

docs/configuration/tpu.md

+1 -1

@@ -4,7 +4,7 @@ This doc serves as a collection of handy tips for optimizing your vLLM on TPU wo

5 ## Get started

7 -Looking for setup and installation instructions? Find them [here](../getting_started/installation/google_tpu.md).

7 +Looking for setup and installation instructions? Find them [here](https://docs.vllm.ai/projects/tpu/en/latest/getting_started/installation/).

9 ### TPU workload sizing

docs/getting_started/installation/.nav.yml

+1 -1

@@ -2,4 +2,4 @@ nav:

2 - README.md

3 - gpu.md

4 - cpu.md

5 - - google_tpu.md

5 + - TPU: https://docs.vllm.ai/projects/tpu/en/latest/getting_started/installation/

docs/getting_started/installation/README.md

+1 -1

@@ -11,7 +11,6 @@ vLLM supports the following hardware platforms:

11 - [ARM AArch64](cpu.md#arm-aarch64)

12 - [Apple silicon](cpu.md#apple-silicon)

13 - [IBM Z (S390X)](cpu.md#ibm-z-s390x)

14 -- [Google TPU](google_tpu.md)

15 ## Hardware Plugins

@@ -20,6 +19,7 @@ The backends below live **outside** the main `vllm` repository and follow the

20 | Accelerator | PyPI / package | Repository |

21 |-------------|----------------|------------|

22 +| Google TPU | `tpu-inference` | <https://github.com/vllm-project/tpu-inference> |

23 | Ascend NPU | `vllm-ascend` | <https://github.com/vllm-project/vllm-ascend> |

24 | Intel Gaudi (HPU) | N/A, install from source | <https://github.com/vllm-project/vllm-gaudi> |

25 | MetaX MACA GPU | N/A, install from source | <https://github.com/MetaX-MACA/vLLM-metax> |

docs/getting_started/installation/google_tpu.md deleted

-193

@@ -1,193 +0,0 @@

1 -# Google TPU

2 -

3 -Tensor Processing Units (TPUs) are Google's custom-developed application-specific

4 -integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs

5 -are available in different versions each with different hardware specifications.

6 -For more information about TPUs, see [TPU System Architecture](https://cloud.google.com/tpu/docs/system-architecture-tpu-vm).

7 -For more information on the TPU versions supported with vLLM, see:

8 -

9 -- [TPU v6e](https://cloud.google.com/tpu/docs/v6e)

10 -- [TPU v5e](https://cloud.google.com/tpu/docs/v5e)

11 -- [TPU v5p](https://cloud.google.com/tpu/docs/v5p)

12 -- [TPU v4](https://cloud.google.com/tpu/docs/v4)

13 -

14 -These TPU versions allow you to configure the physical arrangements of the TPU

15 -chips. This can improve throughput and networking performance. For more

16 -information see:

17 -

18 -- [TPU v6e topologies](https://cloud.google.com/tpu/docs/v6e#configurations)

19 -- [TPU v5e topologies](https://cloud.google.com/tpu/docs/v5e#tpu-v5e-config)

20 -- [TPU v5p topologies](https://cloud.google.com/tpu/docs/v5p#tpu-v5p-config)

21 -- [TPU v4 topologies](https://cloud.google.com/tpu/docs/v4#tpu-v4-config)

22 -

23 -In order for you to use Cloud TPUs you need to have TPU quota granted to your

24 -Google Cloud Platform project. TPU quotas specify how many TPUs you can use in a

25 -GPC project and are specified in terms of TPU version, the number of TPU you

26 -want to use, and quota type. For more information, see [TPU quota](https://cloud.google.com/tpu/docs/quota#tpu_quota).

27 -

28 -For TPU pricing information, see [Cloud TPU pricing](https://cloud.google.com/tpu/pricing).

29 -

30 -You may need additional persistent storage for your TPU VMs. For more

31 -information, see [Storage options for Cloud TPU data](https://cloud.devsite.corp.google.com/tpu/docs/storage-options).

32 -

33 -!!! warning

34 - There are no pre-built wheels for this device, so you must either use the pre-built Docker image or build vLLM from source.

35 -

36 -## Requirements

37 -

38 -- Google Cloud TPU VM

39 -- TPU versions: v6e, v5e, v5p, v4

40 -- Python: 3.11 or newer

41 -

42 -### Provision Cloud TPUs

43 -

44 -You can provision Cloud TPUs using the [Cloud TPU API](https://cloud.google.com/tpu/docs/reference/rest)

45 -or the [queued resources](https://cloud.google.com/tpu/docs/queued-resources)

46 -API (preferred). This section shows how to create TPUs using the queued resource API. For

47 -more information about using the Cloud TPU API, see [Create a Cloud TPU using the Create Node API](https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm#create-node-api).

48 -Queued resources enable you to request Cloud TPU resources in a queued manner.

49 -When you request queued resources, the request is added to a queue maintained by

50 -the Cloud TPU service. When the requested resource becomes available, it's

51 -assigned to your Google Cloud project for your immediate exclusive use.

52 -

53 -!!! note

54 - In all of the following commands, replace the ALL CAPS parameter names with

55 - appropriate values. See the parameter descriptions table for more information.

56 -

57 -### Provision Cloud TPUs with GKE

58 -

59 -For more information about using TPUs with GKE, see:

60 -

61 -- [About TPUs in GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/tpus)

62 -- [Deploy TPU workloads in GKE Standard](https://cloud.google.com/kubernetes-engine/docs/how-to/tpus)

63 -- [Plan for TPUs in GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/plan-tpus)

64 -

65 -## Configure a new environment

66 -

67 -### Provision a Cloud TPU with the queued resource API

68 -

69 -Create a TPU v5e with 4 TPU chips:

70 -

71 -```bash

72 -gcloud alpha compute tpus queued-resources create QUEUED_RESOURCE_ID \

73 - --node-id TPU_NAME \

74 - --project PROJECT_ID \

75 - --zone ZONE \

76 - --accelerator-type ACCELERATOR_TYPE \

77 - --runtime-version RUNTIME_VERSION \

78 - --service-account SERVICE_ACCOUNT

79 -```

80 -

81 -| Parameter name | Description |

82 -|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

83 -| QUEUED_RESOURCE_ID | The user-assigned ID of the queued resource request. |

84 -| TPU_NAME | The user-assigned name of the TPU which is created when the queued resource request is allocated. |

85 -| PROJECT_ID | Your Google Cloud project |

86 -| ZONE | The GCP zone where you want to create your Cloud TPU. The value you use depends on the version of TPUs you are using. For more information, see [TPU regions and zones] |

87 -| ACCELERATOR_TYPE | The TPU version you want to use. Specify the TPU version, for example `v5litepod-4` specifies a v5e TPU with 4 cores, `v6e-1` specifies a v6e TPU with 1 core. For more information, see [TPU versions]. |

88 -| RUNTIME_VERSION | The TPU VM runtime version to use. For example, use `v2-alpha-tpuv6e` for a VM loaded with one or more v6e TPU(s). |

89 -| SERVICE_ACCOUNT | The email address for your service account. You can find it in the IAM Cloud Console under *Service Accounts*. For example: `tpu-service-account@<your_project_ID>.iam.gserviceaccount.com` |

90 -

91 -Connect to your TPU VM using SSH:

92 -

93 -```bash

94 -gcloud compute tpus tpu-vm ssh TPU_NAME --project PROJECT_ID --zone ZONE

95 -```

96 -

97 -!!! note

98 - When configuring `RUNTIME_VERSION` ("TPU software version") on GCP, ensure it matches the TPU generation you've selected by referencing the [TPU VM images] compatibility matrix. Using an incompatible version may prevent vLLM from running correctly.

99 -

100 -[TPU versions]: https://cloud.google.com/tpu/docs/runtimes

101 -[TPU VM images]: https://cloud.google.com/tpu/docs/runtimes

102 -[TPU regions and zones]: https://cloud.google.com/tpu/docs/regions-zones

103 -

104 -## Set up using Python

105 -

106 -### Pre-built wheels

107 -

108 -Currently, there are no pre-built TPU wheels.

109 -

110 -### Build wheel from source

111 -

112 -Install Miniconda:

113 -

114 -```bash

115 -wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

116 -bash Miniconda3-latest-Linux-x86_64.sh

117 -source ~/.bashrc

118 -```

119 -

120 -Create and activate a Conda environment for vLLM:

121 -

122 -```bash

123 -conda create -n vllm python=3.12 -y

124 -conda activate vllm

125 -```

126 -

127 -Clone the vLLM repository and go to the vLLM directory:

128 -

129 -```bash

130 -git clone https://github.com/vllm-project/vllm.git && cd vllm

131 -```

132 -

133 -Uninstall the existing `torch` and `torch_xla` packages:

134 -

135 -```bash

136 -pip uninstall torch torch-xla -y

137 -```

138 -

139 -Install build dependencies:

140 -

141 -```bash

142 -pip install -r requirements/tpu.txt

143 -sudo apt-get install --no-install-recommends --yes libopenblas-base libopenmpi-dev libomp-dev

144 -```

145 -

146 -Run the setup script:

147 -

148 -```bash

149 -VLLM_TARGET_DEVICE="tpu" python -m pip install -e .

150 -```

151 -

152 -## Set up using Docker

153 -

154 -### Pre-built images

155 -

156 -See [Using Docker](../../deployment/docker.md) for instructions on using the official Docker image, making sure to substitute the image name `vllm/vllm-openai` with `vllm/vllm-tpu`.

157 -

158 -### Build image from source

159 -

160 -You can use [docker/Dockerfile.tpu](../../../docker/Dockerfile.tpu) to build a Docker image with TPU support.

161 -

162 -```bash

163 -docker build -f docker/Dockerfile.tpu -t vllm-tpu .

164 -```

165 -

166 -Run the Docker image with the following command:

167 -

168 -```bash

169 -# Make sure to add `--privileged --net host --shm-size=16G`.

170 -docker run --privileged --net host --shm-size=16G -it vllm-tpu

171 -```

172 -

173 -!!! note

174 - Since TPU relies on XLA which requires static shapes, vLLM bucketizes the

175 - possible input shapes and compiles an XLA graph for each shape. The

176 - compilation time may take 20~30 minutes in the first run. However, the

177 - compilation time reduces to ~5 minutes afterwards because the XLA graphs are

178 - cached in the disk (in `VLLM_XLA_CACHE_PATH` or `~/.cache/vllm/xla_cache` by default).

179 -

180 -!!! tip

181 - If you encounter the following error:

182 -

183 - ```console

184 - from torch._C import * # noqa: F403

185 - ImportError: libopenblas.so.0: cannot open shared object file: No such

186 - file or directory

187 - ```

188 -

189 - Install OpenBLAS with the following command:

190 -

191 - ```bash

192 - sudo apt-get install --no-install-recommends --yes libopenblas-base libopenmpi-dev libomp-dev

193 - ```

docs/getting_started/quickstart.md

+11

@@ -63,6 +63,17 @@ This guide will help you quickly get started with vLLM to perform:

63 rocm/vllm-dev:nightly

64 ```

66 +=== "Google TPU"

67 +

68 + To run vLLM on Google TPUs, you need to install the `vllm-tpu` package.

69 +

70 + ```bash

71 + uv pip install vllm-tpu

72 + ```

73 +

74 + !!! note

75 + For more detailed instructions, including Docker, installing from source, and troubleshooting, please refer to the [vLLM on TPU documentation](https://docs.vllm.ai/projects/tpu/en/latest/).

76 +

77 !!! note

78 For more detail and non-CUDA platforms, please refer [here](installation/README.md) for specific instructions on how to install vLLM.