@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 1d4cfe2be1
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 09:30 UTC

[Doc] Updated tpu-installation.rst with more details (#9926)

Signed-off-by: Michael Green <mikegre@google.com>

Michael Green committed Nov 2, 2024 at 14:06 UTC 1d4cfe2be1907408d610489bdca7bc8f8d2345b1
1 file changed +144 -14
docs/source/getting_started/tpu-installation.rst
+144 -14
@@ -1,35 +1,167 @@
1 .. _installation_tpu:
2
3 +#####################
4 Installation with TPU
4 -=====================
5 +#####################
6
6 -vLLM supports Google Cloud TPUs using PyTorch XLA.
7 +Tensor Processing Units (TPUs) are Google's custom-developed application-specific
8 +integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs
9 +are available in different versions each with different hardware specifications.
10 +For more information about TPUs, see `TPU System Architecture <https://cloud.google.com/tpu/docs/system-architecture-tpu-vm>`_.
11 +For more information on the TPU versions supported with vLLM, see:
12 +
13 +* `TPU v6e <https://cloud.google.com/tpu/docs/v6e>`_
14 +* `TPU v5e <https://cloud.google.com/tpu/docs/v5e>`_
15 +* `TPU v5p <https://cloud.google.com/tpu/docs/v5p>`_
16 +* `TPU v4 <https://cloud.google.com/tpu/docs/v4>`_
17 +
18 +These TPU versions allow you to configure the physical arrangements of the TPU
19 +chips. This can improve throughput and networking performance. For more
20 +information see:
21 +
22 +* `TPU v6e topologies <https://cloud.google.com/tpu/docs/v6e#configurations>`_
23 +* `TPU v5e topologies <https://cloud.google.com/tpu/docs/v5e#tpu-v5e-config>`_
24 +* `TPU v5p topologies <https://cloud.google.com/tpu/docs/v5p#tpu-v5p-config>`_
25 +* `TPU v4 topologies <https://cloud.google.com/tpu/docs/v4#tpu-v4-config>`_
26 +
27 +In order for you to use Cloud TPUs you need to have TPU quota granted to your
28 +Google Cloud Platform project. TPU quotas specify how many TPUs you can use in a
29 +GPC project and are specified in terms of TPU version, the number of TPU you
30 +want to use, and quota type. For more information, see `TPU quota <https://cloud.google.com/tpu/docs/quota#tpu_quota>`_.
31 +
32 +For TPU pricing information, see `Cloud TPU pricing <https://cloud.google.com/tpu/pricing>`_.
33 +
34 +You may need additional persistent storage for your TPU VMs. For more
35 +information, see `Storage options for Cloud TPU data <https://cloud.devsite.corp.google.com/tpu/docs/storage-options>`_.
36
37 Requirements
38 ------------
39
11 -* Google Cloud TPU VM (single & multi host)
12 -* TPU versions: v5e, v5p, v4
13 -* Python: 3.10
40 +* Google Cloud TPU VM
41 +* TPU versions: v6e, v5e, v5p, v4
42 +* Python: 3.10 or newer
43 +
44 +Provision Cloud TPUs
45 +====================
46 +
47 +You can provision Cloud TPUs using the `Cloud TPU API <https://cloud.google.com/tpu/docs/reference/rest>`_`
48 +or the `queued resources <https://cloud.google.com/tpu/docs/queued-resources>`_`
49 +API. This section shows how to create TPUs using the queued resource API.
50 +For more information about using the Cloud TPU API, see `Create a Cloud TPU using the Create Node API <https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm#create-node-api>`_.
51 +`Queued resources <https://cloud.devsite.corp.google.com/tpu/docs/queued-resources>`_
52 +enable you to request Cloud TPU resources in a queued manner. When you request
53 +queued resources, the request is added to a queue maintained by the Cloud TPU
54 +service. When the requested resource becomes available, it's assigned to your
55 +Google Cloud project for your immediate exclusive use.
56 +
57 +Provision a Cloud TPU with the queued resource API
58 +--------------------------------------------------
59 +Create a TPU v5e with 4 TPU chips:
60 +
61 +.. code-block:: console
62 +
63 + gcloud alpha compute tpus queued-resources create QUEUED_RESOURCE_ID \
64 + --node-id TPU_NAME \
65 + --project PROJECT_ID \
66 + --zone ZONE \
67 + --accelerator-type ACCELERATOR_TYPE \
68 + --runtime-version RUNTIME_VERSION \
69 + --service-account SERVICE_ACCOUNT
70 +
71 +.. list-table:: Parameter descriptions
72 + :header-rows: 1
73 +
74 + * - Parameter name
75 + - Description
76 + * - QUEUED_RESOURCE_ID
77 + - The user-assigned ID of the queued resource request.
78 + * - TPU_NAME
79 + - The user-assigned name of the TPU which is created when the queued
80 + resource request is allocated.
81 + * - PROJECT_ID
82 + - Your Google Cloud project
83 + * - ZONE
84 + - The `zone <https://cloud.google.com/tpu/docs/regions-zones>`_ where you
85 + want to create your Cloud TPU.
86 + * - ACCELERATOR_TYPE
87 + - The TPU version you want to use. Specify the TPU version, followed by a
88 + '-' and the number of TPU cores. For example `v5e-4` specifies a v5e TPU
89 + with 4 cores. For more information, see `TPU versions <https://cloud.devsite.corp.google.com/tpu/docs/system-architecture-tpu-vm#versions>`_.
90 + * - RUNTIME_VERSION
91 + - The TPU VM runtime version to use. For more information see `TPU VM images <https://cloud.google.com/tpu/docs/runtimes>`_.
92 + * - SERVICE_ACCOUNT
93 + - The email address for your service account. You can find it in the IAM
94 + Cloud Console under *Service Accounts*. For example:
95 + `tpu-service-account@<your_project_ID>.iam.gserviceaccount.com`
96 +
97 +Connect to your TPU using SSH:
98 +
99 +.. code-block:: bash
100 +
101 + gcloud compute tpus tpu-vm ssh TPU_NAME
102 +
103 +Create and activate a Conda environment for vLLM:
104 +
105 +.. code-block:: bash
106
15 -Installation options:
107 + conda create -n vllm python=3.10 -y
108 + conda activate vllm
109
17 -1. :ref:`Build a docker image with Dockerfile <build_docker_tpu>`.
18 -2. :ref:`Build from source <build_from_source_tpu>`.
110 +Clone the vLLM repository and go to the vLLM directory:
111 +
112 +.. code-block:: bash
113 +
114 + git clone https://github.com/vllm-project/vllm.git && cd vllm
115 +
116 +Uninstall the existing `torch` and `torch_xla` packages:
117 +
118 +.. code-block:: bash
119 +
120 + pip uninstall torch torch-xla -y
121 +
122 +Install `torch` and `torch_xla`
123 +
124 +.. code-block:: bash
125 +
126 + pip install --pre torch==2.6.0.dev20241028+cpu torchvision==0.20.0.dev20241028+cpu --index-url https://download.pytorch.org/whl/nightly/cpu
127 + pip install 'torch_xla[tpu] @ https://storage.googleapis.com/pytorch-xla-releases/wheels/tpuvm/torch_xla-2.6.0.dev-cp310-cp310-linux_x86_64.whl' -f https://storage.googleapis.com/libtpu-releases/index.html
128 +
129 +Install JAX and Pallas:
130 +
131 +.. code-block:: bash
132 +
133 + pip install torch_xla[pallas] -f https://storage.googleapis.com/jax-releases/jax_nightly_releases.html -f https://storage.googleapis.com/jax-releases/jaxlib_nightly_releases.html
134 + pip install jaxlib==0.4.32.dev20240829 jax==0.4.32.dev20240829 -f https://storage.googleapis.com/jax-releases/jax_nightly_releases.html -f https://storage.googleapis.com/jax-releases/jaxlib_nightly_releases.html
135 +
136 +Install other build dependencies:
137 +
138 +.. code-block:: bash
139 +
140 + pip install -r requirements-tpu.txt
141 + VLLM_TARGET_DEVICE="tpu" python setup.py develop
142 + sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev
143 +
144 +Provision Cloud TPUs with GKE
145 +-----------------------------
146 +
147 +For more information about using TPUs with GKE, see
148 +https://cloud.google.com/kubernetes-engine/docs/how-to/tpus
149 +https://cloud.google.com/kubernetes-engine/docs/concepts/tpus
150 +https://cloud.google.com/kubernetes-engine/docs/concepts/plan-tpus
151
152 .. _build_docker_tpu:
153
154 Build a docker image with :code:`Dockerfile.tpu`
155 ------------------------------------------------
156
25 -`Dockerfile.tpu <https://github.com/vllm-project/vllm/blob/main/Dockerfile.tpu>`_ is provided to build a docker image with TPU support.
157 +You can use `Dockerfile.tpu <https://github.com/vllm-project/vllm/blob/main/Dockerfile.tpu>`_
158 +to build a Docker image with TPU support.
159
160 .. code-block:: console
161
162 $ docker build -f Dockerfile.tpu -t vllm-tpu .
163
31 -
32 -You can run the docker image with the following command:
164 +Run the Docker image with the following command:
165
166 .. code-block:: console
167
@@ -75,14 +207,12 @@ Next, build vLLM from source. This will only take a few seconds:
207
208 $ VLLM_TARGET_DEVICE="tpu" python setup.py develop
209
78 -
210 .. note::
211
212 Since TPU relies on XLA which requires static shapes, vLLM bucketizes the possible input shapes and compiles an XLA graph for each different shape.
213 The compilation time may take 20~30 minutes in the first run.
214 However, the compilation time reduces to ~5 minutes afterwards because the XLA graphs are cached in the disk (in :code:`VLLM_XLA_CACHE_PATH` or :code:`~/.cache/vllm/xla_cache` by default).
215
85 -
216 .. tip::
217
218 If you encounter the following error:
@@ -93,7 +223,7 @@ Next, build vLLM from source. This will only take a few seconds:
223 ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
224
225
96 - Please install OpenBLAS with the following command:
226 + Install OpenBLAS with the following command:
227
228 .. code-block:: console
229