A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 07:00 UTC
[Docs] Misc updates to TPU installation instructions (#10165)
Michael Green committed
Nov 15, 2024 at 21:26 UTC
4f168f69a3e856bda3f30e02fcee7db2a01ff32b
1 file changed
+35
-19
docs/source/getting_started/tpu-installation.rst
+35
-19
@@ -44,15 +44,18 @@ Requirements
44
Provision Cloud TPUs
45
====================
46
47
-You can provision Cloud TPUs using the `Cloud TPU API <https://cloud.google.com/tpu/docs/reference/rest>`_`
48
-or the `queued resources <https://cloud.google.com/tpu/docs/queued-resources>`_`
49
-API. This section shows how to create TPUs using the queued resource API.
50
-For more information about using the Cloud TPU API, see `Create a Cloud TPU using the Create Node API <https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm#create-node-api>`_.
51
-`Queued resources <https://cloud.devsite.corp.google.com/tpu/docs/queued-resources>`_
52
-enable you to request Cloud TPU resources in a queued manner. When you request
53
-queued resources, the request is added to a queue maintained by the Cloud TPU
54
-service. When the requested resource becomes available, it's assigned to your
55
-Google Cloud project for your immediate exclusive use.
47
+You can provision Cloud TPUs using the `Cloud TPU API <https://cloud.google.com/tpu/docs/reference/rest>`_
48
+or the `queued resources <https://cloud.google.com/tpu/docs/queued-resources>`_
49
+API. This section shows how to create TPUs using the queued resource API. For
50
+more information about using the Cloud TPU API, see `Create a Cloud TPU using the Create Node API <https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm#create-node-api>`_.
51
+Queued resources enable you to request Cloud TPU resources in a queued manner.
52
+When you request queued resources, the request is added to a queue maintained by
53
+the Cloud TPU service. When the requested resource becomes available, it's
54
+assigned to your Google Cloud project for your immediate exclusive use.
55
+
56
+.. note::
57
+ In all of the following commands, replace the ALL CAPS parameter names with
58
+ appropriate values. See the parameter descriptions table for more information.
59
60
Provision a Cloud TPU with the queued resource API
61
--------------------------------------------------
@@ -68,6 +71,7 @@ Create a TPU v5e with 4 TPU chips:
71
--runtime-version RUNTIME_VERSION \
72
--service-account SERVICE_ACCOUNT
73
74
+
75
.. list-table:: Parameter descriptions
76
:header-rows: 1
77
@@ -81,12 +85,13 @@ Create a TPU v5e with 4 TPU chips:
85
* - PROJECT_ID
86
- Your Google Cloud project
87
* - ZONE
84
- - The `zone <https://cloud.google.com/tpu/docs/regions-zones>`_ where you
85
- want to create your Cloud TPU.
88
+ - The GCP zone where you want to create your Cloud TPU. The value you use
89
+ depends on the version of TPUs you are using. For more information, see
90
+ `TPU regions and zones <https://cloud.google.com/tpu/docs/regions-zones>`_
91
* - ACCELERATOR_TYPE
87
- - The TPU version you want to use. Specify the TPU version, followed by a
88
- '-' and the number of TPU cores. For example `v5e-4` specifies a v5e TPU
89
- with 4 cores. For more information, see `TPU versions <https://cloud.devsite.corp.google.com/tpu/docs/system-architecture-tpu-vm#versions>`_.
92
+ - The TPU version you want to use. Specify the TPU version, for example
93
+ `v5litepod-4` specifies a v5e TPU with 4 cores. For more information,
94
+ see `TPU versions <https://cloud.devsite.corp.google.com/tpu/docs/system-architecture-tpu-vm#versions>`_.
95
* - RUNTIME_VERSION
96
- The TPU VM runtime version to use. For more information see `TPU VM images <https://cloud.google.com/tpu/docs/runtimes>`_.
97
* - SERVICE_ACCOUNT
@@ -98,7 +103,15 @@ Connect to your TPU using SSH:
103
104
.. code-block:: bash
105
101
- gcloud compute tpus tpu-vm ssh TPU_NAME
106
+ gcloud compute tpus tpu-vm ssh TPU_NAME --zone ZONE
107
+
108
+Install Miniconda
109
+
110
+.. code-block:: bash
111
+
112
+ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
113
+ bash Miniconda3-latest-Linux-x86_64.sh
114
+ source ~/.bashrc
115
116
Create and activate a Conda environment for vLLM:
117
@@ -162,9 +175,11 @@ Run the Docker image with the following command:
175
176
.. note::
177
165
- Since TPU relies on XLA which requires static shapes, vLLM bucketizes the possible input shapes and compiles an XLA graph for each different shape.
166
- The compilation time may take 20~30 minutes in the first run.
167
- However, the compilation time reduces to ~5 minutes afterwards because the XLA graphs are cached in the disk (in :code:`VLLM_XLA_CACHE_PATH` or :code:`~/.cache/vllm/xla_cache` by default).
178
+ Since TPU relies on XLA which requires static shapes, vLLM bucketizes the
179
+ possible input shapes and compiles an XLA graph for each shape. The
180
+ compilation time may take 20~30 minutes in the first run. However, the
181
+ compilation time reduces to ~5 minutes afterwards because the XLA graphs are
182
+ cached in the disk (in :code:`VLLM_XLA_CACHE_PATH` or :code:`~/.cache/vllm/xla_cache` by default).
183
184
.. tip::
185
@@ -173,7 +188,8 @@ Run the Docker image with the following command:
188
.. code-block:: console
189
190
from torch._C import * # noqa: F403
176
- ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
191
+ ImportError: libopenblas.so.0: cannot open shared object file: No such
192
+ file or directory
193
194
195
Install OpenBLAS with the following command: