@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 4f168f69a3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

[Docs] Misc updates to TPU installation instructions (#10165)

Michael Green committed Nov 15, 2024 at 21:26 UTC 4f168f69a3e856bda3f30e02fcee7db2a01ff32b
1 file changed +35 -19
docs/source/getting_started/tpu-installation.rst
+35 -19
@@ -44,15 +44,18 @@ Requirements
44 Provision Cloud TPUs
45 ====================
46
47 -You can provision Cloud TPUs using the `Cloud TPU API <https://cloud.google.com/tpu/docs/reference/rest>`_`
48 -or the `queued resources <https://cloud.google.com/tpu/docs/queued-resources>`_`
49 -API. This section shows how to create TPUs using the queued resource API.
50 -For more information about using the Cloud TPU API, see `Create a Cloud TPU using the Create Node API <https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm#create-node-api>`_.
51 -`Queued resources <https://cloud.devsite.corp.google.com/tpu/docs/queued-resources>`_
52 -enable you to request Cloud TPU resources in a queued manner. When you request
53 -queued resources, the request is added to a queue maintained by the Cloud TPU
54 -service. When the requested resource becomes available, it's assigned to your
55 -Google Cloud project for your immediate exclusive use.
47 +You can provision Cloud TPUs using the `Cloud TPU API <https://cloud.google.com/tpu/docs/reference/rest>`_
48 +or the `queued resources <https://cloud.google.com/tpu/docs/queued-resources>`_
49 +API. This section shows how to create TPUs using the queued resource API. For
50 +more information about using the Cloud TPU API, see `Create a Cloud TPU using the Create Node API <https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm#create-node-api>`_.
51 +Queued resources enable you to request Cloud TPU resources in a queued manner.
52 +When you request queued resources, the request is added to a queue maintained by
53 +the Cloud TPU service. When the requested resource becomes available, it's
54 +assigned to your Google Cloud project for your immediate exclusive use.
55 +
56 +.. note::
57 + In all of the following commands, replace the ALL CAPS parameter names with
58 + appropriate values. See the parameter descriptions table for more information.
59
60 Provision a Cloud TPU with the queued resource API
61 --------------------------------------------------
@@ -68,6 +71,7 @@ Create a TPU v5e with 4 TPU chips:
71 --runtime-version RUNTIME_VERSION \
72 --service-account SERVICE_ACCOUNT
73
74 +
75 .. list-table:: Parameter descriptions
76 :header-rows: 1
77
@@ -81,12 +85,13 @@ Create a TPU v5e with 4 TPU chips:
85 * - PROJECT_ID
86 - Your Google Cloud project
87 * - ZONE
84 - - The `zone <https://cloud.google.com/tpu/docs/regions-zones>`_ where you
85 - want to create your Cloud TPU.
88 + - The GCP zone where you want to create your Cloud TPU. The value you use
89 + depends on the version of TPUs you are using. For more information, see
90 + `TPU regions and zones <https://cloud.google.com/tpu/docs/regions-zones>`_
91 * - ACCELERATOR_TYPE
87 - - The TPU version you want to use. Specify the TPU version, followed by a
88 - '-' and the number of TPU cores. For example `v5e-4` specifies a v5e TPU
89 - with 4 cores. For more information, see `TPU versions <https://cloud.devsite.corp.google.com/tpu/docs/system-architecture-tpu-vm#versions>`_.
92 + - The TPU version you want to use. Specify the TPU version, for example
93 + `v5litepod-4` specifies a v5e TPU with 4 cores. For more information,
94 + see `TPU versions <https://cloud.devsite.corp.google.com/tpu/docs/system-architecture-tpu-vm#versions>`_.
95 * - RUNTIME_VERSION
96 - The TPU VM runtime version to use. For more information see `TPU VM images <https://cloud.google.com/tpu/docs/runtimes>`_.
97 * - SERVICE_ACCOUNT
@@ -98,7 +103,15 @@ Connect to your TPU using SSH:
103
104 .. code-block:: bash
105
101 - gcloud compute tpus tpu-vm ssh TPU_NAME
106 + gcloud compute tpus tpu-vm ssh TPU_NAME --zone ZONE
107 +
108 +Install Miniconda
109 +
110 +.. code-block:: bash
111 +
112 + wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
113 + bash Miniconda3-latest-Linux-x86_64.sh
114 + source ~/.bashrc
115
116 Create and activate a Conda environment for vLLM:
117
@@ -162,9 +175,11 @@ Run the Docker image with the following command:
175
176 .. note::
177
165 - Since TPU relies on XLA which requires static shapes, vLLM bucketizes the possible input shapes and compiles an XLA graph for each different shape.
166 - The compilation time may take 20~30 minutes in the first run.
167 - However, the compilation time reduces to ~5 minutes afterwards because the XLA graphs are cached in the disk (in :code:`VLLM_XLA_CACHE_PATH` or :code:`~/.cache/vllm/xla_cache` by default).
178 + Since TPU relies on XLA which requires static shapes, vLLM bucketizes the
179 + possible input shapes and compiles an XLA graph for each shape. The
180 + compilation time may take 20~30 minutes in the first run. However, the
181 + compilation time reduces to ~5 minutes afterwards because the XLA graphs are
182 + cached in the disk (in :code:`VLLM_XLA_CACHE_PATH` or :code:`~/.cache/vllm/xla_cache` by default).
183
184 .. tip::
185
@@ -173,7 +188,8 @@ Run the Docker image with the following command:
188 .. code-block:: console
189
190 from torch._C import * # noqa: F403
176 - ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
191 + ImportError: libopenblas.so.0: cannot open shared object file: No such
192 + file or directory
193
194
195 Install OpenBLAS with the following command: