@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / d0fae88114
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

[DOC] add setup document to support neuron backend (#2777)

Liangfu Chen committed Mar 3, 2024 at 17:03 UTC d0fae881143f07a558ea72b2cae3c4c6dfa94937
2 files changed +136
docs/source/getting_started/neuron-installation.rst new
+135
@@ -0,0 +1,135 @@
1 +.. _installation_neuron:
2 +
3 +Installation with Neuron
4 +========================
5 +
6 +vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK.
7 +At the moment Paged Attention is not supported in Neuron SDK, but naive continuous batching is supported in transformers-neuronx.
8 +Data types currently supported in Neuron SDK are FP16 and BF16.
9 +
10 +Requirements
11 +------------
12 +
13 +* OS: Linux
14 +* Python: 3.8 -- 3.11
15 +* Accelerator: NeuronCore_v2 (in trn1/inf2 instances)
16 +* Pytorch 2.0.1/2.1.1
17 +* AWS Neuron SDK 2.16/2.17 (Verified on python 3.8)
18 +
19 +Installation steps:
20 +
21 +- :ref:`Build from source <build_from_source_neuron>`
22 +
23 + - :ref:`Step 0. Launch Trn1/Inf2 instances <launch_instances>`
24 + - :ref:`Step 1. Install drivers and tools <install_drivers>`
25 + - :ref:`Step 2. Install transformers-neuronx and its dependencies <install_tnx>`
26 + - :ref:`Step 3. Install vLLM from source <install_vllm>`
27 +
28 +.. _build_from_source_neuron:
29 +
30 +Build from source
31 +-----------------
32 +
33 +Following instructions are applicable to Neuron SDK 2.16 and beyond.
34 +
35 +.. _launch_instances:
36 +
37 +Step 0. Launch Trn1/Inf2 instances
38 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
39 +
40 +Here are the steps to launch trn1/inf2 instances, in order to install `PyTorch Neuron ("torch-neuronx") Setup on Ubuntu 22.04 LTS <https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/setup/neuron-setup/pytorch/neuronx/ubuntu/torch-neuronx-ubuntu22.html>`_.
41 +
42 +- Please follow the instructions at `launch an Amazon EC2 Instance <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html#ec2-launch-instance>`_ to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.
43 +- To get more information about instances sizes and pricing see: `Trn1 web page <https://aws.amazon.com/ec2/instance-types/trn1/>`_, `Inf2 web page <https://aws.amazon.com/ec2/instance-types/inf2/>`_
44 +- Select Ubuntu Server 22.04 TLS AMI
45 +- When launching a Trn1/Inf2, please adjust your primary EBS volume size to a minimum of 512GB.
46 +- After launching the instance, follow the instructions in `Connect to your instance <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html>`_ to connect to the instance
47 +
48 +.. _install_drivers:
49 +
50 +Step 1. Install drivers and tools
51 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
52 +
53 +The installation of drivers and tools wouldn't be necessary, if `Deep Learning AMI Neuron <https://docs.aws.amazon.com/dlami/latest/devguide/appendix-ami-release-notes.html>`_ is installed. In case the drivers and tools are not installed on the operating system, follow the steps below:
54 +
55 +.. code-block:: console
56 +
57 + # Configure Linux for Neuron repository updates
58 + . /etc/os-release
59 + sudo tee /etc/apt/sources.list.d/neuron.list > /dev/null <<EOF
60 + deb https://apt.repos.neuron.amazonaws.com ${VERSION_CODENAME} main
61 + EOF
62 + wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | sudo apt-key add -
63 +
64 + # Update OS packages
65 + sudo apt-get update -y
66 +
67 + # Install OS headers
68 + sudo apt-get install linux-headers-$(uname -r) -y
69 +
70 + # Install git
71 + sudo apt-get install git -y
72 +
73 + # install Neuron Driver
74 + sudo apt-get install aws-neuronx-dkms=2.* -y
75 +
76 + # Install Neuron Runtime
77 + sudo apt-get install aws-neuronx-collectives=2.* -y
78 + sudo apt-get install aws-neuronx-runtime-lib=2.* -y
79 +
80 + # Install Neuron Tools
81 + sudo apt-get install aws-neuronx-tools=2.* -y
82 +
83 + # Add PATH
84 + export PATH=/opt/aws/neuron/bin:$PATH
85 +
86 +
87 +.. _install_tnx:
88 +
89 +Step 2. Install transformers-neuronx and its dependencies
90 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
91 +
92 +`transformers-neuronx <https://github.com/aws-neuron/transformers-neuronx>`_ will be the backend to support inference on trn1/inf2 instances.
93 +Follow the steps below to install transformer-neuronx package and its dependencies.
94 +
95 +.. code-block:: console
96 +
97 + # Install Python venv
98 + sudo apt-get install -y python3.10-venv g++
99 +
100 + # Create Python venv
101 + python3.10 -m venv aws_neuron_venv_pytorch
102 +
103 + # Activate Python venv
104 + source aws_neuron_venv_pytorch/bin/activate
105 +
106 + # Install Jupyter notebook kernel
107 + pip install ipykernel
108 + python3.10 -m ipykernel install --user --name aws_neuron_venv_pytorch --display-name "Python (torch-neuronx)"
109 + pip install jupyter notebook
110 + pip install environment_kernels
111 +
112 + # Set pip repository pointing to the Neuron repository
113 + python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com
114 +
115 + # Install wget, awscli
116 + python -m pip install wget
117 + python -m pip install awscli
118 +
119 + # Update Neuron Compiler and Framework
120 + python -m pip install --upgrade neuronx-cc==2.* --pre torch-neuronx==2.1.* torchvision transformers-neuronx
121 +
122 +.. _install_vllm:
123 +
124 +Step 3. Install vLLM from source
125 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
126 +
127 +Once neuronx-cc and transformers-neuronx packages are installed, we will be able to install vllm as follows:
128 +
129 +.. code-block:: console
130 +
131 + $ cd vllm
132 + $ pip install -U -r requirements-neuron.txt
133 + $ pip install .
134 +
135 +If neuron packages are detected correctly in the installation process, ``vllm-0.3.0+neuron212`` will be installed.
docs/source/index.rst
+1
@@ -62,6 +62,7 @@ Documentation
62
63 getting_started/installation
64 getting_started/amd-installation
65 + getting_started/neuron-installation
66 getting_started/quickstart
67
68 .. toctree::