@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 7f9c8902e3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 15:45 UTC

[Hardware][AWS] update neuron to 2.20 (#8676)

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

omrishiv committed Sep 20, 2024 at 15:19 UTC 7f9c8902e3d50a9d715b38e0531280a58d2bbe14
2 files changed +4 -4
Dockerfile.neuron
+2 -2
@@ -1,5 +1,5 @@
1 # default base image
2 -ARG BASE_IMAGE="public.ecr.aws/neuron/pytorch-inference-neuronx:2.1.2-neuronx-py310-sdk2.19.1-ubuntu20.04"
2 +ARG BASE_IMAGE="public.ecr.aws/neuron/pytorch-inference-neuronx:2.1.2-neuronx-py310-sdk2.20.0-ubuntu20.04"
3
4 FROM $BASE_IMAGE
5
@@ -20,7 +20,7 @@ RUN python3 -m pip install --upgrade pip
20 RUN python3 -m pip install --no-cache-dir fastapi ninja tokenizers pandas
21 RUN python3 -m pip install sentencepiece transformers==4.36.2 -U
22 RUN python3 -m pip install transformers-neuronx --extra-index-url=https://pip.repos.neuron.amazonaws.com -U
23 -RUN python3 -m pip install --pre neuronx-cc==2.12.* --extra-index-url=https://pip.repos.neuron.amazonaws.com -U
23 +RUN python3 -m pip install --pre neuronx-cc==2.15.* --extra-index-url=https://pip.repos.neuron.amazonaws.com -U
24
25 COPY ./vllm /app/vllm/vllm
26 COPY ./setup.py /app/vllm/setup.py
requirements-neuron.txt
+2 -2
@@ -2,6 +2,6 @@
2 -r requirements-common.txt
3
4 # Dependencies for Neuron devices
5 -transformers-neuronx >= 0.9.0
6 -torch-neuronx >= 2.1.0
5 +transformers-neuronx >= 0.12.0
6 +torch-neuronx >= 2.1.2
7 neuronx-cc