@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 7c8566aa4f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 15:45 UTC

[Doc] neuron documentation update (#8671)

Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>

omrishiv committed Sep 20, 2024 at 15:04 UTC 7c8566aa4ff16b79a576436fbb50f03643febf07
2 files changed +3 -3
docs/source/getting_started/neuron-installation.rst
+2 -2
@@ -3,8 +3,8 @@
3 Installation with Neuron
4 ========================
5
6 -vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK.
7 -At the moment Paged Attention is not supported in Neuron SDK, but naive continuous batching is supported in transformers-neuronx.
6 +vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK with continuous batching.
7 +Paged Attention and Chunked Prefill are currently in development and will be available soon.
8 Data types currently supported in Neuron SDK are FP16 and BF16.
9
10 Requirements
docs/source/index.rst
+1 -1
@@ -43,7 +43,7 @@ vLLM is flexible and easy to use with:
43 * Tensor parallelism and pipeline parallelism support for distributed inference
44 * Streaming outputs
45 * OpenAI-compatible API server
46 -* Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
46 +* Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Trainium and Inferentia Accelerators.
47 * Prefix caching support
48 * Multi-lora support
49