A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 15:45 UTC
[Doc] neuron documentation update (#8671)
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
omrishiv committed
Sep 20, 2024 at 15:04 UTC
7c8566aa4ff16b79a576436fbb50f03643febf07
2 files changed
+3
-3
docs/source/getting_started/neuron-installation.rst
+2
-2
@@ -3,8 +3,8 @@
3
Installation with Neuron
4
========================
5
6
-vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK.
7
-At the moment Paged Attention is not supported in Neuron SDK, but naive continuous batching is supported in transformers-neuronx.
6
+vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK with continuous batching.
7
+Paged Attention and Chunked Prefill are currently in development and will be available soon.
8
Data types currently supported in Neuron SDK are FP16 and BF16.
9
10
Requirements
docs/source/index.rst
+1
-1
@@ -43,7 +43,7 @@ vLLM is flexible and easy to use with:
43
* Tensor parallelism and pipeline parallelism support for distributed inference
44
* Streaming outputs
45
* OpenAI-compatible API server
46
-* Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
46
+* Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Trainium and Inferentia Accelerators.
47
* Prefix caching support
48
* Multi-lora support
49