A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
Add documentation to Triton server tutorial (#983)
Tanmay Verma committed
Sep 20, 2023 at 10:32 UTC
6f2dd6c37e984dd254d263007b4be0b60964630c
2 files changed
+7
docs/source/index.rst
+1
@@ -64,6 +64,7 @@ Documentation
64
65
serving/distributed_serving
66
serving/run_on_sky
67
+ serving/deploying_with_triton
68
69
.. toctree::
70
:maxdepth: 1
docs/source/serving/deploying_with_triton.rst
new
+6
@@ -0,0 +1,6 @@
1
+.. _deploying_with_triton:
2
+
3
+Deploying with NVIDIA Triton
4
+============================
5
+
6
+The `Triton Inference Server <https://github.com/triton-inference-server>`_ hosts a tutorial demonstrating how to quickly deploy a simple `facebook/opt-125m <https://huggingface.co/facebook/opt-125m>`_ model using vLLM. Please see `Deploying a vLLM model in Triton <https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/vLLM/README.md#deploying-a-vllm-model-in-triton>`_ for more details.