@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 6f2dd6c37e
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

Add documentation to Triton server tutorial (#983)

Tanmay Verma committed Sep 20, 2023 at 10:32 UTC 6f2dd6c37e984dd254d263007b4be0b60964630c
2 files changed +7
docs/source/index.rst
+1
@@ -64,6 +64,7 @@ Documentation
64
65 serving/distributed_serving
66 serving/run_on_sky
67 + serving/deploying_with_triton
68
69 .. toctree::
70 :maxdepth: 1
docs/source/serving/deploying_with_triton.rst new
+6
@@ -0,0 +1,6 @@
1 +.. _deploying_with_triton:
2 +
3 +Deploying with NVIDIA Triton
4 +============================
5 +
6 +The `Triton Inference Server <https://github.com/triton-inference-server>`_ hosts a tutorial demonstrating how to quickly deploy a simple `facebook/opt-125m <https://huggingface.co/facebook/opt-125m>`_ model using vLLM. Please see `Deploying a vLLM model in Triton <https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/vLLM/README.md#deploying-a-vllm-model-in-triton>`_ for more details.