@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 83bdcb6ac3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 22:00 UTC

add FAQ doc under 'serving' (#5946)

ning.zhang committed Jul 1, 2024 at 14:11 UTC 83bdcb6ac32f22f20fd3a60ba67064a0b462801d
2 files changed +13
docs/source/index.rst
+1
@@ -84,6 +84,7 @@ Documentation
84 serving/usage_stats
85 serving/integrations
86 serving/tensorizer
87 + serving/faq
88
89 .. toctree::
90 :maxdepth: 1
docs/source/serving/faq.rst new
+12
@@ -0,0 +1,12 @@
1 +Frequently Asked Questions
2 +========================
3 +
4 + Q: How can I serve multiple models on a single port using the OpenAI API?
5 +
6 +A: Assuming that you're referring to using OpenAI compatible server to serve multiple models at once, that is not currently supported, you can run multiple instances of the server (each serving a different model) at the same time, and have another layer to route the incoming request to the correct server accordingly.
7 +
8 +----------------------------------------
9 +
10 + Q: Which model to use for offline inference embedding?
11 +
12 +A: If you want to use an embedding model, try: https://huggingface.co/intfloat/e5-mistral-7b-instruct. Instead models, such as Llama-3-8b, Mistral-7B-Instruct-v0.3, are generation models rather than an embedding model
\ No newline at end of file