A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 22:00 UTC
add FAQ doc under 'serving' (#5946)
ning.zhang committed
Jul 1, 2024 at 14:11 UTC
83bdcb6ac32f22f20fd3a60ba67064a0b462801d
2 files changed
+13
docs/source/index.rst
+1
@@ -84,6 +84,7 @@ Documentation
84
serving/usage_stats
85
serving/integrations
86
serving/tensorizer
87
+ serving/faq
88
89
.. toctree::
90
:maxdepth: 1
docs/source/serving/faq.rst
new
+12
@@ -0,0 +1,12 @@
1
+Frequently Asked Questions
2
+========================
3
+
4
+ Q: How can I serve multiple models on a single port using the OpenAI API?
5
+
6
+A: Assuming that you're referring to using OpenAI compatible server to serve multiple models at once, that is not currently supported, you can run multiple instances of the server (each serving a different model) at the same time, and have another layer to route the incoming request to the correct server accordingly.
7
+
8
+----------------------------------------
9
+
10
+ Q: Which model to use for offline inference embedding?
11
+
12
+A: If you want to use an embedding model, try: https://huggingface.co/intfloat/e5-mistral-7b-instruct. Instead models, such as Llama-3-8b, Mistral-7B-Instruct-v0.3, are generation models rather than an embedding model
\ No newline at end of file