| 1 | # Frequently Asked Questions |
| 2 | |
| 3 | > Q: How can I serve multiple models on a single port using the OpenAI API? |
| 4 | |
| 5 | A: Assuming that you're referring to using OpenAI compatible server to serve multiple models at once, that is not currently supported, you can run multiple instances of the server (each serving a different model) at the same time, and have another layer to route the incoming request to the correct server accordingly. |
| 6 | |
| 7 | --- |
| 8 | |
| 9 | > Q: Which model to use for offline inference embedding? |
| 10 | |
| 11 | A: You can try [e5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) and [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5); |
| 12 | more are listed [here](../models/supported_models.md). |
| 13 | |
| 14 | By extracting hidden states, vLLM can automatically convert text generation models like [Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B), |
| 15 | [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) into embedding models, |
| 16 | but they are expected to be inferior to models that are specifically trained on embedding tasks. |
| 17 | |
| 18 | --- |
| 19 | |
| 20 | > Q: Can the output of a prompt vary across runs in vLLM? |
| 21 | |
| 22 | A: Yes, it can. vLLM does not guarantee stable log probabilities (logprobs) for the output tokens. Variations in logprobs may occur due to |
| 23 | numerical instability in Torch operations or non-deterministic behavior in batched Torch operations when batching changes. For more details, |
| 24 | see the [Numerical Accuracy section](https://pytorch.org/docs/stable/notes/numerical_accuracy.html#batched-computations-or-slice-computations). |
| 25 | |
| 26 | In vLLM, the same requests might be batched differently due to factors such as other concurrent requests, |
| 27 | changes in batch size, or batch expansion in speculative decoding. These batching variations, combined with numerical instability of Torch operations, |
| 28 | can lead to slightly different logit/logprob values at each step. Such differences can accumulate, potentially resulting in |
| 29 | different tokens being sampled. Once a different token is sampled, further divergence is likely. |
| 30 | |
| 31 | ## Mitigation Strategies |
| 32 | |
| 33 | - For improved stability and reduced variance, use `float32`. Note that this will require more memory. |
| 34 | - If using `bfloat16`, switching to `float16` can also help. |
| 35 | - Using request seeds can aid in achieving more stable generation for temperature > 0, but discrepancies due to precision differences may still occur. |