A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 14:45 UTC
Update openai_compatible_server.md (#16507)
Signed-off-by: Christian Sears <csears@redhat.com>
Christian Sears committed
Apr 11, 2025 at 18:54 UTC
c09632a66c0a4576518eef598aa13aa2d34bd337
1 file changed
+3
-3
docs/source/serving/openai_compatible_server.md
+3
-3
@@ -2,15 +2,15 @@
2
3
# OpenAI-Compatible Server
4
5
-vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more!
5
+vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client.
6
7
-You can start the server via the [`vllm serve`](#vllm-serve) command, or through [Docker](#deployment-docker):
7
+In your terminal, you can [install](../getting_started/installation.md) vLLM, then start the server with the [`vllm serve`](#vllm-serve) command. (You can also use our [Docker](#deployment-docker) image.)
8
9
```bash
10
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
11
```
12
13
-To call the server, you can use the [official OpenAI Python client](https://github.com/openai/openai-python), or any other HTTP client.
13
+To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).
14
15
```python
16
from openai import OpenAI