| 1 | # OpenAI-Compatible Server |
| 2 | |
| 3 | vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client. |
| 4 | |
| 5 | !!! warning "API key authentication does not protect every endpoint" |
| 6 | The `--api-key` option (or `VLLM_API_KEY` environment variable) only |
| 7 | authenticates requests to endpoints under the `/v1`, `/v2`, and |
| 8 | `/inference` path prefixes. Other endpoints on the same HTTP server are |
| 9 | **not** authenticated — most notably `/invocations`, which exposes the same |
| 10 | inference capabilities as the `/v1` endpoints. Do not rely on `--api-key` |
| 11 | alone to secure vLLM. See |
| 12 | [API Key Authentication Limitations](../../usage/security.md#api-key-authentication-limitations) |
| 13 | for the full list of protected and unprotected endpoints and recommended |
| 14 | hardening, such as deploying behind a reverse proxy. |
| 15 | |
| 16 | ## Supported APIs |
| 17 | |
| 18 | We currently support the following OpenAI APIs: |
| 19 | |
| 20 | - [Completions API](#completions-api) (`/v1/completions`) |
| 21 | - Only applicable to [text generation models](../../models/generative_models.md). |
| 22 | - *Note: `suffix` parameter is not supported.* |
| 23 | - [Chat Completions API](#chat-api) (`/v1/chat/completions`) |
| 24 | - Only applicable to [text generation models](../../models/generative_models.md) with a [chat template](../online_serving/README.md#chat-template). |
| 25 | - *Note: `user` parameter is ignored.* |
| 26 | - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls. |
| 27 | - [Chat Completions batch API](#chat-api) (`/v1/chat/completions/batch`) |
| 28 | - [Responses API](#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`) |
| 29 | - Only applicable to [text generation models](../../models/generative_models.md). |
| 30 | - [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) |
| 31 | - Only applicable to [embedding models](../../models/pooling_models/embed.md). |
| 32 | - [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`) |
| 33 | - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). |
| 34 | - [Translation API](./speech_to_text.md#translations-api) (`/v1/audio/translations`) |
| 35 | - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). |
| 36 | |
| 37 | ## Completions API |
| 38 | |
| 39 | In your terminal, you can [install](../../getting_started/installation/README.md) vLLM, then start the server with the [`vllm serve`](../../configuration/serve_args.md) command. (You can also use our [Docker](../../deployment/docker.md) image.) |
| 40 | |
| 41 | ```bash |
| 42 | vllm serve NousResearch/Meta-Llama-3-8B-Instruct \ |
| 43 | --dtype auto \ |
| 44 | --api-key token-abc123 |
| 45 | ``` |
| 46 | |
| 47 | To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python). |
| 48 | |
| 49 | ??? code |
| 50 | |
| 51 | ```python |
| 52 | from openai import OpenAI |
| 53 | client = OpenAI( |
| 54 | base_url="http://localhost:8000/v1", |
| 55 | api_key="token-abc123", |
| 56 | ) |
| 57 | |
| 58 | completion = client.chat.completions.create( |
| 59 | model="NousResearch/Meta-Llama-3-8B-Instruct", |
| 60 | messages=[ |
| 61 | {"role": "user", "content": "Hello!"}, |
| 62 | ], |
| 63 | ) |
| 64 | |
| 65 | print(completion.choices[0].message) |
| 66 | ``` |
| 67 | |
| 68 | !!! tip |
| 69 | vLLM supports some parameters that are not supported by OpenAI, `top_k` for example. |
| 70 | You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`. |
| 71 | |
| 72 | !!! important |
| 73 | By default, the server applies `generation_config.json` from the Hugging Face model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator. |
| 74 | |
| 75 | To disable this behavior, please pass `--generation-config vllm` when launching the server. |
| 76 | |
| 77 | ## Extra Parameters |
| 78 | |
| 79 | vLLM supports a set of parameters that are not part of the OpenAI API. |
| 80 | In order to use them, you can pass them as extra parameters in the OpenAI client. |
| 81 | Or directly merge them into the JSON payload if you are using HTTP call directly. |
| 82 | |
| 83 | ```python |
| 84 | completion = client.chat.completions.create( |
| 85 | model="NousResearch/Meta-Llama-3-8B-Instruct", |
| 86 | messages=[ |
| 87 | {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}, |
| 88 | ], |
| 89 | extra_body={ |
| 90 | "structured_outputs": {"choice": ["positive", "negative"]}, |
| 91 | }, |
| 92 | ) |
| 93 | ``` |
| 94 | |
| 95 | ## Extra HTTP Headers |
| 96 | |
| 97 | The `X-Request-Id` HTTP request header can be enabled with |
| 98 | `--enable-request-id-headers`. |
| 99 | |
| 100 | ??? code |
| 101 | |
| 102 | ```python |
| 103 | completion = client.chat.completions.create( |
| 104 | model="NousResearch/Meta-Llama-3-8B-Instruct", |
| 105 | messages=[ |
| 106 | {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}, |
| 107 | ], |
| 108 | extra_headers={ |
| 109 | "x-request-id": "sentiment-classification-00001", |
| 110 | }, |
| 111 | ) |
| 112 | print(completion._request_id) |
| 113 | |
| 114 | completion = client.completions.create( |
| 115 | model="NousResearch/Meta-Llama-3-8B-Instruct", |
| 116 | prompt="A robot may not injure a human being", |
| 117 | extra_headers={ |
| 118 | "x-request-id": "completion-test", |
| 119 | }, |
| 120 | ) |
| 121 | print(completion._request_id) |
| 122 | ``` |
| 123 | |
| 124 | The Completions, Chat Completions, and Responses APIs also support the |
| 125 | `X-Vllm-Priority` request header. Its value must be an integer and overrides |
| 126 | the `priority` value in the JSON request body. Non-zero priorities require the |
| 127 | server to use priority scheduling. |
| 128 | |
| 129 | ```python |
| 130 | completion = client.chat.completions.create( |
| 131 | model="NousResearch/Meta-Llama-3-8B-Instruct", |
| 132 | messages=[{"role": "user", "content": "Hello!"}], |
| 133 | extra_headers={"X-Vllm-Priority": "-10"}, |
| 134 | ) |
| 135 | ``` |
| 136 | |
| 137 | ## API Reference |
| 138 | |
| 139 | ### Completions API |
| 140 | |
| 141 | Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions); |
| 142 | you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. |
| 143 | |
| 144 | Code example: [examples/basic/online_serving/openai_completion_client.py](../../../examples/basic/online_serving/openai_completion_client.py) |
| 145 | |
| 146 | #### Extra parameters |
| 147 | |
| 148 | The following [sampling parameters](../../api/README.md#inference-parameters) are supported. |
| 149 | |
| 150 | ??? code |
| 151 | |
| 152 | ```python |
| 153 | --8<-- "vllm/entrypoints/openai/completion/protocol.py:completion-sampling-params" |
| 154 | ``` |
| 155 | |
| 156 | The following extra parameters are supported: |
| 157 | |
| 158 | ??? code |
| 159 | |
| 160 | ```python |
| 161 | --8<-- "vllm/entrypoints/openai/completion/protocol.py:completion-extra-params" |
| 162 | ``` |
| 163 | |
| 164 | ### Chat API |
| 165 | |
| 166 | Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat); |
| 167 | you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. |
| 168 | |
| 169 | We support both [Vision](https://platform.openai.com/docs/guides/vision)- and |
| 170 | [Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters; |
| 171 | see our [Multimodal Inputs](../../features/multimodal_inputs.md) guide for more information. |
| 172 | |
| 173 | - *Note: `image_url.detail` parameter is not supported.* |
| 174 | |
| 175 | Code example: [examples/basic/online_serving/openai_chat_completion_client.py](../../../examples/basic/online_serving/openai_chat_completion_client.py) |
| 176 | |
| 177 | #### Extra parameters |
| 178 | |
| 179 | The following [sampling parameters](../../api/README.md#inference-parameters) are supported. |
| 180 | |
| 181 | ??? code |
| 182 | |
| 183 | ```python |
| 184 | --8<-- "vllm/entrypoints/openai/chat_completion/protocol.py:chat-completion-sampling-params" |
| 185 | ``` |
| 186 | |
| 187 | The following extra parameters are supported: |
| 188 | |
| 189 | ??? code |
| 190 | |
| 191 | ```python |
| 192 | --8<-- "vllm/entrypoints/openai/chat_completion/protocol.py:chat-completion-extra-params" |
| 193 | ``` |
| 194 | |
| 195 | ### Responses API |
| 196 | |
| 197 | Our Responses API is compatible with [OpenAI's Responses API](https://platform.openai.com/docs/api-reference/responses); |
| 198 | you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. |
| 199 | |
| 200 | Code example: [examples/tool_calling/openai_responses_client_with_tools.py](../../../examples/tool_calling/openai_responses_client_with_tools.py) |
| 201 | |
| 202 | To get prompt token IDs before generation, use |
| 203 | [`/v1/responses/render`](renderer.md#get-responses-prompt-token-ids). It preprocesses |
| 204 | a self-contained Responses request without inference and returns the prompt |
| 205 | token IDs, so a separate `/tokenize` call is not needed. |
| 206 | |
| 207 | #### Extra parameters |
| 208 | |
| 209 | The following extra parameters in the request object are supported: |
| 210 | |
| 211 | ??? code |
| 212 | |
| 213 | ```python |
| 214 | --8<-- "vllm/entrypoints/openai/responses/protocol.py:responses-extra-params" |
| 215 | ``` |
| 216 | |
| 217 | The following extra parameters in the response object are supported: |
| 218 | |
| 219 | ??? code |
| 220 | |
| 221 | ```python |
| 222 | --8<-- "vllm/entrypoints/openai/responses/protocol.py:responses-response-extra-params" |
| 223 | ``` |