A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 18:30 UTC
Mention `extra_body` as a way top pass vLLM only parameters using the OpenAI client (#15240)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Mar 21, 2025 at 02:18 UTC
6edbfa924df0d1809028296d02dd328bd8139529
1 file changed
+5
docs/source/serving/openai_compatible_server.md
+5
@@ -29,6 +29,11 @@ completion = client.chat.completions.create(
29
print(completion.choices[0].message)
30
```
31
32
+:::{tip}
33
+vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
34
+You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`.
35
+:::
36
+
37
## Supported APIs
38
39
We currently support the following OpenAI APIs: