A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
Update openai_compatible_server.md (#11536)
Co-authored-by: Simon Mo <simon.mo@hey.com>
Robert Shaw committed
Dec 26, 2024 at 19:26 UTC
0c0c2015c526f1fe6f86fdd8d6bd99a935d2d275
1 file changed
+7
-1
docs/source/serving/openai_compatible_server.md
+7
-1
@@ -112,7 +112,13 @@ completion = client.chat.completions.create(
112
113
## Extra HTTP Headers
114
115
-Only `X-Request-Id` HTTP request header is supported for now.
115
+Only `X-Request-Id` HTTP request header is supported for now. It can be enabled
116
+with `--enable-request-id-headers`.
117
+
118
+> Note that enablement of the headers can impact performance significantly at high QPS
119
+> rates. We recommend implementing HTTP headers at the router level (e.g. via Istio),
120
+> rather than within the vLLM layer for this reason.
121
+> See https://github.com/vllm-project/vllm/pull/11529 for more details.
122
123
```python
124
completion = client.chat.completions.create(