@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 0c0c2015c5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 04:30 UTC

Update openai_compatible_server.md (#11536)

Co-authored-by: Simon Mo <simon.mo@hey.com>

Robert Shaw committed Dec 26, 2024 at 19:26 UTC 0c0c2015c526f1fe6f86fdd8d6bd99a935d2d275
1 file changed +7 -1
docs/source/serving/openai_compatible_server.md
+7 -1
@@ -112,7 +112,13 @@ completion = client.chat.completions.create(
112
113 ## Extra HTTP Headers
114
115 -Only `X-Request-Id` HTTP request header is supported for now.
115 +Only `X-Request-Id` HTTP request header is supported for now. It can be enabled
116 +with `--enable-request-id-headers`.
117 +
118 +> Note that enablement of the headers can impact performance significantly at high QPS
119 +> rates. We recommend implementing HTTP headers at the router level (e.g. via Istio),
120 +> rather than within the vLLM layer for this reason.
121 +> See https://github.com/vllm-project/vllm/pull/11529 for more details.
122
123 ```python
124 completion = client.chat.completions.create(