@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / d85c47d6ad
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

Replace "online inference" with "online serving" (#11923)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Jan 10, 2025 at 12:05 UTC d85c47d6ad24c286ae55fd9da231808b8ddd7a7f
11 files changed +16 -16
.buildkite/run-cpu-test.sh
+1 -1
@@ -61,7 +61,7 @@ function cpu_tests() {
61 pytest -s -v -k cpu_model \
62 tests/basic_correctness/test_chunked_prefill.py"
63
64 - # online inference
64 + # online serving
65 docker exec cpu-test-"$BUILDKITE_BUILD_NUMBER"-"$NUMA_NODE" bash -c "
66 set -e
67 export VLLM_CPU_KVCACHE_SPACE=10
docs/source/features/structured_outputs.md
+2 -2
@@ -5,7 +5,7 @@
5 vLLM supports the generation of structured outputs using [outlines](https://github.com/dottxt-ai/outlines), [lm-format-enforcer](https://github.com/noamgat/lm-format-enforcer), or [xgrammar](https://github.com/mlc-ai/xgrammar) as backends for the guided decoding.
6 This document shows you some examples of the different options that are available to generate structured outputs.
7
8 -## Online Inference (OpenAI API)
8 +## Online Serving (OpenAI API)
9
10 You can generate structured outputs using the OpenAI's [Completions](https://platform.openai.com/docs/api-reference/completions) and [Chat](https://platform.openai.com/docs/api-reference/chat) API.
11
@@ -239,7 +239,7 @@ The main available options inside `GuidedDecodingParams` are:
239 - `backend`
240 - `whitespace_pattern`
241
242 -These parameters can be used in the same way as the parameters from the Online Inference examples above.
242 +These parameters can be used in the same way as the parameters from the Online Serving examples above.
243 One example for the usage of the `choices` parameter is shown below:
244
245 ```python
docs/source/getting_started/installation/hpu-gaudi.md
+2 -2
@@ -83,7 +83,7 @@ $ python setup.py develop
83 ## Supported Features
84
85 - [Offline inference](#offline-inference)
86 -- Online inference via [OpenAI-Compatible Server](#openai-compatible-server)
86 +- Online serving via [OpenAI-Compatible Server](#openai-compatible-server)
87 - HPU autodetection - no need to manually select device within vLLM
88 - Paged KV cache with algorithms enabled for Intel Gaudi accelerators
89 - Custom Intel Gaudi implementations of Paged Attention, KV cache ops,
@@ -385,5 +385,5 @@ the below:
385 completely. With HPU Graphs disabled, you are trading latency and
386 throughput at lower batches for potentially higher throughput on
387 higher batches. You can do that by adding `--enforce-eager` flag to
388 - server (for online inference), or by passing `enforce_eager=True`
388 + server (for online serving), or by passing `enforce_eager=True`
389 argument to LLM constructor (for offline inference).
docs/source/getting_started/quickstart.md
+1 -1
@@ -5,7 +5,7 @@
5 This guide will help you quickly get started with vLLM to perform:
6
7 - [Offline batched inference](#quickstart-offline)
8 -- [Online inference using OpenAI-compatible server](#quickstart-online)
8 +- [Online serving using OpenAI-compatible server](#quickstart-online)
9
10 ## Prerequisites
11
docs/source/models/generative_models.md
+1 -1
@@ -118,7 +118,7 @@ print("Loaded chat template:", custom_template)
118 outputs = llm.chat(conversation, chat_template=custom_template)
119 ```
120
121 -## Online Inference
121 +## Online Serving
122
123 Our [OpenAI-Compatible Server](#openai-compatible-server) provides endpoints that correspond to the offline APIs:
124
docs/source/models/pooling_models.md
+1 -1
@@ -127,7 +127,7 @@ print(f"Score: {score}")
127
128 A code example can be found here: <gh-file:examples/offline_inference/offline_inference_scoring.py>
129
130 -## Online Inference
130 +## Online Serving
131
132 Our [OpenAI-Compatible Server](#openai-compatible-server) provides endpoints that correspond to the offline APIs:
133
docs/source/models/supported_models.md
+2 -2
@@ -552,7 +552,7 @@ See [this page](#multimodal-inputs) on how to pass multi-modal inputs to the mod
552
553 ````{important}
554 To enable multiple multi-modal items per text prompt, you have to set `limit_mm_per_prompt` (offline inference)
555 -or `--limit-mm-per-prompt` (online inference). For example, to enable passing up to 4 images per text prompt:
555 +or `--limit-mm-per-prompt` (online serving). For example, to enable passing up to 4 images per text prompt:
556
557 Offline inference:
558 ```python
@@ -562,7 +562,7 @@ llm = LLM(
562 )
563 ```
564
565 -Online inference:
565 +Online serving:
566 ```bash
567 vllm serve Qwen/Qwen2-VL-7B-Instruct --limit-mm-per-prompt image=4
568 ```
docs/source/serving/multimodal_inputs.md
+1 -1
@@ -199,7 +199,7 @@ for o in outputs:
199 print(generated_text)
200 ```
201
202 -## Online Inference
202 +## Online Serving
203
204 Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions API](https://platform.openai.com/docs/api-reference/chat).
205
examples/online_serving/openai_chat_completion_client_for_multimodal.py
+2 -2
@@ -1,5 +1,5 @@
1 """An example showing how to use vLLM to serve multimodal models
2 -and run online inference with OpenAI client.
2 +and run online serving with OpenAI client.
3
4 Launch the vLLM server with the following command:
5
@@ -309,7 +309,7 @@ def main(args) -> None:
309
310 if __name__ == "__main__":
311 parser = FlexibleArgumentParser(
312 - description='Demo on using OpenAI client for online inference with '
312 + description='Demo on using OpenAI client for online serving with '
313 'multimodal language models served with vLLM.')
314 parser.add_argument('--chat-type',
315 '-c',
tests/models/decoder_only/audio_language/test_ultravox.py
+2 -2
@@ -237,8 +237,8 @@ def test_models_with_multiple_audios(vllm_runner, audio_assets, dtype: str,
237
238
239 @pytest.mark.asyncio
240 -async def test_online_inference(client, audio_assets):
241 - """Exercises online inference with/without chunked prefill enabled."""
240 +async def test_online_serving(client, audio_assets):
241 + """Exercises online serving with/without chunked prefill enabled."""
242
243 messages = [{
244 "role":
vllm/model_executor/models/molmo.py
+1 -1
@@ -1068,7 +1068,7 @@ def input_processor_for_molmo(ctx: InputContext, inputs: DecoderOnlyInputs):
1068 trust_remote_code=model_config.trust_remote_code)
1069
1070 # NOTE: message formatting for raw text prompt is only applied for
1071 - # offline inference; for online inference, the prompt is always in
1071 + # offline inference; for online serving, the prompt is always in
1072 # instruction format and tokenized.
1073 if prompt is not None and re.match(r"^User:[\s\S]*?(Assistant:)*$",
1074 prompt):