A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
Replace "online inference" with "online serving" (#11923)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Jan 10, 2025 at 12:05 UTC
d85c47d6ad24c286ae55fd9da231808b8ddd7a7f
11 files changed
+16
-16
.buildkite/run-cpu-test.sh
+1
-1
@@ -61,7 +61,7 @@ function cpu_tests() {
61
pytest -s -v -k cpu_model \
62
tests/basic_correctness/test_chunked_prefill.py"
63
64
- # online inference
64
+ # online serving
65
docker exec cpu-test-"$BUILDKITE_BUILD_NUMBER"-"$NUMA_NODE" bash -c "
66
set -e
67
export VLLM_CPU_KVCACHE_SPACE=10
docs/source/features/structured_outputs.md
+2
-2
@@ -5,7 +5,7 @@
5
vLLM supports the generation of structured outputs using [outlines](https://github.com/dottxt-ai/outlines), [lm-format-enforcer](https://github.com/noamgat/lm-format-enforcer), or [xgrammar](https://github.com/mlc-ai/xgrammar) as backends for the guided decoding.
6
This document shows you some examples of the different options that are available to generate structured outputs.
7
8
-## Online Inference (OpenAI API)
8
+## Online Serving (OpenAI API)
9
10
You can generate structured outputs using the OpenAI's [Completions](https://platform.openai.com/docs/api-reference/completions) and [Chat](https://platform.openai.com/docs/api-reference/chat) API.
11
@@ -239,7 +239,7 @@ The main available options inside `GuidedDecodingParams` are:
239
- `backend`
240
- `whitespace_pattern`
241
242
-These parameters can be used in the same way as the parameters from the Online Inference examples above.
242
+These parameters can be used in the same way as the parameters from the Online Serving examples above.
243
One example for the usage of the `choices` parameter is shown below:
244
245
```python
docs/source/getting_started/installation/hpu-gaudi.md
+2
-2
@@ -83,7 +83,7 @@ $ python setup.py develop
83
## Supported Features
84
85
- [Offline inference](#offline-inference)
86
-- Online inference via [OpenAI-Compatible Server](#openai-compatible-server)
86
+- Online serving via [OpenAI-Compatible Server](#openai-compatible-server)
87
- HPU autodetection - no need to manually select device within vLLM
88
- Paged KV cache with algorithms enabled for Intel Gaudi accelerators
89
- Custom Intel Gaudi implementations of Paged Attention, KV cache ops,
@@ -385,5 +385,5 @@ the below:
385
completely. With HPU Graphs disabled, you are trading latency and
386
throughput at lower batches for potentially higher throughput on
387
higher batches. You can do that by adding `--enforce-eager` flag to
388
- server (for online inference), or by passing `enforce_eager=True`
388
+ server (for online serving), or by passing `enforce_eager=True`
389
argument to LLM constructor (for offline inference).
docs/source/getting_started/quickstart.md
+1
-1
@@ -5,7 +5,7 @@
5
This guide will help you quickly get started with vLLM to perform:
6
7
- [Offline batched inference](#quickstart-offline)
8
-- [Online inference using OpenAI-compatible server](#quickstart-online)
8
+- [Online serving using OpenAI-compatible server](#quickstart-online)
9
10
## Prerequisites
11
docs/source/models/generative_models.md
+1
-1
@@ -118,7 +118,7 @@ print("Loaded chat template:", custom_template)
118
outputs = llm.chat(conversation, chat_template=custom_template)
119
```
120
121
-## Online Inference
121
+## Online Serving
122
123
Our [OpenAI-Compatible Server](#openai-compatible-server) provides endpoints that correspond to the offline APIs:
124
docs/source/models/pooling_models.md
+1
-1
@@ -127,7 +127,7 @@ print(f"Score: {score}")
127
128
A code example can be found here: <gh-file:examples/offline_inference/offline_inference_scoring.py>
129
130
-## Online Inference
130
+## Online Serving
131
132
Our [OpenAI-Compatible Server](#openai-compatible-server) provides endpoints that correspond to the offline APIs:
133
docs/source/models/supported_models.md
+2
-2
@@ -552,7 +552,7 @@ See [this page](#multimodal-inputs) on how to pass multi-modal inputs to the mod
552
553
````{important}
554
To enable multiple multi-modal items per text prompt, you have to set `limit_mm_per_prompt` (offline inference)
555
-or `--limit-mm-per-prompt` (online inference). For example, to enable passing up to 4 images per text prompt:
555
+or `--limit-mm-per-prompt` (online serving). For example, to enable passing up to 4 images per text prompt:
556
557
Offline inference:
558
```python
@@ -562,7 +562,7 @@ llm = LLM(
562
)
563
```
564
565
-Online inference:
565
+Online serving:
566
```bash
567
vllm serve Qwen/Qwen2-VL-7B-Instruct --limit-mm-per-prompt image=4
568
```
docs/source/serving/multimodal_inputs.md
+1
-1
@@ -199,7 +199,7 @@ for o in outputs:
199
print(generated_text)
200
```
201
202
-## Online Inference
202
+## Online Serving
203
204
Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions API](https://platform.openai.com/docs/api-reference/chat).
205
examples/online_serving/openai_chat_completion_client_for_multimodal.py
+2
-2
@@ -1,5 +1,5 @@
1
"""An example showing how to use vLLM to serve multimodal models
2
-and run online inference with OpenAI client.
2
+and run online serving with OpenAI client.
3
4
Launch the vLLM server with the following command:
5
@@ -309,7 +309,7 @@ def main(args) -> None:
309
310
if __name__ == "__main__":
311
parser = FlexibleArgumentParser(
312
- description='Demo on using OpenAI client for online inference with '
312
+ description='Demo on using OpenAI client for online serving with '
313
'multimodal language models served with vLLM.')
314
parser.add_argument('--chat-type',
315
'-c',
tests/models/decoder_only/audio_language/test_ultravox.py
+2
-2
@@ -237,8 +237,8 @@ def test_models_with_multiple_audios(vllm_runner, audio_assets, dtype: str,
237
238
239
@pytest.mark.asyncio
240
-async def test_online_inference(client, audio_assets):
241
- """Exercises online inference with/without chunked prefill enabled."""
240
+async def test_online_serving(client, audio_assets):
241
+ """Exercises online serving with/without chunked prefill enabled."""
242
243
messages = [{
244
"role":
vllm/model_executor/models/molmo.py
+1
-1
@@ -1068,7 +1068,7 @@ def input_processor_for_molmo(ctx: InputContext, inputs: DecoderOnlyInputs):
1068
trust_remote_code=model_config.trust_remote_code)
1069
1070
# NOTE: message formatting for raw text prompt is only applied for
1071
- # offline inference; for online inference, the prompt is always in
1071
+ # offline inference; for online serving, the prompt is always in
1072
# instruction format and tokenized.
1073
if prompt is not None and re.match(r"^User:[\s\S]*?(Assistant:)*$",
1074
prompt):