A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 2, 22:00 UTC
[Docs] Reorganize offline inference docs. (#43552)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
wang.yuqi committed
May 25, 2026 at 13:44 UTC
1b26fa361e7aa459951d08b315533803862f04d2
1 file changed
+67
-13
docs/serving/offline_inference.md
+67
-13
@@ -2,24 +2,78 @@
2
3
Offline inference is possible in your own code using vLLM's [`LLM`][vllm.LLM] class.
4
5
-For example, the following code downloads the [`facebook/opt-125m`](https://huggingface.co/facebook/opt-125m) model from HuggingFace
6
-and runs it in vLLM using the default configuration.
5
+## Model Types
6
8
-```python
9
-from vllm import LLM
7
+vLLM models can be categorized into two types:
8
11
-# Initialize the vLLM engine.
12
-llm = LLM(model="facebook/opt-125m")
13
-```
9
+- **[Generative Models](../models/supported_models.md)** - Models that produce text completions or chat responses (e.g., LLaMA, Qwen, DeepSeek). Use `LLM.generate()` and `LLM.chat()` for these models.
10
15
-After initializing the `LLM` instance, use the available APIs to perform model inference.
16
-The available APIs depend on the model type:
11
+- **[Pooling Models](../models/pooling_models/README.md)** - These models do not generate content. They are primarily used for classification and retrieval tasks, such as bge-m3 and Qwen3 Reranker.
12
18
-- [Generative models](../models/generative_models.md) output logprobs which are sampled from to obtain the final output text.
19
-- [Pooling models](../models/pooling_models/README.md) output their hidden states directly.
13
+## Generative APIs
14
21
-!!! info
22
- [API Reference](../api/README.md#offline-inference)
15
+For further details on generative models, please refer to [this page](../models/supported_models.md).
16
+
17
+- `LLM.generate` - Generates completions for the given input prompts.
18
+- `LLM.chat` - Generates responses for a chat conversation.
19
+
20
+## Asynchronous Queue APIs
21
+
22
+- `LLM.enqueue` - Enqueues prompts for generation without waiting for completion.
23
+- `LLM.enqueue_chat` - Enqueues chat conversations for generation without waiting.
24
+- `LLM.wait_for_completion` - Waits for all enqueued requests to complete and returns results.
25
+
26
+## Pooling APIs
27
+
28
+For further details on pooling models, please refer to [this page](../models/pooling_models/README.md).
29
+
30
+- `LLM.classify` - Only applicable to [classification models](../models/pooling_models/classify.md).
31
+- `LLM.embed` - Only applicable to [embedding models](../models/pooling_models/embed.md).
32
+- `LLM.score` - Applicable to [score models](../models/pooling_models/scoring.md) (cross-encoder, bi-encoder, late-interaction).
33
+- `LLM.encode` - Applicable to all [pooling models](../models/pooling_models/README.md).
34
+
35
+## Profiling APIs
36
+
37
+For further details on profiling, please refer to [this page](../contributing/profiling.md).
38
+
39
+- `LLM.start_profile` - Starts profiling with an optional custom trace prefix.
40
+- `LLM.stop_profile` - Stops the ongoing profiling session.
41
+
42
+## Sleep Mode APIs
43
+
44
+For further details on sleep mode, please refer to [this page](../features/sleep_mode.md).
45
+
46
+- `LLM.sleep` - Puts the engine into sleep mode.
47
+- `LLM.wake_up` - Wakes up the engine from sleep mode.
48
+
49
+## Cache Management APIs
50
+
51
+- `LLM.reset_mm_cache` - Resets the multi-modal cache.
52
+- `LLM.reset_prefix_cache` - Resets the prefix cache.
53
+
54
+## Metrics APIs
55
+
56
+For further details on metrics, please refer to [this page](../design/metrics.md).
57
+
58
+- `LLM.get_metrics` - Returns a snapshot of aggregated metrics from Prometheus.
59
+
60
+## Weight Transfer APIs (RL Training)
61
+
62
+For further details on Weight Transfer, please refer to [this page](../training/weight_transfer/README.md).
63
+
64
+- `LLM.init_weight_transfer_engine` - Initializes the weight transfer engine for RL training.
65
+- `LLM.start_weight_update` - Starts a new weight update cycle.
66
+- `LLM.update_weights` - Updates the model weights.
67
+- `LLM.finish_weight_update` - Finishes the current weight update cycle.
68
+
69
+## Additional APIs
70
+
71
+- `LLM.collective_rpc` - Executes a method or callable collectively across all workers.
72
+- `LLM.apply_model` - Applies a function directly to the model inside each worker.
73
+
74
+## API Reference
75
+
76
+[Offline Inference](../api/README.md#offline-inference)
77
78
## Ray Data LLM API
79