A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 09:45 UTC
[doc] add missing import (#18133)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
May 14, 2025 at 18:57 UTC
9ccc6ded425139b386edb6a268af66ede2082beb
1 file changed
+2
docs/source/serving/offline_inference.md
+2
@@ -74,6 +74,8 @@ Tensor parallelism (`tensor_parallel_size` option) can be used to split the mode
74
The following code splits the model across 2 GPUs.
75
76
```python
77
+from vllm import LLM
78
+
79
llm = LLM(model="ibm-granite/granite-3.1-8b-instruct",
80
tensor_parallel_size=2)
81
```