af107d5a0e · tduncan/Nvidia-Optimization-Stack-VLLM

docs/cli/README.md

+1 -1

@@ -16,7 +16,7 @@ vllm {chat,complete,serve,bench,collect-env,run-batch}

16

17 Start the vLLM OpenAI Compatible API server.

18

19 -??? Examples

19 +??? console "Examples"

20

21 ```bash

22 # Start with a model

docs/configuration/conserving_memory.md

+2 -2

@@ -57,7 +57,7 @@ By default, we optimize model inference using CUDA graphs which take up extra me

57

58 You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage:

59

60 -??? Code

60 +??? code

61

62 ```python

63 from vllm import LLM

@@ -129,7 +129,7 @@ reduce the size of the processed multi-modal inputs, which in turn saves memory.

129

130 Here are some examples:

131

132 -??? Code

132 +??? code

133

134 ```python

135 from vllm import LLM

docs/configuration/env_vars.md

+1 -1

@@ -7,7 +7,7 @@ vLLM uses the following environment variables to configure the system:

7

8 All environment variables used by vLLM are prefixed with `VLLM_`. **Special care should be taken for Kubernetes users**: please do not name the service as `vllm`, otherwise environment variables set by Kubernetes might conflict with vLLM's environment variables, because [Kubernetes sets environment variables for each service with the capitalized service name as the prefix](https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables).

9

10 -??? Code

10 +??? code

11

12 ```python

13 --8<-- "vllm/envs.py:env-vars-definition"

docs/contributing/README.md

+1 -1

@@ -95,7 +95,7 @@ For additional features and advanced configurations, refer to the official [MkDo

95

96 ## Testing

97

98 -??? note "Commands"

98 +??? console "Commands"

99

100 ```bash

101 pip install -r requirements/dev.txt

docs/contributing/model/basic.md

+1 -1

@@ -27,7 +27,7 @@ All vLLM modules within the model must include a `prefix` argument in their cons

27

28 The initialization code should look like this:

29

30 -??? Code

30 +??? code

31

32 ```python

33 from torch import nn

docs/contributing/model/multimodal.md

+20 -20

@@ -12,7 +12,7 @@ Further update the model as follows:

12

13 - Implement [get_placeholder_str][vllm.model_executor.models.interfaces.SupportsMultiModal.get_placeholder_str] to define the placeholder string which is used to represent the multi-modal item in the text prompt. This should be consistent with the chat template of the model.

14

15 - ??? Code

15 + ??? code

16

17 ```python

18 class YourModelForImage2Seq(nn.Module):

@@ -41,7 +41,7 @@ Further update the model as follows:

41

42 - Implement [get_multimodal_embeddings][vllm.model_executor.models.interfaces.SupportsMultiModal.get_multimodal_embeddings] that returns the embeddings from running the multimodal inputs through the multimodal tokenizer of the model. Below we provide a boilerplate of a typical implementation pattern, but feel free to adjust it to your own needs.

43

44 - ??? Code

44 + ??? code

45

46 ```python

47 class YourModelForImage2Seq(nn.Module):

@@ -71,7 +71,7 @@ Further update the model as follows:

71

72 - Implement [get_input_embeddings][vllm.model_executor.models.interfaces.SupportsMultiModal.get_input_embeddings] to merge `multimodal_embeddings` with text embeddings from the `input_ids`. If input processing for the model is implemented correctly (see sections below), then you can leverage the utility function we provide to easily merge the embeddings.

73

74 - ??? Code

74 + ??? code

75

76 ```python

77 from .utils import merge_multimodal_embeddings

@@ -155,7 +155,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

155

156 Looking at the code of HF's `LlavaForConditionalGeneration`:

157

158 - ??? Code

158 + ??? code

159

160 ```python

161 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L530-L544

@@ -179,7 +179,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

179 The number of placeholder feature tokens per image is `image_features.shape[1]`.

180 `image_features` is calculated inside the `get_image_features` method:

181

182 - ??? Code

182 + ??? code

183

184 ```python

185 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L290-L300

@@ -217,7 +217,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

217

218 To find the sequence length, we turn to the code of `CLIPVisionEmbeddings`:

219

220 - ??? Code

220 + ??? code

221

222 ```python

223 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L247-L257

@@ -244,7 +244,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

244

245 Overall, the number of placeholder feature tokens for an image can be calculated as:

246

247 - ??? Code

247 + ??? code

248

249 ```python

250 def get_num_image_tokens(

@@ -269,7 +269,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

269 Notice that the number of image tokens doesn't depend on the image width and height.

270 We can simply use a dummy `image_size` to calculate the multimodal profiling data:

271

272 - ??? Code

272 + ??? code

273

274 ```python

275 # NOTE: In actuality, this is usually implemented as part of the

@@ -314,7 +314,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

314

315 Looking at the code of HF's `FuyuForCausalLM`:

316

317 - ??? Code

317 + ??? code

318

319 ```python

320 # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/modeling_fuyu.py#L311-L322

@@ -344,7 +344,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

344 In `FuyuImageProcessor.preprocess`, the images are resized and padded to the target `FuyuImageProcessor.size`,

345 returning the dimensions after resizing (but before padding) as metadata.

346

347 - ??? Code

347 + ??? code

348

349 ```python

350 # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L541-L544

@@ -382,7 +382,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

382

383 In `FuyuImageProcessor.preprocess_with_tokenizer_info`, the images are split into patches based on this metadata:

384

385 - ??? Code

385 + ??? code

386

387 ```python

388 # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L425

@@ -420,7 +420,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

420

421 The number of patches is in turn defined by `FuyuImageProcessor.get_num_patches`:

422

423 - ??? Code

423 + ??? code

424

425 ```python

426 # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L552-L562

@@ -457,7 +457,7 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

457

458 For the multimodal image profiling data, the logic is very similar to LLaVA:

459

460 - ??? Code

460 + ??? code

461

462 ```python

463 def get_dummy_mm_data(

@@ -546,7 +546,7 @@ return a schema of the tensors outputted by the HF processor that are related to

546 In order to support the use of [MultiModalFieldConfig.batched][] like in LLaVA,

547 we remove the extra batch dimension by overriding [BaseMultiModalProcessor._call_hf_processor][]:

548

549 - ??? Code

549 + ??? code

550

551 ```python

552 def _call_hf_processor(

@@ -623,7 +623,7 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies

623 It simply repeats each input `image_token` a number of times equal to the number of placeholder feature tokens (`num_image_tokens`).

624 Based on this, we override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] as follows:

625

626 - ??? Code

626 + ??? code

627

628 ```python

629 def _get_prompt_updates(

@@ -668,7 +668,7 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies

668

669 We define a helper function to return `ncols` and `nrows` directly:

670

671 - ??? Code

671 + ??? code

672

673 ```python

674 def get_image_feature_grid_size(

@@ -698,7 +698,7 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies

698

699 Based on this, we can initially define our replacement tokens as:

700

701 - ??? Code

701 + ??? code

702

703 ```python

704 def get_replacement(item_idx: int):

@@ -718,7 +718,7 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies

718 However, this is not entirely correct. After `FuyuImageProcessor.preprocess_with_tokenizer_info` is called,

719 a BOS token (`<s>`) is also added to the promopt:

720

721 - ??? Code

721 + ??? code

722

723 ```python

724 # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L435

@@ -745,7 +745,7 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies

745 To assign the vision embeddings to only the image tokens, instead of a string

746 you can return an instance of [PromptUpdateDetails][vllm.multimodal.processing.PromptUpdateDetails]:

747

748 - ??? Code

748 + ??? code

749

750 ```python

751 hf_config = self.info.get_hf_config()

@@ -772,7 +772,7 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies

772 Finally, noticing that the HF processor removes the `|ENDOFTEXT|` token from the tokenized prompt,

773 we can search for it to conduct the replacement at the start of the string:

774

775 - ??? Code

775 + ??? code

776

777 ```python

778 def _get_prompt_updates(

docs/contributing/profiling.md

+1 -1

@@ -125,7 +125,7 @@ to manually kill the profiler and generate your `nsys-rep` report.

125

126 You can view these profiles either as summaries in the CLI, using `nsys stats [profile-file]`, or in the GUI by installing Nsight [locally following the directions here](https://developer.nvidia.com/nsight-systems/get-started).

127

128 -??? CLI example

128 +??? console "CLI example"

129

130 ```bash

131 nsys stats report1.nsys-rep

docs/deployment/docker.md

+1 -1

@@ -97,7 +97,7 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--

97 flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits.

98 Keep an eye on memory usage with parallel jobs as it can be substantial (see example below).

99

100 -??? Command

100 +??? console "Command"

101

102 ```bash

103 # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)

docs/deployment/frameworks/autogen.md

+1 -1

@@ -30,7 +30,7 @@ python -m vllm.entrypoints.openai.api_server \

30

31 - Call it with AutoGen:

32

33 -??? Code

33 +??? code

34

35 ```python

36 import asyncio

docs/deployment/frameworks/cerebrium.md

+3 -3

@@ -34,7 +34,7 @@ vllm = "latest"

34

35 Next, let us add our code to handle inference for the LLM of your choice (`mistralai/Mistral-7B-Instruct-v0.1` for this example), add the following code to your `main.py`:

36

37 -??? Code

37 +??? code

38

39 ```python

40 from vllm import LLM, SamplingParams

@@ -64,7 +64,7 @@ cerebrium deploy

64

65 If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case`/run`)

66

67 -??? Command

67 +??? console "Command"

68

69 ```python

70 curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \

@@ -82,7 +82,7 @@ If successful, you should be returned a CURL command that you can call inference

82

83 You should get a response like:

84

85 -??? Response

85 +??? console "Response"

86

87 ```python

88 {

docs/deployment/frameworks/dstack.md

+3 -3

@@ -26,7 +26,7 @@ dstack init

26

27 Next, to provision a VM instance with LLM of your choice (`NousResearch/Llama-2-7b-chat-hf` for this example), create the following `serve.dstack.yml` file for the dstack `Service`:

28

29 -??? Config

29 +??? code "Config"

30

31 ```yaml

32 type: service

@@ -48,7 +48,7 @@ Next, to provision a VM instance with LLM of your choice (`NousResearch/Llama-2-

48

49 Then, run the following CLI for provisioning:

50

51 -??? Command

51 +??? console "Command"

52

53 ```console

54 $ dstack run . -f serve.dstack.yml

@@ -79,7 +79,7 @@ Then, run the following CLI for provisioning:

79

80 After the provisioning, you can interact with the model by using the OpenAI SDK:

81

82 -??? Code

82 +??? code

83

84 ```python

85 from openai import OpenAI

docs/deployment/frameworks/haystack.md

+1 -1

@@ -27,7 +27,7 @@ vllm serve mistralai/Mistral-7B-Instruct-v0.1

27

28 - Use the `OpenAIGenerator` and `OpenAIChatGenerator` components in Haystack to query the vLLM server.

29

30 -??? Code

30 +??? code

31

32 ```python

33 from haystack.components.generators.chat import OpenAIChatGenerator

docs/deployment/frameworks/litellm.md

+1 -1

@@ -34,7 +34,7 @@ vllm serve qwen/Qwen1.5-0.5B-Chat

34

35 - Call it with litellm:

36

37 -??? Code

37 +??? code

38

39 ```python

40 import litellm

docs/deployment/frameworks/lws.md

+2 -2

@@ -17,7 +17,7 @@ vLLM can be deployed with [LWS](https://github.com/kubernetes-sigs/lws) on Kuber

17

18 Deploy the following yaml file `lws.yaml`

19

20 -??? Yaml

20 +??? code "Yaml"

21

22 ```yaml

23 apiVersion: leaderworkerset.x-k8s.io/v1

@@ -177,7 +177,7 @@ curl http://localhost:8080/v1/completions \

177

178 The output should be similar to the following

179

180 -??? Output

180 +??? console "Output"

181

182 ```text

183 {

docs/deployment/frameworks/skypilot.md

+6 -6

@@ -24,7 +24,7 @@ sky check

24

25 See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml).

26

27 -??? Yaml

27 +??? code "Yaml"

28

29 ```yaml

30 resources:

@@ -95,7 +95,7 @@ HF_TOKEN="your-huggingface-token" \

95

96 SkyPilot can scale up the service to multiple service replicas with built-in autoscaling, load-balancing and fault-tolerance. You can do it by adding a services section to the YAML file.

97

98 -??? Yaml

98 +??? code "Yaml"

99

100 ```yaml

101 service:

@@ -111,7 +111,7 @@ SkyPilot can scale up the service to multiple service replicas with built-in aut

111 max_completion_tokens: 1

112 ```

113

114 -??? Yaml

114 +??? code "Yaml"

115

116 ```yaml

117 service:

@@ -186,7 +186,7 @@ vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP([Spot]{'L4': 1}) R

186

187 After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:

188

189 -??? Commands

189 +??? console "Commands"

190

191 ```bash

192 ENDPOINT=$(sky serve status --endpoint 8081 vllm)

@@ -220,7 +220,7 @@ service:

220

221 This will scale the service up to when the QPS exceeds 2 for each replica.

222

223 -??? Yaml

223 +??? code "Yaml"

224

225 ```yaml

226 service:

@@ -285,7 +285,7 @@ sky serve down vllm

285

286 It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas.

287

288 -??? Yaml

288 +??? code "Yaml"

289

290 ```yaml

291 envs:

docs/deployment/integrations/production-stack.md

+3 -3

@@ -60,7 +60,7 @@ And then you can send out a query to the OpenAI-compatible API to check the avai

60 curl -o- http://localhost:30080/models

61 ```

62

63 -??? Output

63 +??? console "Output"

64

65 ```json

66 {

@@ -89,7 +89,7 @@ curl -X POST http://localhost:30080/completions \

89 }'

90 ```

91

92 -??? Output

92 +??? console "Output"

93

94 ```json

95 {

@@ -121,7 +121,7 @@ sudo helm uninstall vllm

121

122 The core vLLM production stack configuration is managed with YAML. Here is the example configuration used in the installation above:

123

124 -??? Yaml

124 +??? code "Yaml"

125

126 ```yaml

127 servingEngineSpec:

docs/deployment/k8s.md

+2 -2

@@ -29,7 +29,7 @@ Alternatively, you can deploy vLLM to Kubernetes using any of the following:

29

30 First, create a Kubernetes PVC and Secret for downloading and storing Hugging Face model:

31

32 -??? Config

32 +??? console "Config"

33

34 ```bash

35 cat <<EOF |kubectl apply -f -

@@ -57,7 +57,7 @@ First, create a Kubernetes PVC and Secret for downloading and storing Hugging Fa

57

58 Next, start the vLLM server as a Kubernetes Deployment and Service:

59

60 -??? Config

60 +??? console "Config"

61

62 ```bash

63 cat <<EOF |kubectl apply -f -

docs/deployment/nginx.md

+2 -2

@@ -36,7 +36,7 @@ docker build . -f Dockerfile.nginx --tag nginx-lb

36

37 Create a file named `nginx_conf/nginx.conf`. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another `server vllmN:8000 max_fails=3 fail_timeout=10000s;` entry to `upstream backend`.

38

39 -??? Config

39 +??? console "Config"

40

41 ```console

42 upstream backend {

@@ -95,7 +95,7 @@ Notes:

95 - The below example assumes GPU backend used. If you are using CPU backend, remove `--gpus device=ID`, add `VLLM_CPU_KVCACHE_SPACE` and `VLLM_CPU_OMP_THREADS_BIND` environment variables to the docker run command.

96 - Adjust the model name that you want to use in your vLLM servers if you don't want to use `Llama-2-7b-chat-hf`.

97

98 -??? Commands

98 +??? console "Commands"

99

100 ```console

101 mkdir -p ~/.cache/huggingface/hub/

docs/design/arch_overview.md

+2 -2

@@ -22,7 +22,7 @@ server.

22

23 Here is a sample of `LLM` class usage:

24

25 -??? Code

25 +??? code

26

27 ```python

28 from vllm import LLM, SamplingParams

@@ -180,7 +180,7 @@ vision-language model.

180

181 To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one:

182

183 - ??? Code

183 + ??? code

184

185 ```python

186 class MyOldModel(nn.Module):

docs/design/kernel/paged_attention.md

+1 -1

@@ -448,7 +448,7 @@ elements of the entire head for all context tokens. However, overall,

448 all results for output have been calculated but are just stored in

449 different thread register memory.

450

451 -??? Code

451 +??? code

452

453 ```cpp

454 float* out_smem = reinterpret_cast<float*>(shared_mem);

docs/design/plugin_system.md

+1 -1

@@ -13,7 +13,7 @@ Plugins are user-registered code that vLLM executes. Given vLLM's architecture (

13

14 vLLM's plugin system uses the standard Python `entry_points` mechanism. This mechanism allows developers to register functions in their Python packages for use by other packages. An example of a plugin:

15

16 -??? Code

16 +??? code

17

18 ```python

19 # inside `setup.py` file

docs/design/v1/p2p_nccl_connector.md

+10 -10

@@ -61,7 +61,7 @@ To address the above issues, I have designed and developed a local Tensor memory

61

62 # Install vLLM

63

64 -??? Commands

64 +??? console "Commands"

65

66 ```shell

67 # Enter the home directory or your working directory.

@@ -106,7 +106,7 @@ python3 disagg_prefill_proxy_xpyd.py &

106

107 ### Prefill1 (e.g. 10.0.1.2 or 10.0.1.1)

108

109 -??? Command

109 +??? console "Command"

110

111 ```shell

112 VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=0 vllm serve {your model directory} \

@@ -128,7 +128,7 @@ python3 disagg_prefill_proxy_xpyd.py &

128

129 ### Decode1 (e.g. 10.0.1.3 or 10.0.1.1)

130

131 -??? Command

131 +??? console "Command"

132

133 ```shell

134 VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=1 vllm serve {your model directory} \

@@ -150,7 +150,7 @@ python3 disagg_prefill_proxy_xpyd.py &

150

151 ### Decode2 (e.g. 10.0.1.4 or 10.0.1.1)

152

153 -??? Command

153 +??? console "Command"

154

155 ```shell

156 VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=2 vllm serve {your model directory} \

@@ -172,7 +172,7 @@ python3 disagg_prefill_proxy_xpyd.py &

172

173 ### Decode3 (e.g. 10.0.1.5 or 10.0.1.1)

174

175 -??? Command

175 +??? console "Command"

176

177 ```shell

178 VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=3 vllm serve {your model directory} \

@@ -203,7 +203,7 @@ python3 disagg_prefill_proxy_xpyd.py &

203

204 ### Prefill1 (e.g. 10.0.1.2 or 10.0.1.1)

205

206 -??? Command

206 +??? console "Command"

207

208 ```shell

209 VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=0 vllm serve {your model directory} \

@@ -225,7 +225,7 @@ python3 disagg_prefill_proxy_xpyd.py &

225

226 ### Prefill2 (e.g. 10.0.1.3 or 10.0.1.1)

227

228 -??? Command

228 +??? console "Command"

229

230 ```shell

231 VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=1 vllm serve {your model directory} \

@@ -247,7 +247,7 @@ python3 disagg_prefill_proxy_xpyd.py &

247

248 ### Prefill3 (e.g. 10.0.1.4 or 10.0.1.1)

249

250 -??? Command

250 +??? console "Command"

251

252 ```shell

253 VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=2 vllm serve {your model directory} \

@@ -269,7 +269,7 @@ python3 disagg_prefill_proxy_xpyd.py &

269

270 ### Decode1 (e.g. 10.0.1.5 or 10.0.1.1)

271

272 -??? Command

272 +??? console "Command"

273

274 ```shell

275 VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=3 vllm serve {your model directory} \

@@ -304,7 +304,7 @@ curl -X POST -s http://10.0.1.1:10001/v1/completions \

304

305 # Benchmark

306

307 -??? Command

307 +??? console "Command"

308

309 ```shell

310 python3 benchmark_serving.py \

docs/design/v1/torch_compile.md

+2 -2

@@ -28,7 +28,7 @@ A unique aspect of vLLM's `torch.compile` integration, is that we guarantee all

28

29 In the very verbose logs, we can see:

30

31 -??? Logs

31 +??? console "Logs"

32

33 ```text

34 DEBUG 03-07 03:06:52 [decorators.py:203] Start compiling function <code object forward at 0x7f08acf40c90, file "xxx/vllm/model_executor/models/llama.py", line 339>

@@ -110,7 +110,7 @@ Then it will also compile a specific kernel just for batch size `1, 2, 4, 8`. At

110

111 When all the shapes are known, `torch.compile` can compare different configs, and often find some better configs to run the kernel. For example, we can see the following log:

112

113 -??? Logs

113 +??? console "Logs"

114

115 ```

116 AUTOTUNE mm(8x2048, 2048x3072)

docs/features/lora.md

+4 -4

@@ -29,7 +29,7 @@ We can now submit the prompts and call `llm.generate` with the `lora_request` pa

29 of `LoRARequest` is a human identifiable name, the second parameter is a globally unique ID for the adapter and

30 the third parameter is the path to the LoRA adapter.

31

32 -??? Code

32 +??? code

33

34 ```python

35 sampling_params = SamplingParams(

@@ -70,7 +70,7 @@ The server entrypoint accepts all other LoRA configuration parameters (`max_lora

70 etc.), which will apply to all forthcoming requests. Upon querying the `/models` endpoint, we should see our LoRA along

71 with its base model (if `jq` is not installed, you can follow [this guide](https://jqlang.org/download/) to install it.):

72

73 -??? Command

73 +??? console "Command"

74

75 ```bash

76 curl localhost:8000/v1/models | jq .

@@ -172,7 +172,7 @@ Alternatively, follow these example steps to implement your own plugin:

172

173 1. Implement the LoRAResolver interface.

174

175 - ??? Example of a simple S3 LoRAResolver implementation

175 + ??? code "Example of a simple S3 LoRAResolver implementation"

176

177 ```python

178 import os

@@ -238,7 +238,7 @@ The new format of `--lora-modules` is mainly to support the display of parent mo

238 - The `parent` field of LoRA model `sql-lora` now links to its base model `meta-llama/Llama-2-7b-hf`. This correctly reflects the hierarchical relationship between the base model and the LoRA adapter.

239 - The `root` field points to the artifact location of the lora adapter.

240

241 -??? Command output

241 +??? console "Command output"

242

243 ```bash

244 $ curl http://localhost:8000/v1/models

docs/features/multimodal_inputs.md

+10 -10

@@ -20,7 +20,7 @@ To input multi-modal data, follow this schema in [vllm.inputs.PromptType][]:

20

21 You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:

22

23 -??? Code

23 +??? code

24

25 ```python

26 from vllm import LLM

@@ -68,7 +68,7 @@ Full example: <gh-file:examples/offline_inference/vision_language.py>

68

69 To substitute multiple images inside the same text prompt, you can pass in a list of images instead:

70

71 -??? Code

71 +??? code

72

73 ```python

74 from vllm import LLM

@@ -146,7 +146,7 @@ for o in outputs:

146

147 Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:

148

149 -??? Code

149 +??? code

150

151 ```python

152 from vllm import LLM

@@ -193,7 +193,7 @@ Full example: <gh-file:examples/offline_inference/audio_language.py>

193 To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,

194 pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.

195

196 -??? Code

196 +??? code

197

198 ```python

199 from vllm import LLM

@@ -220,7 +220,7 @@ pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the cor

220

221 For Qwen2-VL and MiniCPM-V, we accept additional parameters alongside the embeddings:

222

223 -??? Code

223 +??? code

224

225 ```python

226 # Construct the prompt based on your model

@@ -288,7 +288,7 @@ vllm serve microsoft/Phi-3.5-vision-instruct --task generate \

288

289 Then, you can use the OpenAI client as follows:

290

291 -??? Code

291 +??? code

292

293 ```python

294 from openai import OpenAI

@@ -366,7 +366,7 @@ vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --task generate --max-model

366

367 Then, you can use the OpenAI client as follows:

368

369 -??? Code

369 +??? code

370

371 ```python

372 from openai import OpenAI

@@ -430,7 +430,7 @@ vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b

430

431 Then, you can use the OpenAI client as follows:

432

433 -??? Code

433 +??? code

434

435 ```python

436 import base64

@@ -486,7 +486,7 @@ Then, you can use the OpenAI client as follows:

486

487 Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:

488

489 -??? Code

489 +??? code

490

491 ```python

492 chat_completion_from_url = client.chat.completions.create(

@@ -531,7 +531,7 @@ pass a tensor of shape to the corresponding field of the multi-modal dictionary.

531 For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.

532 The following example demonstrates how to pass image embeddings to the OpenAI server:

533

534 -??? Code

534 +??? code

535

536 ```python

537 image_embedding = torch.load(...)

docs/features/quantization/auto_awq.md

+2 -2

@@ -15,7 +15,7 @@ pip install autoawq

15

16 After installing AutoAWQ, you are ready to quantize a model. Please refer to the [AutoAWQ documentation](https://casper-hansen.github.io/AutoAWQ/examples/#basic-quantization) for further details. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:

17

18 -??? Code

18 +??? code

19

20 ```python

21 from awq import AutoAWQForCausalLM

@@ -51,7 +51,7 @@ python examples/offline_inference/llm_engine_example.py \

51

52 AWQ models are also supported directly through the LLM entrypoint:

53

54 -??? Code

54 +??? code

55

56 ```python

57 from vllm import LLM, SamplingParams

docs/features/quantization/bitblas.md

+1 -1

@@ -43,7 +43,7 @@ llm = LLM(

43

44 ## Read gptq format checkpoint

45

46 -??? Code

46 +??? code

47

48 ```python

49 from vllm import LLM

docs/features/quantization/fp8.md

+1 -1

@@ -58,7 +58,7 @@ For FP8 quantization, we can recover accuracy with simple RTN quantization. We r

58

59 Since simple RTN does not require data for weight quantization and the activations are quantized dynamically, we do not need any calibration data for this quantization flow.

60

61 -??? Code

61 +??? code

62

63 ```python

64 from llmcompressor.transformers import oneshot

docs/features/quantization/gguf.md

+1 -1

@@ -41,7 +41,7 @@ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

41

42 You can also use the GGUF model directly through the LLM entrypoint:

43

44 -??? Code

44 +??? code

45

46 ```python

47 from vllm import LLM, SamplingParams

docs/features/quantization/gptqmodel.md

+2 -2

@@ -31,7 +31,7 @@ After installing GPTQModel, you are ready to quantize a model. Please refer to t

31

32 Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:

33

34 -??? Code

34 +??? code

35

36 ```python

37 from datasets import load_dataset

@@ -69,7 +69,7 @@ python examples/offline_inference/llm_engine_example.py \

69

70 GPTQModel quantized models are also supported directly through the LLM entrypoint:

71

72 -??? Code

72 +??? code

73

74 ```python

75 from vllm import LLM, SamplingParams

docs/features/quantization/int4.md

+3 -3

@@ -53,7 +53,7 @@ When quantizing weights to INT4, you need sample data to estimate the weight upd

53 It's best to use calibration data that closely matches your deployment data.

54 For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:

55

56 -??? Code

56 +??? code

57

58 ```python

59 from datasets import load_dataset

@@ -78,7 +78,7 @@ For a general-purpose instruction-tuned model, you can use a dataset like `ultra

78

79 Now, apply the quantization algorithms:

80

81 -??? Code

81 +??? code

82

83 ```python

84 from llmcompressor.transformers import oneshot

@@ -141,7 +141,7 @@ lm_eval --model vllm \

141

142 The following is an example of an expanded quantization recipe you can tune to your own use case:

143

144 -??? Code

144 +??? code

145

146 ```python

147 from compressed_tensors.quantization import (

docs/features/quantization/int8.md

+2 -2

@@ -54,7 +54,7 @@ When quantizing activations to INT8, you need sample data to estimate the activa

54 It's best to use calibration data that closely matches your deployment data.

55 For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:

56

57 -??? Code

57 +??? code

58

59 ```python

60 from datasets import load_dataset

@@ -81,7 +81,7 @@ For a general-purpose instruction-tuned model, you can use a dataset like `ultra

81

82 Now, apply the quantization algorithms:

83

84 -??? Code

84 +??? code

85

86 ```python

87 from llmcompressor.transformers import oneshot

docs/features/quantization/modelopt.md

+2 -2

@@ -14,7 +14,7 @@ You can quantize HuggingFace models using the example scripts provided in the Te

14

15 Below is an example showing how to quantize a model using modelopt's PTQ API:

16

17 -??? Code

17 +??? code

18

19 ```python

20 import modelopt.torch.quantization as mtq

@@ -50,7 +50,7 @@ with torch.inference_mode():

50

51 The quantized checkpoint can then be deployed with vLLM. As an example, the following code shows how to deploy `nvidia/Llama-3.1-8B-Instruct-FP8`, which is the FP8 quantized checkpoint derived from `meta-llama/Llama-3.1-8B-Instruct`, using vLLM:

52

53 -??? Code

53 +??? code

54

55 ```python

56 from vllm import LLM, SamplingParams

docs/features/quantization/quantized_kvcache.md

+2 -2

@@ -35,7 +35,7 @@ Studies have shown that FP8 E4M3 quantization typically only minimally degrades

35

36 Here is an example of how to enable FP8 quantization:

37

38 -??? Code

38 +??? code

39

40 ```python

41 # To calculate kv cache scales on the fly enable the calculate_kv_scales

@@ -73,7 +73,7 @@ pip install llmcompressor

73

74 Here's a complete example using `meta-llama/Llama-3.1-8B-Instruct` (most models can use this same pattern):

75

76 -??? Code

76 +??? code

77

78 ```python

79 from datasets import load_dataset

docs/features/quantization/quark.md

+5 -5

@@ -42,7 +42,7 @@ The Quark quantization process can be listed for 5 steps as below:

42 Quark uses [Transformers](https://huggingface.co/docs/transformers/en/index)

43 to fetch model and tokenizer.

44

45 -??? Code

45 +??? code

46

47 ```python

48 from transformers import AutoTokenizer, AutoModelForCausalLM

@@ -65,7 +65,7 @@ Quark uses the [PyTorch Dataloader](https://pytorch.org/tutorials/beginner/basic

65 to load calibration data. For more details about how to use calibration datasets efficiently, please refer

66 to [Adding Calibration Datasets](https://quark.docs.amd.com/latest/pytorch/calibration_datasets.html).

67

68 -??? Code

68 +??? code

69

70 ```python

71 from datasets import load_dataset

@@ -98,7 +98,7 @@ kv-cache and the quantization algorithm is AutoSmoothQuant.

98 AutoSmoothQuant config file for Llama is

99 `examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json`.

100

101 -??? Code

101 +??? code

102

103 ```python

104 from quark.torch.quantization import (Config, QuantizationConfig,

@@ -145,7 +145,7 @@ HuggingFace `safetensors`, you can refer to

145 [HuggingFace format exporting](https://quark.docs.amd.com/latest/pytorch/export/quark_export_hf.html)

146 for more exporting format details.

147

148 -??? Code

148 +??? code

149

150 ```python

151 import torch

@@ -176,7 +176,7 @@ for more exporting format details.

176

177 Now, you can load and run the Quark quantized model directly through the LLM entrypoint:

178

179 -??? Code

179 +??? code

180

181 ```python

182 from vllm import LLM, SamplingParams

docs/features/quantization/torchao.md

+1 -1

@@ -15,7 +15,7 @@ pip install \

15 ## Quantizing HuggingFace Models

16 You can quantize your own huggingface model with torchao, e.g. [transformers](https://huggingface.co/docs/transformers/main/en/quantization/torchao) and [diffusers](https://huggingface.co/docs/diffusers/en/quantization/torchao), and save the checkpoint to huggingface hub like [this](https://huggingface.co/jerryzh168/llama3-8b-int8wo) with the following example code:

17

18 -??? Code

18 +??? code

19

20 ```Python

21 import torch

docs/features/reasoning_outputs.md

+6 -6

@@ -33,7 +33,7 @@ vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \

33

34 Next, make a request to the model that should return the reasoning content in the response.

35

36 -??? Code

36 +??? code

37

38 ```python

39 from openai import OpenAI

@@ -70,7 +70,7 @@ The `reasoning_content` field contains the reasoning steps that led to the final

70

71 Streaming chat completions are also supported for reasoning models. The `reasoning_content` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).

72

73 -??? Json

73 +??? console "Json"

74

75 ```json

76 {

@@ -95,7 +95,7 @@ Streaming chat completions are also supported for reasoning models. The `reasoni

95

96 OpenAI Python client library does not officially support `reasoning_content` attribute for streaming output. But the client supports extra attributes in the response. You can use `hasattr` to check if the `reasoning_content` attribute is present in the response. For example:

97

98 -??? Code

98 +??? code

99

100 ```python

101 from openai import OpenAI

@@ -152,7 +152,7 @@ Remember to check whether the `reasoning_content` exists in the response before

152

153 The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the `content` field, not from the `reasoning_content`.

154

155 -??? Code

155 +??? code

156

157 ```python

158 from openai import OpenAI

@@ -200,7 +200,7 @@ For more examples, please refer to <gh-file:examples/online_serving/openai_chat_

200

201 You can add a new `ReasoningParser` similar to <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.

202

203 -??? Code

203 +??? code

204

205 ```python

206 # import the required packages

@@ -258,7 +258,7 @@ You can add a new `ReasoningParser` similar to <gh-file:vllm/reasoning/deepseek_

258

259 Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.

260

261 -??? Code

261 +??? code

262

263 ```python

264 @dataclass

docs/features/spec_decode.md

+5 -5

@@ -18,7 +18,7 @@ Speculative decoding is a technique which improves inter-token latency in memory

18

19 The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.

20

21 -??? Code

21 +??? code

22

23 ```python

24 from vllm import LLM, SamplingParams

@@ -62,7 +62,7 @@ python -m vllm.entrypoints.openai.api_server \

62

63 Then use a client:

64

65 -??? Code

65 +??? code

66

67 ```python

68 from openai import OpenAI

@@ -103,7 +103,7 @@ Then use a client:

103 The following code configures vLLM to use speculative decoding where proposals are generated by

104 matching n-grams in the prompt. For more information read [this thread.](https://x.com/joao_gante/status/1747322413006643259)

105

106 -??? Code

106 +??? code

107

108 ```python

109 from vllm import LLM, SamplingParams

@@ -137,7 +137,7 @@ draft models that conditioning draft predictions on both context vectors and sam

137 For more information see [this blog](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/) or

138 [this technical report](https://arxiv.org/abs/2404.19124).

139

140 -??? Code

140 +??? code

141

142 ```python

143 from vllm import LLM, SamplingParams

@@ -185,7 +185,7 @@ A variety of speculative models of this type are available on HF hub:

185 The following code configures vLLM to use speculative decoding where proposals are generated by

186 an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](gh-file:examples/offline_inference/eagle.py).

187

188 -??? Code

188 +??? code

189

190 ```python

191 from vllm import LLM, SamplingParams

docs/features/structured_outputs.md

+8 -8

@@ -33,7 +33,7 @@ text.

33

34 Now let´s see an example for each of the cases, starting with the `guided_choice`, as it´s the easiest one:

35

36 -??? Code

36 +??? code

37

38 ```python

39 from openai import OpenAI

@@ -55,7 +55,7 @@ Now let´s see an example for each of the cases, starting with the `guided_choic

55

56 The next example shows how to use the `guided_regex`. The idea is to generate an email address, given a simple regex template:

57

58 -??? Code

58 +??? code

59

60 ```python

61 completion = client.chat.completions.create(

@@ -79,7 +79,7 @@ For this we can use the `guided_json` parameter in two different ways:

79

80 The next example shows how to use the `guided_json` parameter with a Pydantic model:

81

82 -??? Code

82 +??? code

83

84 ```python

85 from pydantic import BaseModel

@@ -127,7 +127,7 @@ difficult to use, but it´s really powerful. It allows us to define complete

127 languages like SQL queries. It works by using a context free EBNF grammar.

128 As an example, we can use to define a specific format of simplified SQL queries:

129

130 -??? Code

130 +??? code

131

132 ```python

133 simplified_sql_grammar = """

@@ -169,7 +169,7 @@ vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r

169

170 Note that you can use reasoning with any provided structured outputs feature. The following uses one with JSON schema:

171

172 -??? Code

172 +??? code

173

174 ```python

175 from pydantic import BaseModel

@@ -212,7 +212,7 @@ For the following examples, vLLM was setup using `vllm serve meta-llama/Llama-3.

212

213 Here is a simple example demonstrating how to get structured output using Pydantic models:

214

215 -??? Code

215 +??? code

216

217 ```python

218 from pydantic import BaseModel

@@ -248,7 +248,7 @@ Age: 28

248

249 Here is a more complex example using nested Pydantic models to handle a step-by-step math solution:

250

251 -??? Code

251 +??? code

252

253 ```python

254 from typing import List

@@ -308,7 +308,7 @@ These parameters can be used in the same way as the parameters from the Online

308 Serving examples above. One example for the usage of the `choice` parameter is

309 shown below:

310

311 -??? Code

311 +??? code

312

313 ```python

314 from vllm import LLM, SamplingParams

docs/features/tool_calling.md

+2 -2

@@ -15,7 +15,7 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \

15

16 Next, make a request to the model that should result in it using the available tools:

17

18 -??? Code

18 +??? code

19

20 ```python

21 from openai import OpenAI

@@ -320,7 +320,7 @@ A tool parser plugin is a Python file containing one or more ToolParser implemen

320

321 Here is a summary of a plugin file:

322

323 -??? Code

323 +??? code

324

325 ```python

326

docs/getting_started/installation/cpu.md

+2 -2

@@ -76,7 +76,7 @@ Currently, there are no pre-built CPU wheels.

76

77 ### Build image from source

78

79 -??? Commands

79 +??? console "Commands"

80

81 ```bash

82 docker build -f docker/Dockerfile.cpu \

@@ -149,7 +149,7 @@ vllm serve facebook/opt-125m

149

150 - If using vLLM CPU backend on a machine with hyper-threading, it is recommended to bind only one OpenMP thread on each physical CPU core using `VLLM_CPU_OMP_THREADS_BIND` or using auto thread binding feature by default. On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:

151

152 -??? Commands

152 +??? console "Commands"

153

154 ```console

155 $ lscpu -e # check the mapping between logical CPU cores and physical CPU cores

docs/getting_started/installation/gpu/rocm.inc.md

+2 -2

@@ -95,7 +95,7 @@ Currently, there are no pre-built ROCm wheels.

95

96 4. Build vLLM. For example, vLLM on ROCM 6.3 can be built with the following steps:

97

98 - ??? Commands

98 + ??? console "Commands"

99

100 ```bash

101 pip install --upgrade pip

@@ -206,7 +206,7 @@ DOCKER_BUILDKIT=1 docker build \

206

207 To run the above docker image `vllm-rocm`, use the below command:

208

209 -??? Command

209 +??? console "Command"

210

211 ```bash

212 docker run -it \

docs/getting_started/installation/intel_gaudi.md

+2 -2

@@ -237,7 +237,7 @@ As an example, if a request of 3 sequences, with max sequence length of 412 come

237

238 Warmup is an optional, but highly recommended step occurring before vLLM server starts listening. It executes a forward pass for each bucket with dummy data. The goal is to pre-compile all graphs and not incur any graph compilation overheads within bucket boundaries during server runtime. Each warmup step is logged during vLLM startup:

239

240 -??? Logs

240 +??? console "Logs"

241

242 ```text

243 INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:79.16 GiB

@@ -286,7 +286,7 @@ When there's large amount of requests pending, vLLM scheduler will attempt to fi

286

287 Each described step is logged by vLLM server, as follows (negative values correspond to memory being released):

288

289 -??? Logs

289 +??? console "Logs"

290

291 ```text

292 INFO 08-02 17:37:44 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]

docs/getting_started/quickstart.md

+2 -2

@@ -147,7 +147,7 @@ curl http://localhost:8000/v1/completions \

147

148 Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the `openai` Python package:

149

150 -??? Code

150 +??? code

151

152 ```python

153 from openai import OpenAI

@@ -186,7 +186,7 @@ curl http://localhost:8000/v1/chat/completions \

186

187 Alternatively, you can use the `openai` Python package:

188

189 -??? Code

189 +??? code

190

191 ```python

192 from openai import OpenAI

docs/mkdocs/stylesheets/extra.css

+30

@@ -39,6 +39,8 @@ body[data-md-color-scheme="slate"] .md-nav__item--section > label.md-nav__link .

39 :root {

40 --md-admonition-icon--announcement: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" width="16" height="16"><path d="M3.25 9a.75.75 0 0 1 .75.75c0 2.142.456 3.828.733 4.653a.122.122 0 0 0 .05.064.212.212 0 0 0 .117.033h1.31c.085 0 .18-.042.258-.152a.45.45 0 0 0 .075-.366A16.743 16.743 0 0 1 6 9.75a.75.75 0 0 1 1.5 0c0 1.588.25 2.926.494 3.85.293 1.113-.504 2.4-1.783 2.4H4.9c-.686 0-1.35-.41-1.589-1.12A16.4 16.4 0 0 1 2.5 9.75.75.75 0 0 1 3.25 9Z"></path><path d="M0 6a4 4 0 0 1 4-4h2.75a.75.75 0 0 1 .75.75v6.5a.75.75 0 0 1-.75.75H4a4 4 0 0 1-4-4Zm4-2.5a2.5 2.5 0 1 0 0 5h2v-5Z"></path><path d="M15.59.082A.75.75 0 0 1 16 .75v10.5a.75.75 0 0 1-1.189.608l-.002-.001h.001l-.014-.01a5.775 5.775 0 0 0-.422-.25 10.63 10.63 0 0 0-1.469-.64C11.576 10.484 9.536 10 6.75 10a.75.75 0 0 1 0-1.5c2.964 0 5.174.516 6.658 1.043.423.151.787.302 1.092.443V2.014c-.305.14-.669.292-1.092.443C11.924 2.984 9.713 3.5 6.75 3.5a.75.75 0 0 1 0-1.5c2.786 0 4.826-.484 6.155-.957.665-.236 1.154-.47 1.47-.64.144-.077.284-.161.421-.25l.014-.01a.75.75 0 0 1 .78-.061Z"></path></svg>');

41 --md-admonition-icon--important: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" width="16" height="16"><path d="M4.47.22A.749.749 0 0 1 5 0h6c.199 0 .389.079.53.22l4.25 4.25c.141.14.22.331.22.53v6a.749.749 0 0 1-.22.53l-4.25 4.25A.749.749 0 0 1 11 16H5a.749.749 0 0 1-.53-.22L.22 11.53A.749.749 0 0 1 0 11V5c0-.199.079-.389.22-.53Zm.84 1.28L1.5 5.31v5.38l3.81 3.81h5.38l3.81-3.81V5.31L10.69 1.5ZM8 4a.75.75 0 0 1 .75.75v3.5a.75.75 0 0 1-1.5 0v-3.5A.75.75 0 0 1 8 4Zm0 8a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z"></path></svg>');

42 + --md-admonition-icon--code: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16"><path d="m11.28 3.22 4.25 4.25a.75.75 0 0 1 0 1.06l-4.25 4.25a.749.749 0 0 1-1.275-.326.75.75 0 0 1 .215-.734L13.94 8l-3.72-3.72a.749.749 0 0 1 .326-1.275.75.75 0 0 1 .734.215m-6.56 0a.75.75 0 0 1 1.042.018.75.75 0 0 1 .018 1.042L2.06 8l3.72 3.72a.749.749 0 0 1-.326 1.275.75.75 0 0 1-.734-.215L.47 8.53a.75.75 0 0 1 0-1.06Z"/></svg>');

43 + --md-admonition-icon--console: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16"><path d="M0 2.75C0 1.784.784 1 1.75 1h12.5c.966 0 1.75.784 1.75 1.75v10.5A1.75 1.75 0 0 1 14.25 15H1.75A1.75 1.75 0 0 1 0 13.25Zm1.75-.25a.25.25 0 0 0-.25.25v10.5c0 .138.112.25.25.25h12.5a.25.25 0 0 0 .25-.25V2.75a.25.25 0 0 0-.25-.25ZM7.25 8a.75.75 0 0 1-.22.53l-2.25 2.25a.749.749 0 0 1-1.275-.326.75.75 0 0 1 .215-.734L5.44 8 3.72 6.28a.749.749 0 0 1 .326-1.275.75.75 0 0 1 .734.215l2.25 2.25c.141.14.22.331.22.53m1.5 1.5h3a.75.75 0 0 1 0 1.5h-3a.75.75 0 0 1 0-1.5"/></svg>');

44 }

45

46 .md-typeset .admonition.announcement,

@@ -49,6 +51,14 @@ body[data-md-color-scheme="slate"] .md-nav__item--section > label.md-nav__link .

51 .md-typeset details.important {

52 border-color: rgb(239, 85, 82);

53 }

54 +.md-typeset .admonition.code,

55 +.md-typeset details.code {

56 + border-color: #64dd17

57 +}

58 +.md-typeset .admonition.console,

59 +.md-typeset details.console {

60 + border-color: #64dd17

61 +}

62

63 .md-typeset .announcement > .admonition-title,

64 .md-typeset .announcement > summary {

@@ -58,6 +68,14 @@ body[data-md-color-scheme="slate"] .md-nav__item--section > label.md-nav__link .

68 .md-typeset .important > summary {

69 background-color: rgb(239, 85, 82, 0.1);

70 }

71 +.md-typeset .code > .admonition-title,

72 +.md-typeset .code > summary {

73 + background-color: #64dd171a;

74 +}

75 +.md-typeset .console > .admonition-title,

76 +.md-typeset .console > summary {

77 + background-color: #64dd171a;

78 +}

79

80 .md-typeset .announcement > .admonition-title::before,

81 .md-typeset .announcement > summary::before {

@@ -71,6 +89,18 @@ body[data-md-color-scheme="slate"] .md-nav__item--section > label.md-nav__link .

89 -webkit-mask-image: var(--md-admonition-icon--important);

90 mask-image: var(--md-admonition-icon--important);

91 }

92 +.md-typeset .code > .admonition-title::before,

93 +.md-typeset .code > summary::before {

94 + background-color: #64dd17;

95 + -webkit-mask-image: var(--md-admonition-icon--code);

96 + mask-image: var(--md-admonition-icon--code);

97 +}

98 +.md-typeset .console > .admonition-title::before,

99 +.md-typeset .console > summary::before {

100 + background-color: #64dd17;

101 + -webkit-mask-image: var(--md-admonition-icon--console);

102 + mask-image: var(--md-admonition-icon--console);

103 +}

104

105 /* Make label fully visible on hover */

106 .md-content__button[href*="edit"]:hover::after {

docs/models/generative_models.md

+1 -1

@@ -85,7 +85,7 @@ and automatically applies the model's [chat template](https://huggingface.co/doc

85 In general, only instruction-tuned models have a chat template.

86 Base models may perform poorly as they are not trained to respond to the chat conversation.

87

88 -??? Code

88 +??? code

89

90 ```python

91 from vllm import LLM

docs/models/supported_models.md

+1 -1

@@ -642,7 +642,7 @@ Specified using `--task generate`.

642

643 For the best results, we recommend using the following dependency versions (tested on A10 and L40):

644

645 - ??? Dependency versions

645 + ??? code "Dependency versions"

646

647 ```text

648 # Core vLLM-compatible dependencies with Molmo accuracy setup (tested on L40)

docs/serving/integrations/langchain.md

+1 -1

@@ -13,7 +13,7 @@ pip install langchain langchain_community -q

13

14 To run inference on a single or multiple GPUs, use `VLLM` class from `langchain`.

15

16 -??? Code

16 +??? code

17

18 ```python

19 from langchain_community.llms import VLLM

docs/serving/openai_compatible_server.md

+20 -20

@@ -15,7 +15,7 @@ vllm serve NousResearch/Meta-Llama-3-8B-Instruct \

15

16 To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).

17

18 -??? Code

18 +??? code

19

20 ```python

21 from openai import OpenAI

@@ -146,7 +146,7 @@ completion = client.chat.completions.create(

146 Only `X-Request-Id` HTTP request header is supported for now. It can be enabled

147 with `--enable-request-id-headers`.

148

149 -??? Code

149 +??? code

150

151 ```python

152 completion = client.chat.completions.create(

@@ -185,7 +185,7 @@ Code example: <gh-file:examples/online_serving/openai_completion_client.py>

185

186 The following [sampling parameters][sampling-params] are supported.

187

188 -??? Code

188 +??? code

189

190 ```python

191 --8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"

@@ -193,7 +193,7 @@ The following [sampling parameters][sampling-params] are supported.

193

194 The following extra parameters are supported:

195

196 -??? Code

196 +??? code

197

198 ```python

199 --8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"

@@ -217,7 +217,7 @@ Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>

217

218 The following [sampling parameters][sampling-params] are supported.

219

220 -??? Code

220 +??? code

221

222 ```python

223 --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"

@@ -225,7 +225,7 @@ The following [sampling parameters][sampling-params] are supported.

225

226 The following extra parameters are supported:

227

228 -??? Code

228 +??? code

229

230 ```python

231 --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"

@@ -268,7 +268,7 @@ and passing a list of `messages` in the request. Refer to the examples below for

268

269 Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

270

271 - ??? Code

271 + ??? code

272

273 ```python

274 import requests

@@ -327,7 +327,7 @@ The following [pooling parameters][pooling-params] are supported.

327

328 The following extra parameters are supported by default:

329

330 -??? Code

330 +??? code

331

332 ```python

333 --8<-- "vllm/entrypoints/openai/protocol.py:embedding-extra-params"

@@ -335,7 +335,7 @@ The following extra parameters are supported by default:

335

336 For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:

337

338 -??? Code

338 +??? code

339

340 ```python

341 --8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"

@@ -358,7 +358,7 @@ Code example: <gh-file:examples/online_serving/openai_transcription_client.py>

358

359 The following [sampling parameters][sampling-params] are supported.

360

361 -??? Code

361 +??? code

362

363 ```python

364 --8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"

@@ -366,7 +366,7 @@ The following [sampling parameters][sampling-params] are supported.

366

367 The following extra parameters are supported:

368

369 -??? Code

369 +??? code

370

371 ```python

372 --8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"

@@ -446,7 +446,7 @@ curl -v "http://127.0.0.1:8000/classify" \

446 }'

447 ```

448

449 -??? Response

449 +??? console "Response"

450

451 ```bash

452 {

@@ -494,7 +494,7 @@ curl -v "http://127.0.0.1:8000/classify" \

494 }'

495 ```

496

497 -??? Response

497 +??? console "Response"

498

499 ```bash

500 {

@@ -564,7 +564,7 @@ curl -X 'POST' \

564 }'

565 ```

566

567 -??? Response

567 +??? console "Response"

568

569 ```bash

570 {

@@ -589,7 +589,7 @@ You can pass a string to `text_1` and a list to `text_2`, forming multiple sente

589 where each pair is built from `text_1` and a string in `text_2`.

590 The total number of pairs is `len(text_2)`.

591

592 -??? Request

592 +??? console "Request"

593

594 ```bash

595 curl -X 'POST' \

@@ -606,7 +606,7 @@ The total number of pairs is `len(text_2)`.

606 }'

607 ```

608

609 -??? Response

609 +??? console "Response"

610

611 ```bash

612 {

@@ -634,7 +634,7 @@ You can pass a list to both `text_1` and `text_2`, forming multiple sentence pai

634 where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).

635 The total number of pairs is `len(text_2)`.

636

637 -??? Request

637 +??? console "Request"

638

639 ```bash

640 curl -X 'POST' \

@@ -655,7 +655,7 @@ The total number of pairs is `len(text_2)`.

655 }'

656 ```

657

658 -??? Response

658 +??? console "Response"

659

660 ```bash

661 {

@@ -716,7 +716,7 @@ Code example: <gh-file:examples/online_serving/jinaai_rerank_client.py>

716 Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.

717 Result documents will be sorted by relevance, and the `index` property can be used to determine original order.

718

719 -??? Request

719 +??? console "Request"

720

721 ```bash

722 curl -X 'POST' \

@@ -734,7 +734,7 @@ Result documents will be sorted by relevance, and the `index` property can be us

734 }'

735 ```

736

737 -??? Response

737 +??? console "Response"

738

739 ```bash

740 {

docs/usage/metrics.md

+2 -2

@@ -12,7 +12,7 @@ vllm serve unsloth/Llama-3.2-1B-Instruct

12

13 Then query the endpoint to get the latest metrics from the server:

14

15 -??? Output

15 +??? console "Output"

16

17 ```console

18 $ curl http://0.0.0.0:8000/metrics

@@ -33,7 +33,7 @@ Then query the endpoint to get the latest metrics from the server:

33

34 The following metrics are exposed:

35

36 -??? Code

36 +??? code

37

38 ```python

39 --8<-- "vllm/engine/metrics.py:metrics-definitions"

docs/usage/troubleshooting.md

+3 -3

@@ -60,7 +60,7 @@ To identify the particular CUDA operation that causes the error, you can add `--

60

61 If GPU/CPU communication cannot be established, you can use the following Python script and follow the instructions below to confirm whether the GPU/CPU communication is working correctly.

62

63 -??? Code

63 +??? code

64

65 ```python

66 # Test PyTorch NCCL

@@ -170,7 +170,7 @@ WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously

170

171 or an error from Python that looks like this:

172

173 -??? Logs

173 +??? console "Logs"

174

175 ```console

176 RuntimeError:

@@ -214,7 +214,7 @@ if __name__ == '__main__':

214

215 vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](gh-pr:10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:

216

217 -??? Code

217 +??? code

218

219 ```python

220 import torch

docs/usage/usage_stats.md

+1 -1

@@ -10,7 +10,7 @@ The list of data collected by the latest version of vLLM can be found here: <gh-

10

11 Here is an example as of v0.4.0:

12

13 -??? Output

13 +??? console "Output"

14

15 ```json

16 {

Make distinct `code` and `console` admonitions so readers are less likely to miss them (#20585)