c3649e4fee · tduncan/Nvidia-Optimization-Stack-VLLM

.buildkite/nightly-benchmarks/nightly-annotation.md

+1 -1

@@ -16,7 +16,7 @@ Please download the visualization scripts in the post

16 - Download `nightly-benchmarks.zip`.

17 - In the same folder, run the following code:

18

19 - ```console

19 + ```bash

20 export HF_TOKEN=<your HF token>

21 apt update

22 apt install -y git

docs/deployment/docker.md

+6 -6

@@ -10,7 +10,7 @@ title: Using Docker

10 vLLM offers an official Docker image for deployment.

11 The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai](https://hub.docker.com/r/vllm/vllm-openai/tags).

12

13 -```console

13 +```bash

14 docker run --runtime nvidia --gpus all \

15 -v ~/.cache/huggingface:/root/.cache/huggingface \

16 --env "HUGGING_FACE_HUB_TOKEN=<secret>" \

@@ -22,7 +22,7 @@ docker run --runtime nvidia --gpus all \

22

23 This image can also be used with other container engines such as [Podman](https://podman.io/).

24

25 -```console

25 +```bash

26 podman run --gpus all \

27 -v ~/.cache/huggingface:/root/.cache/huggingface \

28 --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \

@@ -71,7 +71,7 @@ You can add any other [engine-args][engine-args] you need after the image tag (`

71

72 You can build and run vLLM from source via the provided <gh-file:docker/Dockerfile>. To build vLLM:

73

74 -```console

74 +```bash

75 # optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2

76 DOCKER_BUILDKIT=1 docker build . \

77 --target vllm-openai \

@@ -99,7 +99,7 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--

99

100 ??? Command

101

102 - ```console

102 + ```bash

103 # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)

104 python3 use_existing_torch.py

105 DOCKER_BUILDKIT=1 docker build . \

@@ -118,7 +118,7 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--

118

119 Run the following command on your host machine to register QEMU user static handlers:

120

121 - ```console

121 + ```bash

122 docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

123 ```

124

@@ -128,7 +128,7 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--

128

129 To run vLLM with the custom-built Docker image:

130

131 -```console

131 +```bash

132 docker run --runtime nvidia --gpus all \

133 -v ~/.cache/huggingface:/root/.cache/huggingface \

134 -p 8000:8000 \

docs/deployment/frameworks/anything-llm.md

+1 -1

@@ -15,7 +15,7 @@ It allows you to deploy a large language model (LLM) server with vLLM as the bac

15

16 - Start the vLLM server with the supported chat completion model, e.g.

17

18 -```console

18 +```bash

19 vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096

20 ```

21

docs/deployment/frameworks/autogen.md

+2 -2

@@ -11,7 +11,7 @@ title: AutoGen

11

12 - Setup [AutoGen](https://microsoft.github.io/autogen/0.2/docs/installation/) environment

13

14 -```console

14 +```bash

15 pip install vllm

16

17 # Install AgentChat and OpenAI client from Extensions

@@ -23,7 +23,7 @@ pip install -U "autogen-agentchat" "autogen-ext[openai]"

23

24 - Start the vLLM server with the supported chat completion model, e.g.

25

26 -```console

26 +```bash

27 python -m vllm.entrypoints.openai.api_server \

28 --model mistralai/Mistral-7B-Instruct-v0.2

29 ```

docs/deployment/frameworks/cerebrium.md

+3 -3

@@ -11,14 +11,14 @@ vLLM can be run on a cloud based GPU machine with [Cerebrium](https://www.cerebr

11

12 To install the Cerebrium client, run:

13

14 -```console

14 +```bash

15 pip install cerebrium

16 cerebrium login

17 ```

18

19 Next, create your Cerebrium project, run:

20

21 -```console

21 +```bash

22 cerebrium init vllm-project

23 ```

24

@@ -58,7 +58,7 @@ Next, let us add our code to handle inference for the LLM of your choice (`mistr

58

59 Then, run the following code to deploy it to the cloud:

60

61 -```console

61 +```bash

62 cerebrium deploy

63 ```

64

docs/deployment/frameworks/chatbox.md

+1 -1

@@ -15,7 +15,7 @@ It allows you to deploy a large language model (LLM) server with vLLM as the bac

15

16 - Start the vLLM server with the supported chat completion model, e.g.

17

18 -```console

18 +```bash

19 vllm serve qwen/Qwen1.5-0.5B-Chat

20 ```

21

docs/deployment/frameworks/dify.md

+2 -2

@@ -18,13 +18,13 @@ This guide walks you through deploying Dify using a vLLM backend.

18

19 - Start the vLLM server with the supported chat completion model, e.g.

20

21 -```console

21 +```bash

22 vllm serve Qwen/Qwen1.5-7B-Chat

23 ```

24

25 - Start the Dify server with docker compose ([details](https://github.com/langgenius/dify?tab=readme-ov-file#quick-start)):

26

27 -```console

27 +```bash

28 git clone https://github.com/langgenius/dify.git

29 cd dify

30 cd docker

docs/deployment/frameworks/dstack.md

+2 -2

@@ -11,14 +11,14 @@ vLLM can be run on a cloud based GPU machine with [dstack](https://dstack.ai/),

11

12 To install dstack client, run:

13

14 -```console

14 +```bash

15 pip install "dstack[all]

16 dstack server

17 ```

18

19 Next, to configure your dstack project, run:

20

21 -```console

21 +```bash

22 mkdir -p vllm-dstack

23 cd vllm-dstack

24 dstack init

docs/deployment/frameworks/haystack.md

+2 -2

@@ -13,7 +13,7 @@ It allows you to deploy a large language model (LLM) server with vLLM as the bac

13

14 - Setup vLLM and Haystack environment

15

16 -```console

16 +```bash

17 pip install vllm haystack-ai

18 ```

19

@@ -21,7 +21,7 @@ pip install vllm haystack-ai

21

22 - Start the vLLM server with the supported chat completion model, e.g.

23

24 -```console

24 +```bash

25 vllm serve mistralai/Mistral-7B-Instruct-v0.1

26 ```

27

docs/deployment/frameworks/helm.md

+2 -2

@@ -22,7 +22,7 @@ Before you begin, ensure that you have the following:

22

23 To install the chart with the release name `test-vllm`:

24

25 -```console

25 +```bash

26 helm upgrade --install --create-namespace --namespace=ns-vllm test-vllm . -f values.yaml --set secrets.s3endpoint=$ACCESS_POINT --set secrets.s3bucketname=$BUCKET --set secrets.s3accesskeyid=$ACCESS_KEY --set secrets.s3accesskey=$SECRET_KEY

27 ```

28

@@ -30,7 +30,7 @@ helm upgrade --install --create-namespace --namespace=ns-vllm test-vllm . -f val

30

31 To uninstall the `test-vllm` deployment:

32

33 -```console

33 +```bash

34 helm uninstall test-vllm --namespace=ns-vllm

35 ```

36

docs/deployment/frameworks/litellm.md

+3 -3

@@ -18,7 +18,7 @@ And LiteLLM supports all models on VLLM.

18

19 - Setup vLLM and litellm environment

20

21 -```console

21 +```bash

22 pip install vllm litellm

23 ```

24

@@ -28,7 +28,7 @@ pip install vllm litellm

28

29 - Start the vLLM server with the supported chat completion model, e.g.

30

31 -```console

31 +```bash

32 vllm serve qwen/Qwen1.5-0.5B-Chat

33 ```

34

@@ -56,7 +56,7 @@ vllm serve qwen/Qwen1.5-0.5B-Chat

56

57 - Start the vLLM server with the supported embedding model, e.g.

58

59 -```console

59 +```bash

60 vllm serve BAAI/bge-base-en-v1.5

61 ```

62

docs/deployment/frameworks/open-webui.md

+2 -2

@@ -7,13 +7,13 @@ title: Open WebUI

7

8 2. Start the vLLM server with the supported chat completion model, e.g.

9

10 -```console

10 +```bash

11 vllm serve qwen/Qwen1.5-0.5B-Chat

12 ```

13

14 1. Start the [Open WebUI](https://github.com/open-webui/open-webui) docker container (replace the vllm serve host and vllm serve port):

15

16 -```console

16 +```bash

17 docker run -d -p 3000:8080 \

18 --name open-webui \

19 -v open-webui:/app/backend/data \

docs/deployment/frameworks/retrieval_augmented_generation.md

+6 -6

@@ -15,7 +15,7 @@ Here are the integrations:

15

16 - Setup vLLM and langchain environment

17

18 -```console

18 +```bash

19 pip install -U vllm \

20 langchain_milvus langchain_openai \

21 langchain_community beautifulsoup4 \

@@ -26,14 +26,14 @@ pip install -U vllm \

26

27 - Start the vLLM server with the supported embedding model, e.g.

28

29 -```console

29 +```bash

30 # Start embedding service (port 8000)

31 vllm serve ssmits/Qwen2-7B-Instruct-embed-base

32 ```

33

34 - Start the vLLM server with the supported chat completion model, e.g.

35

36 -```console

36 +```bash

37 # Start chat service (port 8001)

38 vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001

39 ```

@@ -52,7 +52,7 @@ python retrieval_augmented_generation_with_langchain.py

52

53 - Setup vLLM and llamaindex environment

54

55 -```console

55 +```bash

56 pip install vllm \

57 llama-index llama-index-readers-web \

58 llama-index-llms-openai-like \

@@ -64,14 +64,14 @@ pip install vllm \

64

65 - Start the vLLM server with the supported embedding model, e.g.

66

67 -```console

67 +```bash

68 # Start embedding service (port 8000)

69 vllm serve ssmits/Qwen2-7B-Instruct-embed-base

70 ```

71

72 - Start the vLLM server with the supported chat completion model, e.g.

73

74 -```console

74 +```bash

75 # Start chat service (port 8001)

76 vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001

77 ```

docs/deployment/frameworks/skypilot.md

+8 -8

@@ -15,7 +15,7 @@ vLLM can be **run and scaled to multiple service replicas on clouds and Kubernet

15 - Check that you have installed SkyPilot ([docs](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html)).

16 - Check that `sky check` shows clouds or Kubernetes are enabled.

17

18 -```console

18 +```bash

19 pip install skypilot-nightly

20 sky check

21 ```

@@ -71,7 +71,7 @@ See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypil

71

72 Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, ...):

73

74 -```console

74 +```bash

75 HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN

76 ```

77

@@ -83,7 +83,7 @@ Check the output of the command. There will be a shareable gradio link (like the

83

84 **Optional**: Serve the 70B model instead of the default 8B and use more GPU:

85

86 -```console

86 +```bash

87 HF_TOKEN="your-huggingface-token" \

88 sky launch serving.yaml \

89 --gpus A100:8 \

@@ -159,7 +159,7 @@ SkyPilot can scale up the service to multiple service replicas with built-in aut

159

160 Start the serving the Llama-3 8B model on multiple replicas:

161

162 -```console

162 +```bash

163 HF_TOKEN="your-huggingface-token" \

164 sky serve up -n vllm serving.yaml \

165 --env HF_TOKEN

@@ -167,7 +167,7 @@ HF_TOKEN="your-huggingface-token" \

167

168 Wait until the service is ready:

169

170 -```console

170 +```bash

171 watch -n10 sky serve status vllm

172 ```

173

@@ -271,13 +271,13 @@ This will scale the service up to when the QPS exceeds 2 for each replica.

271

272 To update the service with the new config:

273

274 -```console

274 +```bash

275 HF_TOKEN="your-huggingface-token" sky serve update vllm serving.yaml --env HF_TOKEN

276 ```

277

278 To stop the service:

279

280 -```console

280 +```bash

281 sky serve down vllm

282 ```

283

@@ -317,7 +317,7 @@ It is also possible to access the Llama-3 service with a separate GUI frontend,

317

318 1. Start the chat web UI:

319

320 - ```console

320 + ```bash

321 sky launch \

322 -c gui ./gui.yaml \

323 --env ENDPOINT=$(sky serve status --endpoint vllm)

docs/deployment/frameworks/streamlit.md

+3 -3

@@ -15,13 +15,13 @@ It can be quickly integrated with vLLM as a backend API server, enabling powerfu

15

16 - Start the vLLM server with the supported chat completion model, e.g.

17

18 -```console

18 +```bash

19 vllm serve qwen/Qwen1.5-0.5B-Chat

20 ```

21

22 - Install streamlit and openai:

23

24 -```console

24 +```bash

25 pip install streamlit openai

26 ```

27

@@ -29,7 +29,7 @@ pip install streamlit openai

29

30 - Start the streamlit web UI and start to chat:

31

32 -```console

32 +```bash

33 streamlit run streamlit_openai_chatbot_webserver.py

34

35 # or specify the VLLM_API_BASE or VLLM_API_KEY

docs/deployment/integrations/llamastack.md

+1 -1

@@ -7,7 +7,7 @@ vLLM is also available via [Llama Stack](https://github.com/meta-llama/llama-sta

7

8 To install Llama Stack, run

9

10 -```console

10 +```bash

11 pip install llama-stack -q

12 ```

13

docs/deployment/k8s.md

+3 -3

@@ -115,7 +115,7 @@ Next, start the vLLM server as a Kubernetes Deployment and Service:

115

116 We can verify that the vLLM server has started successfully via the logs (this might take a couple of minutes to download the model):

117

118 -```console

118 +```bash

119 kubectl logs -l app.kubernetes.io/name=vllm

120 ...

121 INFO: Started server process [1]

@@ -358,14 +358,14 @@ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

358

359 Apply the deployment and service configurations using `kubectl apply -f <filename>`:

360

361 - ```console

361 + ```bash

362 kubectl apply -f deployment.yaml

363 kubectl apply -f service.yaml

364 ```

365

366 To test the deployment, run the following `curl` command:

367

368 - ```console

368 + ```bash

369 curl http://mistral-7b.default.svc.cluster.local/v1/completions \

370 -H "Content-Type: application/json" \

371 -d '{

docs/deployment/nginx.md

+8 -8

@@ -11,13 +11,13 @@ This document shows how to launch multiple vLLM serving containers and use Nginx

11

12 This guide assumes that you have just cloned the vLLM project and you're currently in the vllm root directory.

13

14 -```console

14 +```bash

15 export vllm_root=`pwd`

16 ```

17

18 Create a file named `Dockerfile.nginx`:

19

20 -```console

20 +```dockerfile

21 FROM nginx:latest

22 RUN rm /etc/nginx/conf.d/default.conf

23 EXPOSE 80

@@ -26,7 +26,7 @@ CMD ["nginx", "-g", "daemon off;"]

26

27 Build the container:

28

29 -```console

29 +```bash

30 docker build . -f Dockerfile.nginx --tag nginx-lb

31 ```

32

@@ -60,14 +60,14 @@ Create a file named `nginx_conf/nginx.conf`. Note that you can add as many serve

60

61 ## Build vLLM Container

62

63 -```console

63 +```bash

64 cd $vllm_root

65 docker build -f docker/Dockerfile . --tag vllm

66 ```

67

68 If you are behind proxy, you can pass the proxy settings to the docker build command as shown below:

69

70 -```console

70 +```bash

71 cd $vllm_root

72 docker build \

73 -f docker/Dockerfile . \

@@ -80,7 +80,7 @@ docker build \

80

81 ## Create Docker Network

82

83 -```console

83 +```bash

84 docker network create vllm_nginx

85 ```

86

@@ -129,7 +129,7 @@ Notes:

129

130 ## Launch Nginx

131

132 -```console

132 +```bash

133 docker run \

134 -itd \

135 -p 8000:80 \

@@ -142,7 +142,7 @@ docker run \

142

143 ## Verify That vLLM Servers Are Ready

144

145 -```console

145 +```bash

146 docker logs vllm0 | grep Uvicorn

147 docker logs vllm1 | grep Uvicorn

148 ```

docs/features/multimodal_inputs.md

+3 -3

@@ -307,7 +307,7 @@ Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for

307 By default, the timeout for fetching images through HTTP URL is `5` seconds.

308 You can override this by setting the environment variable:

309

310 - ```console

310 + ```bash

311 export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>

312 ```

313

@@ -370,7 +370,7 @@ Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for

370 By default, the timeout for fetching videos through HTTP URL is `30` seconds.

371 You can override this by setting the environment variable:

372

373 - ```console

373 + ```bash

374 export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>

375 ```

376

@@ -476,7 +476,7 @@ Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for

476 By default, the timeout for fetching audios through HTTP URL is `10` seconds.

477 You can override this by setting the environment variable:

478

479 - ```console

479 + ```bash

480 export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>

481 ```

482

docs/features/quantization/auto_awq.md

+2 -2

@@ -9,7 +9,7 @@ The main benefits are lower latency and memory usage.

9

10 You can quantize your own models by installing AutoAWQ or picking one of the [6500+ models on Huggingface](https://huggingface.co/models?search=awq).

11

12 -```console

12 +```bash

13 pip install autoawq

14 ```

15

@@ -43,7 +43,7 @@ After installing AutoAWQ, you are ready to quantize a model. Please refer to the

43

44 To run an AWQ model with vLLM, you can use [TheBloke/Llama-2-7b-Chat-AWQ](https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ) with the following command:

45

46 -```console

46 +```bash

47 python examples/offline_inference/llm_engine_example.py \

48 --model TheBloke/Llama-2-7b-Chat-AWQ \

49 --quantization awq

docs/features/quantization/bitblas.md

+1 -1

@@ -12,7 +12,7 @@ vLLM now supports [BitBLAS](https://github.com/microsoft/BitBLAS) for more effic

12

13 Below are the steps to utilize BitBLAS with vLLM.

14

15 -```console

15 +```bash

16 pip install bitblas>=0.1.0

17 ```

18

docs/features/quantization/bnb.md

+2 -2

@@ -9,7 +9,7 @@ Compared to other quantization methods, BitsAndBytes eliminates the need for cal

9

10 Below are the steps to utilize BitsAndBytes with vLLM.

11

12 -```console

12 +```bash

13 pip install bitsandbytes>=0.45.3

14 ```

15

@@ -54,6 +54,6 @@ llm = LLM(

54

55 Append the following to your model arguments for 4bit inflight quantization:

56

57 -```console

57 +```bash

58 --quantization bitsandbytes

59 ```

docs/features/quantization/fp8.md

+5 -5

@@ -23,7 +23,7 @@ The FP8 types typically supported in hardware have two distinct representations,

23

24 To produce performant FP8 quantized models with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:

25

26 -```console

26 +```bash

27 pip install llmcompressor

28 ```

29

@@ -81,7 +81,7 @@ Since simple RTN does not require data for weight quantization and the activatio

81

82 Install `vllm` and `lm-evaluation-harness` for evaluation:

83

84 -```console

84 +```bash

85 pip install vllm lm-eval==0.4.4

86 ```

87

@@ -99,9 +99,9 @@ Evaluate accuracy with `lm_eval` (for example on 250 samples of `gsm8k`):

99 !!! note

100 Quantized models can be sensitive to the presence of the `bos` token. `lm_eval` does not add a `bos` token by default, so make sure to include the `add_bos_token=True` argument when running your evaluations.

101

102 -```console

103 -$ MODEL=$PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic

104 -$ lm_eval \

102 +```bash

103 +MODEL=$PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic

104 +lm_eval \

105 --model vllm \

106 --model_args pretrained=$MODEL,add_bos_token=True \

107 --tasks gsm8k --num_fewshot 5 --batch_size auto --limit 250

docs/features/quantization/gguf.md

+3 -3

@@ -11,7 +11,7 @@ title: GGUF

11

12 To run a GGUF model with vLLM, you can download and use the local GGUF model from [TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF](https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF) with the following command:

13

14 -```console

14 +```bash

15 wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

16 # We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.

17 vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

@@ -20,7 +20,7 @@ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

20

21 You can also add `--tensor-parallel-size 2` to enable tensor parallelism inference with 2 GPUs:

22

23 -```console

23 +```bash

24 # We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.

25 vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

26 --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 \

@@ -32,7 +32,7 @@ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

32

33 GGUF assumes that huggingface can convert the metadata to a config file. In case huggingface doesn't support your model you can manually create a config and pass it as hf-config-path

34

35 -```console

35 +```bash

36 # If you model is not supported by huggingface you can manually provide a huggingface compatible config path

37 vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

38 --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 \

docs/features/quantization/gptqmodel.md

+2 -2

@@ -21,7 +21,7 @@ for more details on this and other advanced features.

21

22 You can quantize your own models by installing [GPTQModel](https://github.com/ModelCloud/GPTQModel) or picking one of the [5000+ models on Huggingface](https://huggingface.co/models?search=gptq).

23

24 -```console

24 +```bash

25 pip install -U gptqmodel --no-build-isolation -v

26 ```

27

@@ -60,7 +60,7 @@ Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:

60

61 To run an GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:

62

63 -```console

63 +```bash

64 python examples/offline_inference/llm_engine_example.py \

65 --model ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2

66 ```

docs/features/quantization/int4.md

+4 -4

@@ -14,13 +14,13 @@ Please visit the HF collection of [quantized INT4 checkpoints of popular LLMs re

14

15 To use INT4 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:

16

17 -```console

17 +```bash

18 pip install llmcompressor

19 ```

20

21 Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:

22

23 -```console

23 +```bash

24 pip install vllm lm-eval==0.4.4

25 ```

26

@@ -116,8 +116,8 @@ model = LLM("./Meta-Llama-3-8B-Instruct-W4A16-G128")

116

117 To evaluate accuracy, you can use `lm_eval`:

118

119 -```console

120 -$ lm_eval --model vllm \

119 +```bash

120 +lm_eval --model vllm \

121 --model_args pretrained="./Meta-Llama-3-8B-Instruct-W4A16-G128",add_bos_token=true \

122 --tasks gsm8k \

123 --num_fewshot 5 \

docs/features/quantization/int8.md

+4 -4

@@ -15,13 +15,13 @@ Please visit the HF collection of [quantized INT8 checkpoints of popular LLMs re

15

16 To use INT8 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:

17

18 -```console

18 +```bash

19 pip install llmcompressor

20 ```

21

22 Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:

23

24 -```console

24 +```bash

25 pip install vllm lm-eval==0.4.4

26 ```

27

@@ -122,8 +122,8 @@ model = LLM("./Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token")

122

123 To evaluate accuracy, you can use `lm_eval`:

124

125 -```console

126 -$ lm_eval --model vllm \

125 +```bash

126 +lm_eval --model vllm \

127 --model_args pretrained="./Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token",add_bos_token=true \

128 --tasks gsm8k \

129 --num_fewshot 5 \

docs/features/quantization/modelopt.md

+1 -1

@@ -4,7 +4,7 @@ The [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-O

4

5 We recommend installing the library with:

6

7 -```console

7 +```bash

8 pip install nvidia-modelopt

9 ```

10

docs/features/quantization/quantized_kvcache.md

+1 -1

@@ -65,7 +65,7 @@ For optimal model quality when using FP8 KV Cache, we recommend using calibrated

65

66 First, install the required dependencies:

67

68 -```console

68 +```bash

69 pip install llmcompressor

70 ```

71

docs/features/quantization/quark.md

+6 -6

@@ -13,7 +13,7 @@ AWQ, GPTQ, Rotation and SmoothQuant.

13

14 Before quantizing models, you need to install Quark. The latest release of Quark can be installed with pip:

15

16 -```console

16 +```bash

17 pip install amd-quark

18 ```

19

@@ -22,13 +22,13 @@ for more installation details.

22

23 Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:

24

25 -```console

25 +```bash

26 pip install vllm lm-eval==0.4.4

27 ```

28

29 ## Quantization Process

30

31 -After installing Quark, we will use an example to illustrate how to use Quark.

31 +After installing Quark, we will use an example to illustrate how to use Quark.

32 The Quark quantization process can be listed for 5 steps as below:

33

34 1. Load the model

@@ -209,8 +209,8 @@ Now, you can load and run the Quark quantized model directly through the LLM ent

209

210 Or, you can use `lm_eval` to evaluate accuracy:

211

212 -```console

213 -$ lm_eval --model vllm \

212 +```bash

213 +lm_eval --model vllm \

214 --model_args pretrained=Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant,kv_cache_dtype='fp8',quantization='quark' \

215 --tasks gsm8k

216 ```

@@ -222,7 +222,7 @@ to quantize large language models more conveniently. It supports quantizing mode

222 of different quantization schemes and optimization algorithms. It can export the quantized model

223 and run evaluation tasks on the fly. With the script, the example above can be:

224

225 -```console

225 +```bash

226 python3 quantize_quark.py --model_dir meta-llama/Llama-2-70b-chat-hf \

227 --output_dir /path/to/output \

228 --quant_scheme w_fp8_a_fp8 \

docs/features/quantization/torchao.md

+1 -1

@@ -4,7 +4,7 @@ TorchAO is an architecture optimization library for PyTorch, it provides high pe

4

5 We recommend installing the latest torchao nightly with

6

7 -```console

7 +```bash

8 # Install the latest TorchAO nightly build

9 # Choose the CUDA version that matches your system (cu126, cu128, etc.)

10 pip install \

docs/features/tool_calling.md

+1 -1

@@ -351,7 +351,7 @@ Here is a summary of a plugin file:

351

352 Then you can use this plugin in the command line like this.

353

354 -```console

354 +```bash

355 --enable-auto-tool-choice \

356 --tool-parser-plugin <absolute path of the plugin file>

357 --tool-call-parser example \

docs/getting_started/installation/aws_neuron.md

+5 -5

@@ -26,7 +26,7 @@ The easiest way to launch a Trainium or Inferentia instance with pre-installed N

26 - After launching the instance, follow the instructions in [Connect to your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html) to connect to the instance

27 - Once inside your instance, activate the pre-installed virtual environment for inference by running

28

29 -```console

29 +```bash

30 source /opt/aws_neuronx_venv_pytorch_2_6_nxd_inference/bin/activate

31 ```

32

@@ -47,7 +47,7 @@ Currently, there are no pre-built Neuron wheels.

47

48 To build and install vLLM from source, run:

49

50 -```console

50 +```bash

51 git clone https://github.com/vllm-project/vllm.git

52 cd vllm

53 pip install -U -r requirements/neuron.txt

@@ -66,7 +66,7 @@ Refer to [vLLM User Guide for NxD Inference](https://awsdocs-neuron.readthedocs-

66

67 To install the AWS Neuron fork, run the following:

68

69 -```console

69 +```bash

70 git clone -b neuron-2.23-vllm-v0.7.2 https://github.com/aws-neuron/upstreaming-to-vllm.git

71 cd upstreaming-to-vllm

72 pip install -r requirements/neuron.txt

@@ -100,7 +100,7 @@ to perform most of the heavy lifting which includes PyTorch model initialization

100 To configure NxD Inference features through the vLLM entrypoint, use the `override_neuron_config` setting. Provide the configs you want to override

101 as a dictionary (or JSON object when starting vLLM from the CLI). For example, to disable auto bucketing, include

102

103 -```console

103 +```python

104 override_neuron_config={

105 "enable_bucketing":False,

106 }

@@ -108,7 +108,7 @@ override_neuron_config={

108

109 or when launching vLLM from the CLI, pass

110

111 -```console

111 +```bash

112 --override-neuron-config "{\"enable_bucketing\":false}"

113 ```

114

docs/getting_started/installation/cpu.md

+9 -9

@@ -78,13 +78,13 @@ Currently, there are no pre-built CPU wheels.

78

79 ??? Commands

80

81 - ```console

82 - $ docker build -f docker/Dockerfile.cpu \

81 + ```bash

82 + docker build -f docker/Dockerfile.cpu \

83 --tag vllm-cpu-env \

84 --target vllm-openai .

85

86 - # Launching OpenAI server

87 - $ docker run --rm \

86 + # Launching OpenAI server

87 + docker run --rm \

88 --privileged=true \

89 --shm-size=4g \

90 -p 8000:8000 \

@@ -123,7 +123,7 @@ vLLM CPU backend supports the following vLLM features:

123

124 - We highly recommend to use TCMalloc for high performance memory allocation and better cache locality. For example, on Ubuntu 22.4, you can run:

125

126 -```console

126 +```bash

127 sudo apt-get install libtcmalloc-minimal4 # install TCMalloc library

128 find / -name *libtcmalloc* # find the dynamic link library path

129 export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:$LD_PRELOAD # prepend the library to LD_PRELOAD

@@ -132,7 +132,7 @@ python examples/offline_inference/basic/basic.py # run vLLM

132

133 - When using the online serving, it is recommended to reserve 1-2 CPU cores for the serving framework to avoid CPU oversubscription. For example, on a platform with 32 physical CPU cores, reserving CPU 30 and 31 for the framework and using CPU 0-29 for OpenMP:

134

135 -```console

135 +```bash

136 export VLLM_CPU_KVCACHE_SPACE=40

137 export VLLM_CPU_OMP_THREADS_BIND=0-29

138 vllm serve facebook/opt-125m

@@ -140,7 +140,7 @@ vllm serve facebook/opt-125m

140

141 or using default auto thread binding:

142

143 -```console

143 +```bash

144 export VLLM_CPU_KVCACHE_SPACE=40

145 export VLLM_CPU_NUM_OF_RESERVED_CPU=2

146 vllm serve facebook/opt-125m

@@ -189,7 +189,7 @@ vllm serve facebook/opt-125m

189

190 - Tensor Parallel is supported for serving and offline inferencing. In general each NUMA node is treated as one GPU card. Below is the example script to enable Tensor Parallel = 2 for serving:

191

192 - ```console

192 + ```bash

193 VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" \

194 vllm serve meta-llama/Llama-2-7b-chat-hf \

195 -tp=2 \

@@ -198,7 +198,7 @@ vllm serve facebook/opt-125m

198

199 or using default auto thread binding:

200

201 - ```console

201 + ```bash

202 VLLM_CPU_KVCACHE_SPACE=40 \

203 vllm serve meta-llama/Llama-2-7b-chat-hf \

204 -tp=2 \

docs/getting_started/installation/cpu/apple.inc.md

+2 -2

@@ -25,11 +25,11 @@ Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.

25

26 After installation of XCode and the Command Line Tools, which include Apple Clang, execute the following commands to build and install vLLM from the source.

27

28 -```console

28 +```bash

29 git clone https://github.com/vllm-project/vllm.git

30 cd vllm

31 pip install -r requirements/cpu.txt

32 -pip install -e .

32 +pip install -e .

33 ```

34

35 !!! note

docs/getting_started/installation/cpu/build.inc.md

+5 -5

@@ -1,6 +1,6 @@

1 First, install recommended compiler. We recommend to use `gcc/g++ >= 12.3.0` as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run:

2

3 -```console

3 +```bash

4 sudo apt-get update -y

5 sudo apt-get install -y gcc-12 g++-12 libnuma-dev python3-dev

6 sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12

@@ -8,14 +8,14 @@ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /

8

9 Second, clone vLLM project:

10

11 -```console

11 +```bash

12 git clone https://github.com/vllm-project/vllm.git vllm_source

13 cd vllm_source

14 ```

15

16 Third, install Python packages for vLLM CPU backend building:

17

18 -```console

18 +```bash

19 pip install --upgrade pip

20 pip install "cmake>=3.26.1" wheel packaging ninja "setuptools-scm>=8" numpy

21 pip install -v -r requirements/cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu

@@ -23,13 +23,13 @@ pip install -v -r requirements/cpu.txt --extra-index-url https://download.pytorc

23

24 Finally, build and install vLLM CPU backend:

25

26 -```console

26 +```bash

27 VLLM_TARGET_DEVICE=cpu python setup.py install

28 ```

29

30 If you want to develop vllm, install it in editable mode instead.

31

32 -```console

32 +```bash

33 VLLM_TARGET_DEVICE=cpu python setup.py develop

34 ```

35

docs/getting_started/installation/cpu/s390x.inc.md

+3 -3

@@ -26,7 +26,7 @@ Currently the CPU implementation for s390x architecture supports FP32 datatype o

26

27 Install the following packages from the package manager before building the vLLM. For example on RHEL 9.4:

28

29 -```console

29 +```bash

30 dnf install -y \

31 which procps findutils tar vim git gcc g++ make patch make cython zlib-devel \

32 libjpeg-turbo-devel libtiff-devel libpng-devel libwebp-devel freetype-devel harfbuzz-devel \

@@ -35,7 +35,7 @@ dnf install -y \

35

36 Install rust>=1.80 which is needed for `outlines-core` and `uvloop` python packages installation.

37

38 -```console

38 +```bash

39 curl https://sh.rustup.rs -sSf | sh -s -- -y && \

40 . "$HOME/.cargo/env"

41 ```

@@ -45,7 +45,7 @@ Execute the following commands to build and install vLLM from the source.

45 !!! tip

46 Please build the following dependencies, `torchvision`, `pyarrow` from the source before building vLLM.

47

48 -```console

48 +```bash

49 sed -i '/^torch/d' requirements-build.txt # remove torch from requirements-build.txt since we use nightly builds

50 pip install -v \

51 --extra-index-url https://download.pytorch.org/whl/nightly/cpu \

docs/getting_started/installation/google_tpu.md

+4 -4

@@ -68,7 +68,7 @@ For more information about using TPUs with GKE, see:

68

69 Create a TPU v5e with 4 TPU chips:

70

71 -```console

71 +```bash

72 gcloud alpha compute tpus queued-resources create QUEUED_RESOURCE_ID \

73 --node-id TPU_NAME \

74 --project PROJECT_ID \

@@ -156,13 +156,13 @@ See [deployment-docker-pre-built-image][deployment-docker-pre-built-image] for i

156

157 You can use <gh-file:docker/Dockerfile.tpu> to build a Docker image with TPU support.

158

159 -```console

159 +```bash

160 docker build -f docker/Dockerfile.tpu -t vllm-tpu .

161 ```

162

163 Run the Docker image with the following command:

164

165 -```console

165 +```bash

166 # Make sure to add `--privileged --net host --shm-size=16G`.

167 docker run --privileged --net host --shm-size=16G -it vllm-tpu

168 ```

@@ -185,6 +185,6 @@ docker run --privileged --net host --shm-size=16G -it vllm-tpu

185

186 Install OpenBLAS with the following command:

187

188 - ```console

188 + ```bash

189 sudo apt-get install --no-install-recommends --yes libopenblas-base libopenmpi-dev libomp-dev

190 ```

docs/getting_started/installation/gpu/cuda.inc.md

+17 -17

@@ -22,7 +22,7 @@ Therefore, it is recommended to install vLLM with a **fresh new** environment. I

22

23 You can install vLLM using either `pip` or `uv pip`:

24

25 -```console

25 +```bash

26 # Install vLLM with CUDA 12.8.

27 # If you are using pip.

28 pip install vllm --extra-index-url https://download.pytorch.org/whl/cu128

@@ -37,7 +37,7 @@ We recommend leveraging `uv` to [automatically select the appropriate PyTorch in

37

38 As of now, vLLM's binaries are compiled with CUDA 12.8 and public PyTorch release versions by default. We also provide vLLM binaries compiled with CUDA 12.6, 11.8, and public PyTorch release versions:

39

40 -```console

40 +```bash

41 # Install vLLM with CUDA 11.8.

42 export VLLM_VERSION=0.6.1.post1

43 export PYTHON_VERSION=312

@@ -52,7 +52,7 @@ LLM inference is a fast-evolving field, and the latest code may contain bug fixe

52

53 ##### Install the latest code using `pip`

54

55 -```console

55 +```bash

56 pip install -U vllm \

57 --pre \

58 --extra-index-url https://wheels.vllm.ai/nightly

@@ -62,7 +62,7 @@ pip install -U vllm \

62

63 Another way to install the latest code is to use `uv`:

64

65 -```console

65 +```bash

66 uv pip install -U vllm \

67 --torch-backend=auto \

68 --extra-index-url https://wheels.vllm.ai/nightly

@@ -72,7 +72,7 @@ uv pip install -U vllm \

72

73 If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), due to the limitation of `pip`, you have to specify the full URL of the wheel file by embedding the commit hash in the URL:

74

75 -```console

75 +```bash

76 export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch

77 pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl

78 ```

@@ -83,7 +83,7 @@ Note that the wheels are built with Python 3.8 ABI (see [PEP 425](https://peps.p

83

84 If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:

85

86 -```console

86 +```bash

87 export VLLM_COMMIT=72d9c316d3f6ede485146fe5aabd4e61dbc59069 # use full commit hash from the main branch

88 uv pip install vllm \

89 --torch-backend=auto \

@@ -99,7 +99,7 @@ The `uv` approach works for vLLM `v0.6.6` and later and offers an easy-to-rememb

99

100 If you only need to change Python code, you can build and install vLLM without compilation. Using `pip`'s [`--editable` flag](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs), changes you make to the code will be reflected when you run vLLM:

101

102 -```console

102 +```bash

103 git clone https://github.com/vllm-project/vllm.git

104 cd vllm

105 VLLM_USE_PRECOMPILED=1 pip install --editable .

@@ -118,7 +118,7 @@ This command will do the following:

118

119 In case you see an error about wheel not found when running the above command, it might be because the commit you based on in the main branch was just merged and the wheel is being built. In this case, you can wait for around an hour to try again, or manually assign the previous commit in the installation using the `VLLM_PRECOMPILED_WHEEL_LOCATION` environment variable.

120

121 -```console

121 +```bash

122 export VLLM_COMMIT=72d9c316d3f6ede485146fe5aabd4e61dbc59069 # use full commit hash from the main branch

123 export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl

124 pip install --editable .

@@ -134,7 +134,7 @@ You can find more information about vLLM's wheels in [install-the-latest-code][i

134

135 If you want to modify C++ or CUDA code, you'll need to build vLLM from source. This can take several minutes:

136

137 -```console

137 +```bash

138 git clone https://github.com/vllm-project/vllm.git

139 cd vllm

140 pip install -e .

@@ -160,7 +160,7 @@ There are scenarios where the PyTorch dependency cannot be easily installed via

160

161 To build vLLM using an existing PyTorch installation:

162

163 -```console

163 +```bash

164 git clone https://github.com/vllm-project/vllm.git

165 cd vllm

166 python use_existing_torch.py

@@ -173,7 +173,7 @@ pip install --no-build-isolation -e .

173 Currently, before starting the build process, vLLM fetches cutlass code from GitHub. However, there may be scenarios where you want to use a local version of cutlass instead.

174 To achieve this, you can set the environment variable VLLM_CUTLASS_SRC_DIR to point to your local cutlass directory.

175

176 -```console

176 +```bash

177 git clone https://github.com/vllm-project/vllm.git

178 cd vllm

179 VLLM_CUTLASS_SRC_DIR=/path/to/cutlass pip install -e .

@@ -184,7 +184,7 @@ VLLM_CUTLASS_SRC_DIR=/path/to/cutlass pip install -e .

184 To avoid your system being overloaded, you can limit the number of compilation jobs

185 to be run simultaneously, via the environment variable `MAX_JOBS`. For example:

186

187 -```console

187 +```bash

188 export MAX_JOBS=6

189 pip install -e .

190 ```

@@ -194,7 +194,7 @@ A side effect is a much slower build process.

194

195 Additionally, if you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.

196

197 -```console

197 +```bash

198 # Use `--ipc=host` to make sure the shared memory is large enough.

199 docker run \

200 --gpus all \

@@ -205,14 +205,14 @@ docker run \

205

206 If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from [the official website](https://developer.nvidia.com/cuda-toolkit-archive). After installation, set the environment variable `CUDA_HOME` to the installation path of CUDA Toolkit, and make sure that the `nvcc` compiler is in your `PATH`, e.g.:

207

208 -```console

208 +```bash

209 export CUDA_HOME=/usr/local/cuda

210 export PATH="${CUDA_HOME}/bin:$PATH"

211 ```

212

213 Here is a sanity check to verify that the CUDA Toolkit is correctly installed:

214

215 -```console

215 +```bash

216 nvcc --version # verify that nvcc is in your PATH

217 ${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME

218 ```

@@ -223,7 +223,7 @@ vLLM can fully run only on Linux but for development purposes, you can still bui

223

224 Simply disable the `VLLM_TARGET_DEVICE` environment variable before installing:

225

226 -```console

226 +```bash

227 export VLLM_TARGET_DEVICE=empty

228 pip install -e .

229 ```

@@ -238,7 +238,7 @@ See [deployment-docker-pre-built-image][deployment-docker-pre-built-image] for i

238

239 Another way to access the latest code is to use the docker images:

240

241 -```console

241 +```bash

242 export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch

243 docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}

244 ```

docs/getting_started/installation/gpu/rocm.inc.md

+12 -12

@@ -31,17 +31,17 @@ Currently, there are no pre-built ROCm wheels.

31

32 Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:

33

34 - ```console

34 + ```bash

35 # Install PyTorch

36 - $ pip uninstall torch -y

37 - $ pip install --no-cache-dir --pre torch --index-url https://download.pytorch.org/whl/nightly/rocm6.3

36 + pip uninstall torch -y

37 + pip install --no-cache-dir --pre torch --index-url https://download.pytorch.org/whl/nightly/rocm6.3

38 ```

39

40 1. Install [Triton flash attention for ROCm](https://github.com/ROCm/triton)

41

42 Install ROCm's Triton flash attention (the default triton-mlir branch) following the instructions from [ROCm/triton](https://github.com/ROCm/triton/blob/triton-mlir/README.md)

43

44 - ```console

44 + ```bash

45 python3 -m pip install ninja cmake wheel pybind11

46 pip uninstall -y triton

47 git clone https://github.com/OpenAI/triton.git

@@ -62,7 +62,7 @@ Currently, there are no pre-built ROCm wheels.

62

63 For example, for ROCm 6.3, suppose your gfx arch is `gfx90a`. To get your gfx architecture, run `rocminfo |grep gfx`.

64

65 - ```console

65 + ```bash

66 git clone https://github.com/ROCm/flash-attention.git

67 cd flash-attention

68 git checkout b7d29fb

@@ -76,7 +76,7 @@ Currently, there are no pre-built ROCm wheels.

76

77 3. If you choose to build AITER yourself to use a certain branch or commit, you can build AITER using the following steps:

78

79 - ```console

79 + ```bash

80 python3 -m pip uninstall -y aiter

81 git clone --recursive https://github.com/ROCm/aiter.git

82 cd aiter

@@ -148,7 +148,7 @@ If you choose to build this rocm_base image yourself, the steps are as follows.

148

149 It is important that the user kicks off the docker build using buildkit. Either the user put DOCKER_BUILDKIT=1 as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:

150

151 -```console

151 +```json

152 {

153 "features": {

154 "buildkit": true

@@ -158,7 +158,7 @@ It is important that the user kicks off the docker build using buildkit. Either

158

159 To build vllm on ROCm 6.3 for MI200 and MI300 series, you can use the default:

160

161 -```console

161 +```bash

162 DOCKER_BUILDKIT=1 docker build \

163 -f docker/Dockerfile.rocm_base \

164 -t rocm/vllm-dev:base .

@@ -169,7 +169,7 @@ DOCKER_BUILDKIT=1 docker build \

169 First, build a docker image from <gh-file:docker/Dockerfile.rocm> and launch a docker container from the image.

170 It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:

171

172 -```console

172 +```bash

173 {

174 "features": {

175 "buildkit": true

@@ -187,13 +187,13 @@ Their values can be passed in when running `docker build` with `--build-arg` opt

187

188 To build vllm on ROCm 6.3 for MI200 and MI300 series, you can use the default:

189

190 -```console

190 +```bash

191 DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-rocm .

192 ```

193

194 To build vllm on ROCm 6.3 for Radeon RX7900 series (gfx1100), you should pick the alternative base image:

195

196 -```console

196 +```bash

197 DOCKER_BUILDKIT=1 docker build \

198 --build-arg BASE_IMAGE="rocm/vllm-dev:navi_base" \

199 -f docker/Dockerfile.rocm \

@@ -205,7 +205,7 @@ To run the above docker image `vllm-rocm`, use the below command:

205

206 ??? Command

207

208 - ```console

208 + ```bash

209 docker run -it \

210 --network=host \

211 --group-add=video \

docs/getting_started/installation/gpu/xpu.inc.md

+6 -6

@@ -25,7 +25,7 @@ Currently, there are no pre-built XPU wheels.

25 - First, install required driver and Intel OneAPI 2025.0 or later.

26 - Second, install Python packages for vLLM XPU backend building:

27

28 -```console

28 +```bash

29 git clone https://github.com/vllm-project/vllm.git

30 cd vllm

31 pip install --upgrade pip

@@ -34,7 +34,7 @@ pip install -v -r requirements/xpu.txt

34

35 - Then, build and install vLLM XPU backend:

36

37 -```console

37 +```bash

38 VLLM_TARGET_DEVICE=xpu python setup.py install

39 ```

40

@@ -53,9 +53,9 @@ Currently, there are no pre-built XPU images.

53 # --8<-- [end:pre-built-images]

54 # --8<-- [start:build-image-from-source]

55

56 -```console

57 -$ docker build -f docker/Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .

58 -$ docker run -it \

56 +```bash

57 +docker build -f docker/Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .

58 +docker run -it \

59 --rm \

60 --network=host \

61 --device /dev/dri \

@@ -68,7 +68,7 @@ $ docker run -it \

68

69 XPU platform supports **tensor parallel** inference/serving and also supports **pipeline parallel** as a beta feature for online serving. We require Ray as the distributed runtime backend. For example, a reference execution like following:

70

71 -```console

71 +```bash

72 python -m vllm.entrypoints.openai.api_server \

73 --model=facebook/opt-13b \

74 --dtype=bfloat16 \

docs/getting_started/installation/intel_gaudi.md

+5 -5

@@ -24,7 +24,7 @@ please follow the methods outlined in the

24

25 To verify that the Intel Gaudi software was correctly installed, run:

26

27 -```console

27 +```bash

28 hl-smi # verify that hl-smi is in your PATH and each Gaudi accelerator is visible

29 apt list --installed | grep habana # verify that habanalabs-firmware-tools, habanalabs-graph, habanalabs-rdma-core, habanalabs-thunk and habanalabs-container-runtime are installed

30 pip list | grep habana # verify that habana-torch-plugin, habana-torch-dataloader, habana-pyhlml and habana-media-loader are installed

@@ -42,7 +42,7 @@ for more details.

42

43 Use the following commands to run a Docker image:

44

45 -```console

45 +```bash

46 docker pull vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest

47 docker run \

48 -it \

@@ -65,7 +65,7 @@ Currently, there are no pre-built Intel Gaudi wheels.

65

66 To build and install vLLM from source, run:

67

68 -```console

68 +```bash

69 git clone https://github.com/vllm-project/vllm.git

70 cd vllm

71 pip install -r requirements/hpu.txt

@@ -74,7 +74,7 @@ python setup.py develop

74

75 Currently, the latest features and performance optimizations are developed in Gaudi's [vLLM-fork](https://github.com/HabanaAI/vllm-fork) and we periodically upstream them to vLLM main repo. To install latest [HabanaAI/vLLM-fork](https://github.com/HabanaAI/vllm-fork), run the following:

76

77 -```console

77 +```bash

78 git clone https://github.com/HabanaAI/vllm-fork.git

79 cd vllm-fork

80 git checkout habana_main

@@ -90,7 +90,7 @@ Currently, there are no pre-built Intel Gaudi images.

90

91 ### Build image from source

92

93 -```console

93 +```bash

94 docker build -f docker/Dockerfile.hpu -t vllm-hpu-env .

95 docker run \

96 -it \

docs/getting_started/installation/python_env_setup.inc.md

+1 -1

@@ -1,6 +1,6 @@

1 It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment and install vLLM using the following commands:

2

3 -```console

3 +```bash

4 uv venv --python 3.12 --seed

5 source .venv/bin/activate

6 ```

docs/getting_started/quickstart.md

+7 -7

@@ -19,7 +19,7 @@ If you are using NVIDIA GPUs, you can install vLLM using [pip](https://pypi.org/

19

20 It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment and install vLLM using the following commands:

21

22 -```console

22 +```bash

23 uv venv --python 3.12 --seed

24 source .venv/bin/activate

25 uv pip install vllm --torch-backend=auto

@@ -29,13 +29,13 @@ uv pip install vllm --torch-backend=auto

29

30 Another delightful way is to use `uv run` with `--with [dependency]` option, which allows you to run commands such as `vllm serve` without creating any permanent environment:

31

32 -```console

32 +```bash

33 uv run --with vllm vllm --help

34 ```

35

36 You can also use [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to create and manage Python environments. You can install `uv` to the conda environment through `pip` if you want to manage it within the environment.

37

38 -```console

38 +```bash

39 conda create -n myenv python=3.12 -y

40 conda activate myenv

41 pip install --upgrade uv

@@ -110,7 +110,7 @@ By default, it starts the server at `http://localhost:8000`. You can specify the

110

111 Run the following command to start the vLLM server with the [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) model:

112

113 -```console

113 +```bash

114 vllm serve Qwen/Qwen2.5-1.5B-Instruct

115 ```

116

@@ -124,7 +124,7 @@ vllm serve Qwen/Qwen2.5-1.5B-Instruct

124

125 This server can be queried in the same format as OpenAI API. For example, to list the models:

126

127 -```console

127 +```bash

128 curl http://localhost:8000/v1/models

129 ```

130

@@ -134,7 +134,7 @@ You can pass in the argument `--api-key` or environment variable `VLLM_API_KEY`

134

135 Once your server is started, you can query the model with input prompts:

136

137 -```console

137 +```bash

138 curl http://localhost:8000/v1/completions \

139 -H "Content-Type: application/json" \

140 -d '{

@@ -172,7 +172,7 @@ vLLM is designed to also support the OpenAI Chat Completions API. The chat inter

172

173 You can use the [create chat completion](https://platform.openai.com/docs/api-reference/chat/completions/create) endpoint to interact with the model:

174

175 -```console

175 +```bash

176 curl http://localhost:8000/v1/chat/completions \

177 -H "Content-Type: application/json" \

178 -d '{

docs/models/extensions/runai_model_streamer.md

+9 -9

@@ -9,27 +9,27 @@ Further reading can be found in [Run:ai Model Streamer Documentation](https://gi

9 vLLM supports loading weights in Safetensors format using the Run:ai Model Streamer.

10 You first need to install vLLM RunAI optional dependency:

11

12 -```console

12 +```bash

13 pip3 install vllm[runai]

14 ```

15

16 To run it as an OpenAI-compatible server, add the `--load-format runai_streamer` flag:

17

18 -```console

18 +```bash

19 vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \

20 --load-format runai_streamer

21 ```

22

23 To run model from AWS S3 object store run:

24

25 -```console

25 +```bash

26 vllm serve s3://core-llm/Llama-3-8b \

27 --load-format runai_streamer

28 ```

29

30 To run model from a S3 compatible object store run:

31

32 -```console

32 +```bash

33 RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING=0 \

34 AWS_EC2_METADATA_DISABLED=true \

35 AWS_ENDPOINT_URL=https://storage.googleapis.com \

@@ -44,7 +44,7 @@ You can tune parameters using `--model-loader-extra-config`:

44 You can tune `concurrency` that controls the level of concurrency and number of OS threads reading tensors from the file to the CPU buffer.

45 For reading from S3, it will be the number of client instances the host is opening to the S3 server.

46

47 -```console

47 +```bash

48 vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \

49 --load-format runai_streamer \

50 --model-loader-extra-config '{"concurrency":16}'

@@ -53,7 +53,7 @@ vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \

53 You can control the size of the CPU Memory buffer to which tensors are read from the file, and limit this size.

54 You can read further about CPU buffer memory limiting [here](https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/env-vars.md#runai_streamer_memory_limit).

55

56 -```console

56 +```bash

57 vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \

58 --load-format runai_streamer \

59 --model-loader-extra-config '{"memory_limit":5368709120}'

@@ -66,13 +66,13 @@ vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \

66

67 vLLM also supports loading sharded models using Run:ai Model Streamer. This is particularly useful for large models that are split across multiple files. To use this feature, use the `--load-format runai_streamer_sharded` flag:

68

69 -```console

69 +```bash

70 vllm serve /path/to/sharded/model --load-format runai_streamer_sharded

71 ```

72

73 The sharded loader expects model files to follow the same naming pattern as the regular sharded state loader: `model-rank-{rank}-part-{part}.safetensors`. You can customize this pattern using the `pattern` parameter in `--model-loader-extra-config`:

74

75 -```console

75 +```bash

76 vllm serve /path/to/sharded/model \

77 --load-format runai_streamer_sharded \

78 --model-loader-extra-config '{"pattern":"custom-model-rank-{rank}-part-{part}.safetensors"}'

@@ -82,7 +82,7 @@ To create sharded model files, you can use the script provided in <gh-file:examp

82

83 The sharded loader supports all the same tunable parameters as the regular Run:ai Model Streamer, including `concurrency` and `memory_limit`. These can be configured in the same way:

84

85 -```console

85 +```bash

86 vllm serve /path/to/sharded/model \

87 --load-format runai_streamer_sharded \

88 --model-loader-extra-config '{"concurrency":16, "memory_limit":5368709120}'

docs/models/supported_models.md

+2 -2

@@ -178,7 +178,7 @@ Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project

178

179 If you prefer, you can use the Hugging Face CLI to [download a model](https://huggingface.co/docs/huggingface_hub/guides/cli#huggingface-cli-download) or specific files from a model repository:

180

181 -```console

181 +```bash

182 # Download a model

183 huggingface-cli download HuggingFaceH4/zephyr-7b-beta

184

@@ -193,7 +193,7 @@ huggingface-cli download HuggingFaceH4/zephyr-7b-beta eval_results.json

193

194 Use the Hugging Face CLI to [manage models](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#scan-your-cache) stored in local cache:

195

196 -```console

196 +```bash

197 # List cached models

198 huggingface-cli scan-cache

199

docs/serving/distributed_serving.md

+8 -8

@@ -34,15 +34,15 @@ output = llm.generate("San Francisco is a")

34

35 To run multi-GPU serving, pass in the `--tensor-parallel-size` argument when starting the server. For example, to run API server on 4 GPUs:

36

37 -```console

38 - vllm serve facebook/opt-13b \

37 +```bash

38 +vllm serve facebook/opt-13b \

39 --tensor-parallel-size 4

40 ```

41

42 You can also additionally specify `--pipeline-parallel-size` to enable pipeline parallelism. For example, to run API server on 8 GPUs with pipeline parallelism and tensor parallelism:

43

44 -```console

45 - vllm serve gpt2 \

44 +```bash

45 +vllm serve gpt2 \

46 --tensor-parallel-size 4 \

47 --pipeline-parallel-size 2

48 ```

@@ -55,7 +55,7 @@ The first step, is to start containers and organize them into a cluster. We have

55

56 Pick a node as the head node, and run the following command:

57

58 -```console

58 +```bash

59 bash run_cluster.sh \

60 vllm/vllm-openai \

61 ip_of_head_node \

@@ -66,7 +66,7 @@ bash run_cluster.sh \

66

67 On the rest of the worker nodes, run the following command:

68

69 -```console

69 +```bash

70 bash run_cluster.sh \

71 vllm/vllm-openai \

72 ip_of_head_node \

@@ -87,7 +87,7 @@ Then, on any node, use `docker exec -it node /bin/bash` to enter the container,

87

88 After that, on any node, use `docker exec -it node /bin/bash` to enter the container again. **In the container**, you can use vLLM as usual, just as you have all the GPUs on one node: vLLM will be able to leverage GPU resources of all nodes in the Ray cluster, and therefore, only run the `vllm` command on this node but not other nodes. The common practice is to set the tensor parallel size to the number of GPUs in each node, and the pipeline parallel size to the number of nodes. For example, if you have 16 GPUs in 2 nodes (8 GPUs per node), you can set the tensor parallel size to 8 and the pipeline parallel size to 2:

89

90 -```console

90 +```bash

91 vllm serve /path/to/the/model/in/the/container \

92 --tensor-parallel-size 8 \

93 --pipeline-parallel-size 2

@@ -95,7 +95,7 @@ After that, on any node, use `docker exec -it node /bin/bash` to enter the conta

95

96 You can also use tensor parallel without pipeline parallel, just set the tensor parallel size to the number of GPUs in the cluster. For example, if you have 16 GPUs in 2 nodes (8 GPUs per node), you can set the tensor parallel size to 16:

97

98 -```console

98 +```bash

99 vllm serve /path/to/the/model/in/the/container \

100 --tensor-parallel-size 16

101 ```

docs/serving/integrations/langchain.md

+1 -1

@@ -7,7 +7,7 @@ vLLM is also available via [LangChain](https://github.com/langchain-ai/langchain

7

8 To install LangChain, run

9

10 -```console

10 +```bash

11 pip install langchain langchain_community -q

12 ```

13

docs/serving/integrations/llamaindex.md

+1 -1

@@ -7,7 +7,7 @@ vLLM is also available via [LlamaIndex](https://github.com/run-llama/llama_index

7

8 To install LlamaIndex, run

9

10 -```console

10 +```bash

11 pip install llama-index-llms-vllm -q

12 ```

13

docs/usage/metrics.md

+1 -1

@@ -6,7 +6,7 @@ OpenAI compatible API server.

6

7 You can start the server using Python, or using [Docker][deployment-docker]:

8

9 -```console

9 +```bash

10 vllm serve unsloth/Llama-3.2-1B-Instruct

11 ```

12

docs/usage/troubleshooting.md

+2 -2

@@ -127,13 +127,13 @@ If GPU/CPU communication cannot be established, you can use the following Python

127

128 If you are testing with a single node, adjust `--nproc-per-node` to the number of GPUs you want to use:

129

130 -```console

130 +```bash

131 NCCL_DEBUG=TRACE torchrun --nproc-per-node=<number-of-GPUs> test.py

132 ```

133

134 If you are testing with multi-nodes, adjust `--nproc-per-node` and `--nnodes` according to your setup and set `MASTER_ADDR` to the correct IP address of the master node, reachable from all nodes. Then, run:

135

136 -```console

136 +```bash

137 NCCL_DEBUG=TRACE torchrun --nnodes 2 \

138 --nproc-per-node=2 \

139 --rdzv_backend=c10d \

examples/offline_inference/openai_batch/README.md

+20 -20

@@ -29,14 +29,14 @@ We currently support `/v1/chat/completions`, `/v1/embeddings`, and `/v1/score` e

29

30 To follow along with this example, you can download the example batch, or create your own batch file in your working directory.

31

32 -```console

32 +```bash

33 wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl

34 ```

35

36 Once you've created your batch file it should look like this

37

38 -```console

39 -$ cat offline_inference/openai_batch/openai_example_batch.jsonl

38 +```bash

39 +cat offline_inference/openai_batch/openai_example_batch.jsonl

40 {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}

41 {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}

42 ```

@@ -47,7 +47,7 @@ The batch running tool is designed to be used from the command line.

47

48 You can run the batch with the following command, which will write its results to a file called `results.jsonl`

49

50 -```console

50 +```bash

51 python -m vllm.entrypoints.openai.run_batch \

52 -i offline_inference/openai_batch/openai_example_batch.jsonl \

53 -o results.jsonl \

@@ -56,7 +56,7 @@ python -m vllm.entrypoints.openai.run_batch \

56

57 or use command-line:

58

59 -```console

59 +```bash

60 vllm run-batch \

61 -i offline_inference/openai_batch/openai_example_batch.jsonl \

62 -o results.jsonl \

@@ -67,8 +67,8 @@ vllm run-batch \

67

68 You should now have your results at `results.jsonl`. You can check your results by running `cat results.jsonl`

69

70 -```console

71 -$ cat results.jsonl

70 +```bash

71 +cat results.jsonl

72 {"id":"vllm-383d1c59835645aeb2e07d004d62a826","custom_id":"request-1","response":{"id":"cmpl-61c020e54b964d5a98fa7527bfcdd378","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! It's great to meet you! I'm here to help with any questions or tasks you may have. What's on your mind today?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":25,"total_tokens":56,"completion_tokens":31}},"error":null}

73 {"id":"vllm-42e3d09b14b04568afa3f1797751a267","custom_id":"request-2","response":{"id":"cmpl-f44d049f6b3a42d4b2d7850bb1e31bcc","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"*silence*"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":27,"total_tokens":32,"completion_tokens":5}},"error":null}

74 ```

@@ -79,7 +79,7 @@ The batch runner supports remote input and output urls that are accessible via h

79

80 For example, to run against our example input file located at `https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl`, you can run

81

82 -```console

82 +```bash

83 python -m vllm.entrypoints.openai.run_batch \

84 -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl \

85 -o results.jsonl \

@@ -88,7 +88,7 @@ python -m vllm.entrypoints.openai.run_batch \

88

89 or use command-line:

90

91 -```console

91 +```bash

92 vllm run-batch \

93 -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl \

94 -o results.jsonl \

@@ -112,21 +112,21 @@ To integrate with cloud blob storage, we recommend using presigned urls.

112

113 To follow along with this example, you can download the example batch, or create your own batch file in your working directory.

114

115 -```console

115 +```bash

116 wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl

117 ```

118

119 Once you've created your batch file it should look like this

120

121 -```console

122 -$ cat offline_inference/openai_batch/openai_example_batch.jsonl

121 +```bash

122 +cat offline_inference/openai_batch/openai_example_batch.jsonl

123 {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}

124 {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}

125 ```

126

127 Now upload your batch file to your S3 bucket.

128

129 -```console

129 +```bash

130 aws s3 cp offline_inference/openai_batch/openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl

131 ```

132

@@ -181,7 +181,7 @@ output_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AW

181

182 You can now run the batch runner, using the urls generated in the previous section.

183

184 -```console

184 +```bash

185 python -m vllm.entrypoints.openai.run_batch \

186 -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \

187 -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \

@@ -190,7 +190,7 @@ python -m vllm.entrypoints.openai.run_batch \

190

191 or use command-line:

192

193 -```console

193 +```bash

194 vllm run-batch \

195 -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \

196 -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \

@@ -201,7 +201,7 @@ vllm run-batch \

201

202 Your results are now on S3. You can view them in your terminal by running

203

204 -```console

204 +```bash

205 aws s3 cp s3://MY_BUCKET/MY_OUTPUT_FILE.jsonl -

206 ```

207

@@ -230,8 +230,8 @@ You can run the batch using the same command as in earlier examples.

230

231 You can check your results by running `cat results.jsonl`

232

233 -```console

234 -$ cat results.jsonl

233 +```bash

234 +cat results.jsonl

235 {"id":"vllm-db0f71f7dec244e6bce530e0b4ef908b","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-3580bf4d4ae54d52b67eee266a6eab20","body":{"id":"embd-33ac2efa7996430184461f2e38529746","object":"list","created":444647,"model":"intfloat/e5-mistral-7b-instruct","data":[{"index":0,"object":"embedding","embedding":[0.016204833984375,0.0092010498046875,0.0018358230590820312,-0.0028228759765625,0.001422882080078125,-0.0031147003173828125,...]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0}}},"error":null}

236 ...

237 ```

@@ -261,8 +261,8 @@ You can run the batch using the same command as in earlier examples.

261

262 You can check your results by running `cat results.jsonl`

263

264 -```console

265 -$ cat results.jsonl

264 +```bash

265 +cat results.jsonl

266 {"id":"vllm-f87c5c4539184f618e555744a2965987","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-806ab64512e44071b37d3f7ccd291413","body":{"id":"score-4ee45236897b4d29907d49b01298cdb1","object":"list","created":1737847944,"model":"BAAI/bge-reranker-v2-m3","data":[{"index":0,"object":"score","score":0.0010900497436523438},{"index":1,"object":"score","score":1.0}],"usage":{"prompt_tokens":37,"total_tokens":37,"completion_tokens":0,"prompt_tokens_details":null}}},"error":null}

267 {"id":"vllm-41990c51a26d4fac8419077f12871099","custom_id":"request-2","response":{"status_code":200,"request_id":"vllm-batch-73ce66379026482699f81974e14e1e99","body":{"id":"score-13f2ffe6ba40460fbf9f7f00ad667d75","object":"list","created":1737847944,"model":"BAAI/bge-reranker-v2-m3","data":[{"index":0,"object":"score","score":0.001094818115234375},{"index":1,"object":"score","score":1.0}],"usage":{"prompt_tokens":37,"total_tokens":37,"completion_tokens":0,"prompt_tokens_details":null}}},"error":null}

268 ```

examples/online_serving/opentelemetry/README.md

+8 -8

@@ -2,7 +2,7 @@

2

3 1. Install OpenTelemetry packages:

4

5 - ```console

5 + ```bash

6 pip install \

7 'opentelemetry-sdk>=1.26.0,<1.27.0' \

8 'opentelemetry-api>=1.26.0,<1.27.0' \

@@ -12,7 +12,7 @@

12

13 1. Start Jaeger in a docker container:

14

15 - ```console

15 + ```bash

16 # From: https://www.jaegertracing.io/docs/1.57/getting-started/

17 docker run --rm --name jaeger \

18 -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \

@@ -31,14 +31,14 @@

31

32 1. In a new shell, export Jaeger IP:

33

34 - ```console

34 + ```bash

35 export JAEGER_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' jaeger)

36 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317

37 ```

38

39 Then set vLLM's service name for OpenTelemetry, enable insecure connections to Jaeger and run vLLM:

40

41 - ```console

41 + ```bash

42 export OTEL_SERVICE_NAME="vllm-server"

43 export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true

44 vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"

@@ -46,7 +46,7 @@

46

47 1. In a new shell, send requests with trace context from a dummy client

48

49 - ```console

49 + ```bash

50 export JAEGER_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' jaeger)

51 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317

52 export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true

@@ -67,7 +67,7 @@

67 OpenTelemetry supports either `grpc` or `http/protobuf` as the transport protocol for trace data in the exporter.

68 By default, `grpc` is used. To set `http/protobuf` as the protocol, configure the `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` environment variable as follows:

69

70 -```console

70 +```bash

71 export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf

72 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://$JAEGER_IP:4318/v1/traces

73 vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"

@@ -79,13 +79,13 @@ OpenTelemetry allows automatic instrumentation of FastAPI.

79

80 1. Install the instrumentation library

81

82 - ```console

82 + ```bash

83 pip install opentelemetry-instrumentation-fastapi

84 ```

85

86 1. Run vLLM with `opentelemetry-instrument`

87

88 - ```console

88 + ```bash

89 opentelemetry-instrument vllm serve facebook/opt-125m

90 ```

91

[Docs] Fix syntax highlighting of shell commands (#19870)