279f854519 · tduncan/Nvidia-Optimization-Stack-VLLM

docs/contributing/dockerfile/dockerfile.md

+6 -1

@@ -26,7 +26,12 @@ The edges of the build graph represent:

26 > Commands to regenerate the build graph (make sure to run it **from the \`root\` directory of the vLLM repository** where the dockerfile is present):

27 >

28 > ```bash

29 - > dockerfilegraph -o png --legend --dpi 200 --max-label-length 50 --filename docker/Dockerfile

29 + > dockerfilegraph \

30 + > -o png \

31 + > --legend \

32 + > --dpi 200 \

33 + > --max-label-length 50 \

34 + > --filename docker/Dockerfile

35 > ```

36 >

37 > or in case you want to run it directly with the docker image:

docs/contributing/model/registration.md

+4 -1

@@ -41,7 +41,10 @@ If your model imports modules that initialize CUDA, consider lazy-importing it t

41 ```python

42 from vllm import ModelRegistry

43

44 -ModelRegistry.register_model("YourModelForCausalLM", "your_code:YourModelForCausalLM")

44 +ModelRegistry.register_model(

45 + "YourModelForCausalLM",

46 + "your_code:YourModelForCausalLM"

47 +)

48 ```

49

50 !!! warning

docs/deployment/docker.md

+9 -6

@@ -11,7 +11,7 @@ vLLM offers an official Docker image for deployment.

11 The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai](https://hub.docker.com/r/vllm/vllm-openai/tags).

12

13 ```console

14 -$ docker run --runtime nvidia --gpus all \

14 +docker run --runtime nvidia --gpus all \

15 -v ~/.cache/huggingface:/root/.cache/huggingface \

16 --env "HUGGING_FACE_HUB_TOKEN=<secret>" \

17 -p 8000:8000 \

@@ -23,7 +23,7 @@ $ docker run --runtime nvidia --gpus all \

23 This image can also be used with other container engines such as [Podman](https://podman.io/).

24

25 ```console

26 -$ podman run --gpus all \

26 +podman run --gpus all \

27 -v ~/.cache/huggingface:/root/.cache/huggingface \

28 --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \

29 -p 8000:8000 \

@@ -73,7 +73,10 @@ You can build and run vLLM from source via the provided <gh-file:docker/Dockerfi

73

74 ```console

75 # optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2

76 -DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai --file docker/Dockerfile

76 +DOCKER_BUILDKIT=1 docker build . \

77 + --target vllm-openai \

78 + --tag vllm/vllm-openai \

79 + --file docker/Dockerfile

80 ```

81

82 !!! note

@@ -96,8 +99,8 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--

99

100 ```console

101 # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)

99 -$ python3 use_existing_torch.py

100 -$ DOCKER_BUILDKIT=1 docker build . \

102 +python3 use_existing_torch.py

103 +DOCKER_BUILDKIT=1 docker build . \

104 --file docker/Dockerfile \

105 --target vllm-openai \

106 --platform "linux/arm64" \

@@ -113,7 +116,7 @@ $ DOCKER_BUILDKIT=1 docker build . \

116 To run vLLM with the custom-built Docker image:

117

118 ```console

116 -$ docker run --runtime nvidia --gpus all \

119 +docker run --runtime nvidia --gpus all \

120 -v ~/.cache/huggingface:/root/.cache/huggingface \

121 -p 8000:8000 \

122 --env "HUGGING_FACE_HUB_TOKEN=<secret>" \

docs/deployment/frameworks/skypilot.md

+11 -3

@@ -82,7 +82,11 @@ Check the output of the command. There will be a shareable gradio link (like the

82 **Optional**: Serve the 70B model instead of the default 8B and use more GPU:

83

84 ```console

85 -HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct

85 +HF_TOKEN="your-huggingface-token" \

86 + sky launch serving.yaml \

87 + --gpus A100:8 \

88 + --env HF_TOKEN \

89 + --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct

90 ```

91

92 ## Scale up to multiple replicas

@@ -155,7 +159,9 @@ run: |

159 Start the serving the Llama-3 8B model on multiple replicas:

160

161 ```console

158 -HF_TOKEN="your-huggingface-token" sky serve up -n vllm serving.yaml --env HF_TOKEN

162 +HF_TOKEN="your-huggingface-token" \

163 + sky serve up -n vllm serving.yaml \

164 + --env HF_TOKEN

165 ```

166

167 Wait until the service is ready:

@@ -318,7 +324,9 @@ run: |

324 1. Start the chat web UI:

325

326 ```console

321 - sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)

327 + sky launch \

328 + -c gui ./gui.yaml \

329 + --env ENDPOINT=$(sky serve status --endpoint vllm)

330 ```

331

332 2. Then, we can access the GUI at the returned gradio link:

docs/deployment/frameworks/streamlit.md

+2 -1

@@ -33,7 +33,8 @@ pip install streamlit openai

33 streamlit run streamlit_openai_chatbot_webserver.py

34

35 # or specify the VLLM_API_BASE or VLLM_API_KEY

36 -VLLM_API_BASE="http://vllm-server-host:vllm-server-port/v1" streamlit run streamlit_openai_chatbot_webserver.py

36 +VLLM_API_BASE="http://vllm-server-host:vllm-server-port/v1" \

37 + streamlit run streamlit_openai_chatbot_webserver.py

38

39 # start with debug mode to view more details

40 streamlit run streamlit_openai_chatbot_webserver.py --logger.level=debug

docs/deployment/nginx.md

+31 -4

@@ -77,7 +77,11 @@ If you are behind proxy, you can pass the proxy settings to the docker build com

77

78 ```console

79 cd $vllm_root

80 -docker build -f docker/Dockerfile . --tag vllm --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy

80 +docker build \

81 + -f docker/Dockerfile . \

82 + --tag vllm \

83 + --build-arg http_proxy=$http_proxy \

84 + --build-arg https_proxy=$https_proxy

85 ```

86

87 [](){ #nginxloadbalancer-nginx-docker-network }

@@ -102,8 +106,26 @@ Notes:

106 ```console

107 mkdir -p ~/.cache/huggingface/hub/

108 hf_cache_dir=~/.cache/huggingface/

105 -docker run -itd --ipc host --network vllm_nginx --gpus device=0 --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8081:8000 --name vllm0 vllm --model meta-llama/Llama-2-7b-chat-hf

106 -docker run -itd --ipc host --network vllm_nginx --gpus device=1 --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8082:8000 --name vllm1 vllm --model meta-llama/Llama-2-7b-chat-hf

109 +docker run \

110 + -itd \

111 + --ipc host \

112 + --network vllm_nginx \

113 + --gpus device=0 \

114 + --shm-size=10.24gb \

115 + -v $hf_cache_dir:/root/.cache/huggingface/ \

116 + -p 8081:8000 \

117 + --name vllm0 vllm \

118 + --model meta-llama/Llama-2-7b-chat-hf

119 +docker run \

120 + -itd \

121 + --ipc host \

122 + --network vllm_nginx \

123 + --gpus device=1 \

124 + --shm-size=10.24gb \

125 + -v $hf_cache_dir:/root/.cache/huggingface/ \

126 + -p 8082:8000 \

127 + --name vllm1 vllm \

128 + --model meta-llama/Llama-2-7b-chat-hf

129 ```

130

131 !!! note

@@ -114,7 +136,12 @@ docker run -itd --ipc host --network vllm_nginx --gpus device=1 --shm-size=10.24

136 ## Launch Nginx

137

138 ```console

117 -docker run -itd -p 8000:80 --network vllm_nginx -v ./nginx_conf/:/etc/nginx/conf.d/ --name nginx-lb nginx-lb:latest

139 +docker run \

140 + -itd \

141 + -p 8000:80 \

142 + --network vllm_nginx \

143 + -v ./nginx_conf/:/etc/nginx/conf.d/ \

144 + --name nginx-lb nginx-lb:latest

145 ```

146

147 [](){ #nginxloadbalancer-nginx-verify-nginx }

docs/features/quantization/auto_awq.md

+3 -1

@@ -42,7 +42,9 @@ print(f'Model is quantized and saved at "{quant_path}"')

42 To run an AWQ model with vLLM, you can use [TheBloke/Llama-2-7b-Chat-AWQ](https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ) with the following command:

43

44 ```console

45 -python examples/offline_inference/llm_engine_example.py --model TheBloke/Llama-2-7b-Chat-AWQ --quantization awq

45 +python examples/offline_inference/llm_engine_example.py \

46 + --model TheBloke/Llama-2-7b-Chat-AWQ \

47 + --quantization awq

48 ```

49

50 AWQ models are also supported directly through the LLM entrypoint:

docs/features/quantization/bitblas.md

+13 -2

@@ -33,7 +33,12 @@ import torch

33

34 # "hxbgsyxh/llama-13b-4bit-g-1-bitblas" is a pre-quantized checkpoint.

35 model_id = "hxbgsyxh/llama-13b-4bit-g-1-bitblas"

36 -llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True, quantization="bitblas")

36 +llm = LLM(

37 + model=model_id,

38 + dtype=torch.bfloat16,

39 + trust_remote_code=True,

40 + quantization="bitblas"

41 +)

42 ```

43

44 ## Read gptq format checkpoint

@@ -44,5 +49,11 @@ import torch

49

50 # "hxbgsyxh/llama-13b-4bit-g-1" is a pre-quantized checkpoint.

51 model_id = "hxbgsyxh/llama-13b-4bit-g-1"

47 -llm = LLM(model=model_id, dtype=torch.float16, trust_remote_code=True, quantization="bitblas", max_model_len=1024)

52 +llm = LLM(

53 + model=model_id,

54 + dtype=torch.float16,

55 + trust_remote_code=True,

56 + quantization="bitblas",

57 + max_model_len=1024

58 +)

59 ```

docs/features/quantization/bnb.md

+11 -3

@@ -27,7 +27,11 @@ from vllm import LLM

27 import torch

28 # unsloth/tinyllama-bnb-4bit is a pre-quantized checkpoint.

29 model_id = "unsloth/tinyllama-bnb-4bit"

30 -llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True)

30 +llm = LLM(

31 + model=model_id,

32 + dtype=torch.bfloat16,

33 + trust_remote_code=True

34 +)

35 ```

36

37 ## Inflight quantization: load as 4bit quantization

@@ -38,8 +42,12 @@ For inflight 4bit quantization with BitsAndBytes, you need to explicitly specify

42 from vllm import LLM

43 import torch

44 model_id = "huggyllama/llama-7b"

41 -llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True, \

42 -quantization="bitsandbytes")

45 +llm = LLM(

46 + model=model_id,

47 + dtype=torch.bfloat16,

48 + trust_remote_code=True,

49 + quantization="bitsandbytes"

50 +)

51 ```

52

53 ## OpenAI Compatible Server

docs/features/quantization/gguf.md

+8 -3

@@ -14,14 +14,17 @@ To run a GGUF model with vLLM, you can download and use the local GGUF model fro

14 ```console

15 wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

16 # We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.

17 -vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0

17 +vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

18 + --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0

19 ```

20

21 You can also add `--tensor-parallel-size 2` to enable tensor parallelism inference with 2 GPUs:

22

23 ```console

24 # We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.

24 -vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 --tensor-parallel-size 2

25 +vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

26 + --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 \

27 + --tensor-parallel-size 2

28 ```

29

30 !!! warning

@@ -31,7 +34,9 @@ GGUF assumes that huggingface can convert the metadata to a config file. In case

34

35 ```console

36 # If you model is not supported by huggingface you can manually provide a huggingface compatible config path

34 -vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 --hf-config-path Tinyllama/TInyLlama-1.1B-Chat-v1.0

37 +vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

38 + --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 \

39 + --hf-config-path Tinyllama/TInyLlama-1.1B-Chat-v1.0

40 ```

41

42 You can also use the GGUF model directly through the LLM entrypoint:

docs/features/quantization/gptqmodel.md

+2 -1

@@ -59,7 +59,8 @@ model.save(quant_path)

59 To run an GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:

60

61 ```console

62 -python examples/offline_inference/llm_engine_example.py --model ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2

62 +python examples/offline_inference/llm_engine_example.py \

63 + --model ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2

64 ```

65

66 ## Using GPTQModel with vLLM's Python API

docs/features/quantization/torchao.md

+9 -2

@@ -7,7 +7,9 @@ We recommend installing the latest torchao nightly with

7 ```console

8 # Install the latest TorchAO nightly build

9 # Choose the CUDA version that matches your system (cu126, cu128, etc.)

10 -pip install --pre torchao>=10.0.0 --index-url https://download.pytorch.org/whl/nightly/cu126

10 +pip install \

11 + --pre torchao>=10.0.0 \

12 + --index-url https://download.pytorch.org/whl/nightly/cu126

13 ```

14

15 ## Quantizing HuggingFace Models

@@ -20,7 +22,12 @@ from torchao.quantization import Int8WeightOnlyConfig

22

23 model_name = "meta-llama/Meta-Llama-3-8B"

24 quantization_config = TorchAoConfig(Int8WeightOnlyConfig())

23 -quantized_model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto", quantization_config=quantization_config)

25 +quantized_model = AutoModelForCausalLM.from_pretrained(

26 + model_name,

27 + torch_dtype="auto",

28 + device_map="auto",

29 + quantization_config=quantization_config

30 +)

31 tokenizer = AutoTokenizer.from_pretrained(model_name)

32 input_text = "What are we having for dinner?"

33 input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

docs/features/reasoning_outputs.md

+2 -1

@@ -27,7 +27,8 @@ vLLM currently supports the following reasoning models:

27 To use reasoning models, you need to specify the `--reasoning-parser` flags when making a request to the chat completion endpoint. The `--reasoning-parser` flag specifies the reasoning parser to use for extracting reasoning content from the model output.

28

29 ```bash

30 -vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --reasoning-parser deepseek_r1

30 +vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \

31 + --reasoning-parser deepseek_r1

32 ```

33

34 Next, make a request to the model that should return the reasoning content in the response.

docs/features/spec_decode.md

+7 -2

@@ -45,8 +45,13 @@ for output in outputs:

45 To perform the same with an online mode launch the server:

46

47 ```bash

48 -python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8000 --model facebook/opt-6.7b \

49 - --seed 42 -tp 1 --gpu_memory_utilization 0.8 \

48 +python -m vllm.entrypoints.openai.api_server \

49 + --host 0.0.0.0 \

50 + --port 8000 \

51 + --model facebook/opt-6.7b \

52 + --seed 42 \

53 + -tp 1 \

54 + --gpu_memory_utilization 0.8 \

55 --speculative_config '{"model": "facebook/opt-125m", "num_speculative_tokens": 5}'

56 ```

57

docs/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md

+17 -2

@@ -45,7 +45,15 @@ Use the following commands to run a Docker image:

45

46 ```console

47 docker pull vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest

48 -docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest

48 +docker run \

49 + -it \

50 + --runtime=habana \

51 + -e HABANA_VISIBLE_DEVICES=all \

52 + -e OMPI_MCA_btl_vader_single_copy_mechanism=none \

53 + --cap-add=sys_nice \

54 + --net=host \

55 + --ipc=host \

56 + vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest

57 ```

58

59 # --8<-- [end:requirements]

@@ -91,7 +99,14 @@ Currently, there are no pre-built Intel Gaudi images.

99

100 ```console

101 docker build -f docker/Dockerfile.hpu -t vllm-hpu-env .

94 -docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --rm vllm-hpu-env

102 +docker run \

103 + -it \

104 + --runtime=habana \

105 + -e HABANA_VISIBLE_DEVICES=all \

106 + -e OMPI_MCA_btl_vader_single_copy_mechanism=none \

107 + --cap-add=sys_nice \

108 + --net=host \

109 + --rm vllm-hpu-env

110 ```

111

112 !!! tip

docs/getting_started/installation/ai_accelerator/neuron.inc.md

+9 -3

@@ -38,7 +38,8 @@ The installation of drivers and tools wouldn't be necessary, if [Deep Learning A

38 sudo tee /etc/apt/sources.list.d/neuron.list > /dev/null <<EOF

39 deb https://apt.repos.neuron.amazonaws.com ${VERSION_CODENAME} main

40 EOF

41 -wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | sudo apt-key add -

41 +wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB \

42 + | sudo apt-key add -

43

44 # Update OS packages

45 sudo apt-get update -y

@@ -96,12 +97,17 @@ source aws_neuron_venv_pytorch/bin/activate

97

98 # Install Jupyter notebook kernel

99 pip install ipykernel

99 -python3.10 -m ipykernel install --user --name aws_neuron_venv_pytorch --display-name "Python (torch-neuronx)"

100 +python3.10 -m ipykernel install \

101 + --user \

102 + --name aws_neuron_venv_pytorch \

103 + --display-name "Python (torch-neuronx)"

104 pip install jupyter notebook

105 pip install environment_kernels

106

107 # Set pip repository pointing to the Neuron repository

104 -python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

108 +python -m pip config set \

109 + global.extra-index-url \

110 + https://pip.repos.neuron.amazonaws.com

111

112 # Install wget, awscli

113 python -m pip install wget

docs/getting_started/installation/gpu/cuda.inc.md

+14 -4

@@ -55,7 +55,9 @@ LLM inference is a fast-evolving field, and the latest code may contain bug fixe

55 ##### Install the latest code using `pip`

56

57 ```console

58 -pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

58 +pip install -U vllm \

59 + --pre \

60 + --extra-index-url https://wheels.vllm.ai/nightly

61 ```

62

63 `--pre` is required for `pip` to consider pre-released versions.

@@ -63,7 +65,9 @@ pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

65 Another way to install the latest code is to use `uv`:

66

67 ```console

66 -uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly

68 +uv pip install -U vllm \

69 + --torch-backend=auto \

70 + --extra-index-url https://wheels.vllm.ai/nightly

71 ```

72

73 ##### Install specific revisions using `pip`

@@ -83,7 +87,9 @@ If you want to access the wheels for previous commits (e.g. to bisect the behavi

87

88 ```console

89 export VLLM_COMMIT=72d9c316d3f6ede485146fe5aabd4e61dbc59069 # use full commit hash from the main branch

86 -uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}

90 +uv pip install vllm \

91 + --torch-backend=auto \

92 + --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}

93 ```

94

95 The `uv` approach works for vLLM `v0.6.6` and later and offers an easy-to-remember command. A unique feature of `uv` is that packages in `--extra-index-url` have [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). If the latest public release is `v0.6.6.post1`, `uv`'s behavior allows installing a commit before `v0.6.6.post1` by specifying the `--extra-index-url`. In contrast, `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version.

@@ -192,7 +198,11 @@ Additionally, if you have trouble building vLLM, we recommend using the NVIDIA P

198

199 ```console

200 # Use `--ipc=host` to make sure the shared memory is large enough.

195 -docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3

201 +docker run \

202 + --gpus all \

203 + -it \

204 + --rm \

205 + --ipc=host nvcr.io/nvidia/pytorch:23.10-py3

206 ```

207

208 If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from [the official website](https://developer.nvidia.com/cuda-toolkit-archive). After installation, set the environment variable `CUDA_HOME` to the installation path of CUDA Toolkit, and make sure that the `nvcc` compiler is in your `PATH`, e.g.:

docs/getting_started/installation/gpu/rocm.inc.md

+18 -9

@@ -91,19 +91,22 @@ Currently, there are no pre-built ROCm wheels.

91 4. Build vLLM. For example, vLLM on ROCM 6.3 can be built with the following steps:

92

93 ```bash

94 - $ pip install --upgrade pip

94 + pip install --upgrade pip

95

96 # Build & install AMD SMI

97 - $ pip install /opt/rocm/share/amd_smi

97 + pip install /opt/rocm/share/amd_smi

98

99 # Install dependencies

100 - $ pip install --upgrade numba scipy huggingface-hub[cli,hf_transfer] setuptools_scm

101 - $ pip install "numpy<2"

102 - $ pip install -r requirements/rocm.txt

100 + pip install --upgrade numba \

101 + scipy \

102 + huggingface-hub[cli,hf_transfer] \

103 + setuptools_scm

104 + pip install "numpy<2"

105 + pip install -r requirements/rocm.txt

106

107 # Build vLLM for MI210/MI250/MI300.

105 - $ export PYTORCH_ROCM_ARCH="gfx90a;gfx942"

106 - $ python3 setup.py develop

108 + export PYTORCH_ROCM_ARCH="gfx90a;gfx942"

109 + python3 setup.py develop

110 ```

111

112 This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm installation.

@@ -154,7 +157,9 @@ It is important that the user kicks off the docker build using buildkit. Either

157 To build vllm on ROCm 6.3 for MI200 and MI300 series, you can use the default:

158

159 ```console

157 -DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm_base -t rocm/vllm-dev:base .

160 +DOCKER_BUILDKIT=1 docker build \

161 + -f docker/Dockerfile.rocm_base \

162 + -t rocm/vllm-dev:base .

163 ```

164

165 #### Build an image with vLLM

@@ -189,7 +194,11 @@ DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-rocm .

194 To build vllm on ROCm 6.3 for Radeon RX7900 series (gfx1100), you should pick the alternative base image:

195

196 ```console

192 -DOCKER_BUILDKIT=1 docker build --build-arg BASE_IMAGE="rocm/vllm-dev:navi_base" -f docker/Dockerfile.rocm -t vllm-rocm .

197 +DOCKER_BUILDKIT=1 docker build \

198 + --build-arg BASE_IMAGE="rocm/vllm-dev:navi_base" \

199 + -f docker/Dockerfile.rocm \

200 + -t vllm-rocm \

201 + .

202 ```

203

204 To run the above docker image `vllm-rocm`, use the below command:

docs/models/extensions/runai_model_streamer.md

+21 -7

@@ -16,19 +16,25 @@ pip3 install vllm[runai]

16 To run it as an OpenAI-compatible server, add the `--load-format runai_streamer` flag:

17

18 ```console

19 -vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer

19 +vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \

20 + --load-format runai_streamer

21 ```

22

23 To run model from AWS S3 object store run:

24

25 ```console

25 -vllm serve s3://core-llm/Llama-3-8b --load-format runai_streamer

26 +vllm serve s3://core-llm/Llama-3-8b \

27 + --load-format runai_streamer

28 ```

29

30 To run model from a S3 compatible object store run:

31

32 ```console

31 -RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING=0 AWS_EC2_METADATA_DISABLED=true AWS_ENDPOINT_URL=https://storage.googleapis.com vllm serve s3://core-llm/Llama-3-8b --load-format runai_streamer

33 +RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING=0 \

34 +AWS_EC2_METADATA_DISABLED=true \

35 +AWS_ENDPOINT_URL=https://storage.googleapis.com \

36 +vllm serve s3://core-llm/Llama-3-8b \

37 + --load-format runai_streamer

38 ```

39

40 ## Tunable parameters

@@ -39,14 +45,18 @@ You can tune `concurrency` that controls the level of concurrency and number of

45 For reading from S3, it will be the number of client instances the host is opening to the S3 server.

46

47 ```console

42 -vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer --model-loader-extra-config '{"concurrency":16}'

48 +vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \

49 + --load-format runai_streamer \

50 + --model-loader-extra-config '{"concurrency":16}'

51 ```

52

53 You can control the size of the CPU Memory buffer to which tensors are read from the file, and limit this size.

54 You can read further about CPU buffer memory limiting [here](https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/env-vars.md#runai_streamer_memory_limit).

55

56 ```console

49 -vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer --model-loader-extra-config '{"memory_limit":5368709120}'

57 +vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \

58 + --load-format runai_streamer \

59 + --model-loader-extra-config '{"memory_limit":5368709120}'

60 ```

61

62 !!! note

@@ -63,7 +73,9 @@ vllm serve /path/to/sharded/model --load-format runai_streamer_sharded

73 The sharded loader expects model files to follow the same naming pattern as the regular sharded state loader: `model-rank-{rank}-part-{part}.safetensors`. You can customize this pattern using the `pattern` parameter in `--model-loader-extra-config`:

74

75 ```console

66 -vllm serve /path/to/sharded/model --load-format runai_streamer_sharded --model-loader-extra-config '{"pattern":"custom-model-rank-{rank}-part-{part}.safetensors"}'

76 +vllm serve /path/to/sharded/model \

77 + --load-format runai_streamer_sharded \

78 + --model-loader-extra-config '{"pattern":"custom-model-rank-{rank}-part-{part}.safetensors"}'

79 ```

80

81 To create sharded model files, you can use the script provided in <gh-file:examples/offline_inference/save_sharded_state.py>. This script demonstrates how to save a model in the sharded format that is compatible with the Run:ai Model Streamer sharded loader.

@@ -71,7 +83,9 @@ To create sharded model files, you can use the script provided in <gh-file:examp

83 The sharded loader supports all the same tunable parameters as the regular Run:ai Model Streamer, including `concurrency` and `memory_limit`. These can be configured in the same way:

84

85 ```console

74 -vllm serve /path/to/sharded/model --load-format runai_streamer_sharded --model-loader-extra-config '{"concurrency":16, "memory_limit":5368709120}'

86 +vllm serve /path/to/sharded/model \

87 + --load-format runai_streamer_sharded \

88 + --model-loader-extra-config '{"concurrency":16, "memory_limit":5368709120}'

89 ```

90

91 !!! note

docs/serving/openai_compatible_server.md

+9 -3

@@ -8,7 +8,9 @@ vLLM provides an HTTP server that implements OpenAI's [Completions API](https://

8 In your terminal, you can [install](../getting_started/installation/README.md) vLLM, then start the server with the [`vllm serve`][serve-args] command. (You can also use our [Docker][deployment-docker] image.)

9

10 ```bash

11 -vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123

11 +vllm serve NousResearch/Meta-Llama-3-8B-Instruct \

12 + --dtype auto \

13 + --api-key token-abc123

14 ```

15

16 To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).

@@ -243,7 +245,9 @@ and passing a list of `messages` in the request. Refer to the examples below for

245

246 ```bash

247 vllm serve TIGER-Lab/VLM2Vec-Full --task embed \

246 - --trust-remote-code --max-model-len 4096 --chat-template examples/template_vlm2vec.jinja

248 + --trust-remote-code \

249 + --max-model-len 4096 \

250 + --chat-template examples/template_vlm2vec.jinja

251 ```

252

253 !!! warning

@@ -285,7 +289,9 @@ and passing a list of `messages` in the request. Refer to the examples below for

289

290 ```bash

291 vllm serve MrLight/dse-qwen2-2b-mrl-v1 --task embed \

288 - --trust-remote-code --max-model-len 8192 --chat-template examples/template_dse_qwen2_vl.jinja

292 + --trust-remote-code \

293 + --max-model-len 8192 \

294 + --chat-template examples/template_dse_qwen2_vl.jinja

295 ```

296

297 !!! warning

[doc] improve readability (#18675)