d99c3a4f7b · tduncan/Nvidia-Optimization-Stack-VLLM

docs/contributing/ci/update_pytorch_version.md

+1 -1

@@ -90,7 +90,7 @@ address the long build time at its source, the current workaround is to set `VLL

90 to a custom branch provided by @khluu (`VLLM_CI_BRANCH=khluu/use_postmerge_q`)

91 when manually triggering a build on Buildkite. This branch accomplishes two things:

92

93 -1. Increase the timeout limit to 10 hours so that the build doesn't timeout.

93 +1. Increase the timeout limit to 10 hours so that the build doesn't time out.

94 2. Allow the compiled artifacts to be written to the vLLM sccache S3 bucket

95 to warm it up so that future builds are faster.

96

docs/contributing/model/multimodal.md

+1 -1

@@ -855,7 +855,7 @@ Examples:

855

856 ### Custom HF processor

857

858 -Some models don't define a HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and pass it to [_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor].

858 +Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and pass it to [_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor].

859

860 Examples:

861

docs/deployment/frameworks/lobe-chat.md

+1 -1

@@ -6,6 +6,6 @@ Supports speech-synthesis, multi-modal, and extensible (function call) plugin sy

6

7 One-click FREE deployment of your private OpenAI ChatGPT/Claude/Gemini/Groq/Ollama chat application.

8

9 -It supports vLLM as a AI model provider to efficiently serve large language models.

9 +It supports vLLM as an AI model provider to efficiently serve large language models.

10

11 For details, see the tutorial [Using vLLM in LobeChat](https://lobehub.com/docs/usage/providers/vllm).

docs/deployment/k8s.md

+1 -1

@@ -380,7 +380,7 @@ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

380

381 ### Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"

382

383 -If the startup or readiness probe failureThreshold is too low for the time needed to startup the server, Kubernetes scheduler will kill the container. A couple of indications that this has happened:

383 +If the startup or readiness probe failureThreshold is too low for the time needed to start up the server, Kubernetes scheduler will kill the container. A couple of indications that this has happened:

384

385 1. container log contains "KeyboardInterrupt: terminated"

386 2. `kubectl get events` shows message `Container $NAME failed startup probe, will be restarted`

docs/design/fused_moe_modular_kernel.md

+1 -1

@@ -138,7 +138,7 @@ Typically a FusedMoEPrepareAndFinalize type is backed by an All2All Dispatch & C

138

139 #### Step 1: Add an All2All manager

140

141 -The purpose of the All2All Manager is to setup the All2All kernel implementations. The `FusedMoEPrepareAndFinalize` implementations typically fetch a kernel-implementation "handle" from the All2All Manager to invoke the Dispatch and Combine functions. Please look at the All2All Manager implementations [here](gh-file:vllm/distributed/device_communicators/all2all.py).

141 +The purpose of the All2All Manager is to set up the All2All kernel implementations. The `FusedMoEPrepareAndFinalize` implementations typically fetch a kernel-implementation "handle" from the All2All Manager to invoke the Dispatch and Combine functions. Please look at the All2All Manager implementations [here](gh-file:vllm/distributed/device_communicators/all2all.py).

142

143 #### Step 2: Add a FusedMoEPrepareAndFinalize Type

144

docs/design/metrics.md

+2 -2

@@ -99,11 +99,11 @@ http_request_duration_seconds_count{handler="/v1/completions",method="POST"} 201

99

100 ### Multi-process Mode

101

102 -In v0, metrics are collected in the engine core process and we use multi-process mode to make them available in the API server process. See <gh-pr:7279>.

102 +In v0, metrics are collected in the engine core process and we use multiprocess mode to make them available in the API server process. See <gh-pr:7279>.

103

104 ### Built in Python/Process Metrics

105

106 -The following metrics are supported by default by `prometheus_client`, but they are not exposed when multi-process mode is used:

106 +The following metrics are supported by default by `prometheus_client`, but they are not exposed when multiprocess mode is used:

107

108 - `python_gc_objects_collected_total`

109 - `python_gc_objects_uncollectable_total`

docs/features/lora.md

+1 -1

@@ -52,7 +52,7 @@ Check out <gh-file:examples/offline_inference/multilora_inference.py> for an exa

52 ## Serving LoRA Adapters

53

54 LoRA adapted models can also be served with the Open-AI compatible vLLM server. To do so, we use

55 -`--lora-modules {name}={path} {name}={path}` to specify each LoRA module when we kickoff the server:

55 +`--lora-modules {name}={path} {name}={path}` to specify each LoRA module when we kick off the server:

56

57 ```bash

58 vllm serve meta-llama/Llama-2-7b-hf \

docs/features/reasoning_outputs.md

+1 -1

@@ -143,7 +143,7 @@ OpenAI Python client library does not officially support `reasoning_content` att

143 print(content, end="", flush=True)

144 ```

145

146 -Remember to check whether the `reasoning_content` exists in the response before accessing it. You could checkout the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).

146 +Remember to check whether the `reasoning_content` exists in the response before accessing it. You could check out the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).

147

148 ## Tool Calling

149

docs/features/structured_outputs.md

+1 -1

@@ -205,7 +205,7 @@ This section covers the OpenAI beta wrapper over the `client.chat.completions.cr

205

206 At the time of writing (`openai==1.54.4`), this is a "beta" feature in the OpenAI client library. Code reference can be found [here](https://github.com/openai/openai-python/blob/52357cff50bee57ef442e94d78a0de38b4173fc2/src/openai/resources/beta/chat/completions.py#L100-L104).

207

208 -For the following examples, vLLM was setup using `vllm serve meta-llama/Llama-3.1-8B-Instruct`

208 +For the following examples, vLLM was set up using `vllm serve meta-llama/Llama-3.1-8B-Instruct`

209

210 Here is a simple example demonstrating how to get structured output using Pydantic models:

211

docs/getting_started/installation/aws_neuron.md

+2 -2

@@ -140,8 +140,8 @@ Alternatively, users can directly call the NxDI library to trace and compile you

140

141 - `NEURON_COMPILED_ARTIFACTS`: set this environment variable to point to your pre-compiled model artifacts directory to avoid

142 compilation time upon server initialization. If this variable is not set, the Neuron module will perform compilation and save the

143 - artifacts under `neuron-compiled-artifacts/{unique_hash}/` sub-directory in the model path. If this environment variable is set,

144 - but the directory does not exist, or the contents are invalid, Neuron will also fallback to a new compilation and store the artifacts

143 + artifacts under `neuron-compiled-artifacts/{unique_hash}/` subdirectory in the model path. If this environment variable is set,

144 + but the directory does not exist, or the contents are invalid, Neuron will also fall back to a new compilation and store the artifacts

145 under this specified path.

146 - `NEURON_CONTEXT_LENGTH_BUCKETS`: Bucket sizes for context encoding. (Only applicable to `transformers-neuronx` backend).

147 - `NEURON_TOKEN_GEN_BUCKETS`: Bucket sizes for token generation. (Only applicable to `transformers-neuronx` backend).

docs/getting_started/installation/cpu/apple.inc.md

+1 -1

@@ -1,6 +1,6 @@

1 # --8<-- [start:installation]

2

3 -vLLM has experimental support for macOS with Apple silicon. For now, users must build from source to natively run on macOS.

3 +vLLM has experimental support for macOS with Apple Silicon. For now, users must build from source to natively run on macOS.

4

5 Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.

6

docs/getting_started/installation/gpu/cuda.inc.md

+1 -1

@@ -48,7 +48,7 @@ uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VE

48

49 #### Install the latest code

50

51 -LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for Linux running on a x86 platform with CUDA 12 for every commit since `v0.5.3`.

51 +LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for Linux running on an x86 platform with CUDA 12 for every commit since `v0.5.3`.

52

53 ```bash

54 uv pip install -U vllm \

docs/getting_started/installation/gpu/rocm.inc.md

+2 -2

@@ -149,7 +149,7 @@ Build a docker image from <gh-file:docker/Dockerfile.rocm_base> which setup ROCm

149 **This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**

150 If you choose to build this rocm_base image yourself, the steps are as follows.

151

152 -It is important that the user kicks off the docker build using buildkit. Either the user put DOCKER_BUILDKIT=1 as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:

152 +It is important that the user kicks off the docker build using buildkit. Either the user put DOCKER_BUILDKIT=1 as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:

153

154 ```json

155 {

@@ -170,7 +170,7 @@ DOCKER_BUILDKIT=1 docker build \

170 #### Build an image with vLLM

171

172 First, build a docker image from <gh-file:docker/Dockerfile.rocm> and launch a docker container from the image.

173 -It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:

173 +It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:

174

175 ```bash

176 {

docs/models/pooling_models.md

+1 -1

@@ -258,4 +258,4 @@ Expected output:

258 {"id":"embd-5c21fc9a5c9d4384a1b021daccaf9f64","object":"list","created":1745476417,"model":"jinaai/jina-embeddings-v3","data":[{"index":0,"object":"embedding","embedding":[-0.3828125,-0.1357421875,0.03759765625,0.125,0.21875,0.09521484375,-0.003662109375,0.1591796875,-0.130859375,-0.0869140625,-0.1982421875,0.1689453125,-0.220703125,0.1728515625,-0.2275390625,-0.0712890625,-0.162109375,-0.283203125,-0.055419921875,-0.0693359375,0.031982421875,-0.04052734375,-0.2734375,0.1826171875,-0.091796875,0.220703125,0.37890625,-0.0888671875,-0.12890625,-0.021484375,-0.0091552734375,0.23046875]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0,"prompt_tokens_details":null}}

259 ```

260

261 -A openai client example can be found here: <gh-file:examples/online_serving/openai_embedding_matryoshka_fy.py>

261 +An OpenAI client example can be found here: <gh-file:examples/online_serving/openai_embedding_matryoshka_fy.py>

docs/models/supported_models.md

+1 -1

@@ -40,7 +40,7 @@ If it is `TransformersForCausalLM` or `TransformersForMultimodalLM` then it mean

40

41 #### Custom models

42

43 -If a model is neither supported natively by vLLM or Transformers, it can still be used in vLLM!

43 +If a model is neither supported natively by vLLM nor Transformers, it can still be used in vLLM!

44

45 For a model to be compatible with the Transformers backend for vLLM it must:

46

docs/usage/usage_stats.md

+1 -1

@@ -51,7 +51,7 @@ tail ~/.config/vllm/usage_stats.json

51

52 ## Opting out

53

54 -You can opt-out of usage stats collection by setting the `VLLM_NO_USAGE_STATS` or `DO_NOT_TRACK` environment variable, or by creating a `~/.config/vllm/do_not_track` file:

54 +You can opt out of usage stats collection by setting the `VLLM_NO_USAGE_STATS` or `DO_NOT_TRACK` environment variable, or by creating a `~/.config/vllm/do_not_track` file:

55

56 ```bash

57 # Any of the following methods can disable usage stats collection

[Doc]: fix typos in .md files (including those of #23751) (#23825)