3fb17d26c8 · tduncan/Nvidia-Optimization-Stack-VLLM

docs/source/deployment/frameworks/helm.md

+2 -2

@@ -4,9 +4,9 @@

4

5 A Helm chart to deploy vLLM for Kubernetes

6

7 -Helm is a package manager for Kubernetes. It will help you to deploy vLLM on k8s and automate the deployment of vLLMm Kubernetes applications. With Helm, you can deploy the same framework architecture with different configurations to multiple namespaces by overriding variables values.

7 +Helm is a package manager for Kubernetes. It will help you to deploy vLLM on k8s and automate the deployment of vLLM Kubernetes applications. With Helm, you can deploy the same framework architecture with different configurations to multiple namespaces by overriding variable values.

8

9 -This guide will walk you through the process of deploying vLLM with Helm, including the necessary prerequisites, steps for helm install and documentation on architecture and values file.

9 +This guide will walk you through the process of deploying vLLM with Helm, including the necessary prerequisites, steps for helm installation and documentation on architecture and values file.

10

11 ## Prerequisites

12

docs/source/deployment/k8s.md

+1 -1

@@ -14,7 +14,7 @@ Alternatively, you can also deploy Kubernetes using [helm chart](https://docs.vl

14

15 ## Pre-requisite

16

17 -Ensure that you have a running Kubernetes environment with GPU (you can follow [this tutorial](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) to install a Kubernetes environment on a bare-medal GPU machine).

17 +Ensure that you have a running Kubernetes environment with GPU (you can follow [this tutorial](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) to install a Kubernetes environment on a bare-metal GPU machine).

18

19 ## Deployment using native K8s

20

docs/source/design/kernel/paged_attention.md

+1 -1

@@ -419,7 +419,7 @@ List of `v_vec` for one thread

419 which is also `V_VEC_SIZE` elements from `logits`. Overall, with

420 multiple inner iterations, each warp will process one block of value

421 tokens. And with multiple outer iterations, the whole context value

422 - tokens are processd

422 + tokens are processed

423

424 ```cpp

425 float accs[NUM_ROWS_PER_THREAD];

docs/source/design/v1/metrics.md

+2 -2

@@ -13,7 +13,7 @@ Ensure the v1 LLM Engine exposes a superset of the metrics available in v0.

13 Metrics in vLLM can be categorized as follows:

14

15 1. Server-level metrics: these are global metrics that track the state and performance of the LLM engine. These are typically exposed as Gauges or Counters in Prometheus.

16 -2. Request-level metrics: these are metrics that track the characteristics - e.g. size and timing - of individual requests. These are typically exposed as Histrograms in Prometheus, and are often the SLO that an SRE monitoring vLLM will be tracking.

16 +2. Request-level metrics: these are metrics that track the characteristics - e.g. size and timing - of individual requests. These are typically exposed as Histograms in Prometheus, and are often the SLO that an SRE monitoring vLLM will be tracking.

17

18 The mental model is that the "Server-level Metrics" explain why the "Request-level Metrics" are what they are.

19

@@ -47,7 +47,7 @@ In v0, the following metrics are exposed via a Prometheus-compatible `/metrics`

47 - `vllm:tokens_total` (Counter)

48 - `vllm:iteration_tokens_total` (Histogram)

49 - `vllm:time_in_queue_requests` (Histogram)

50 -- `vllm:model_forward_time_milliseconds` (Histogram

50 +- `vllm:model_forward_time_milliseconds` (Histogram)

51 - `vllm:model_execute_time_milliseconds` (Histogram)

52 - `vllm:request_params_n` (Histogram)

53 - `vllm:request_params_max_tokens` (Histogram)

docs/source/features/lora.md

+1 -1

@@ -110,7 +110,7 @@ In addition to serving LoRA adapters at server startup, the vLLM server now supp

110 LoRA adapters at runtime through dedicated API endpoints. This feature can be particularly useful when the flexibility

111 to change models on-the-fly is needed.

112

113 -Note: Enabling this feature in production environments is risky as user may participate model adapter management.

113 +Note: Enabling this feature in production environments is risky as users may participate in model adapter management.

114

115 To enable dynamic LoRA loading and unloading, ensure that the environment variable `VLLM_ALLOW_RUNTIME_LORA_UPDATING`

116 is set to `True`. When this option is enabled, the API server will log a warning to indicate that dynamic loading is active.

docs/source/getting_started/faq.md

+1 -1

@@ -15,7 +15,7 @@ more are listed [here](#supported-models).

15

16 By extracting hidden states, vLLM can automatically convert text generation models like [Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B),

17 [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) into embedding models,

18 -but they are expected be inferior to models that are specifically trained on embedding tasks.

18 +but they are expected to be inferior to models that are specifically trained on embedding tasks.

19

20 ______________________________________________________________________

21

docs/source/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md

+1 -1

@@ -119,7 +119,7 @@ If you're observing the following error: `docker: Error response from daemon: Un

119

120 ## Supported configurations

121

122 -The following configurations have been validated to be function with

122 +The following configurations have been validated to function with

123 Gaudi2 devices. Configurations that are not listed may or may not work.

124

125 - [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)

docs/source/getting_started/installation/ai_accelerator/openvino.inc.md

+1 -1

@@ -19,7 +19,7 @@ Currently, there are no pre-built OpenVINO wheels.

19

20 ### Build wheel from source

21

22 -First, install Python and ensure you lave the latest pip. For example, on Ubuntu 22.04, you can run:

22 +First, install Python and ensure you have the latest pip. For example, on Ubuntu 22.04, you can run:

23

24 ```console

25 sudo apt-get update -y

docs/source/getting_started/installation/gpu/xpu.inc.md

+4 -4

@@ -1,6 +1,6 @@

1 # Installation

2

3 -vLLM initially supports basic model inferencing and serving on Intel GPU platform.

3 +vLLM initially supports basic model inference and serving on Intel GPU platform.

4

5 :::{attention}

6 There are no pre-built wheels or images for this device, so you must build vLLM from source.

@@ -65,7 +65,7 @@ $ docker run -it \

65

66 ## Supported features

67

68 -XPU platform supports **tensor parallel** inference/serving and also supports **pipeline parallel** as a beta feature for online serving. We requires Ray as the distributed runtime backend. For example, a reference execution likes following:

68 +XPU platform supports **tensor parallel** inference/serving and also supports **pipeline parallel** as a beta feature for online serving. We require Ray as the distributed runtime backend. For example, a reference execution like following:

69

70 ```console

71 python -m vllm.entrypoints.openai.api_server \

@@ -78,6 +78,6 @@ python -m vllm.entrypoints.openai.api_server \

78 -tp=8

79 ```

80

81 -By default, a ray instance will be launched automatically if no existing one is detected in system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring to the <gh-file:examples/online_serving/run_cluster.sh> helper script.

81 +By default, a ray instance will be launched automatically if no existing one is detected in the system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring to the <gh-file:examples/online_serving/run_cluster.sh> helper script.

82

83 -There are some new features coming with ipex-xpu 2.6, eg: **chunked prefill**, **V1 engine support**, **lora**, **MoE**, etc.

83 +There are some new features coming with ipex-xpu 2.6, e.g. **chunked prefill**, **V1 engine support**, **lora**, **MoE**, etc.

docs/source/serving/distributed_serving.md

+2 -2

@@ -20,7 +20,7 @@ There is one edge case: if the model fits in a single node with multiple GPUs, b

20

21 ## Running vLLM on a single node

22

23 -vLLM supports distributed tensor-parallel and pipeline-parallel inference and serving. Currently, we support [Megatron-LM's tensor parallel algorithm](https://arxiv.org/pdf/1909.08053.pdf). We manage the distributed runtime with either [Ray](https://github.com/ray-project/ray) or python native multiprocessing. Multiprocessing can be used when deploying on a single node, multi-node inferencing currently requires Ray.

23 +vLLM supports distributed tensor-parallel and pipeline-parallel inference and serving. Currently, we support [Megatron-LM's tensor parallel algorithm](https://arxiv.org/pdf/1909.08053.pdf). We manage the distributed runtime with either [Ray](https://github.com/ray-project/ray) or python native multiprocessing. Multiprocessing can be used when deploying on a single node, multi-node inference currently requires Ray.

24

25 Multiprocessing will be used by default when not running in a Ray placement group and if there are sufficient GPUs available on the same node for the configured `tensor_parallel_size`, otherwise Ray will be used. This default can be overridden via the `LLM` class `distributed_executor_backend` argument or `--distributed-executor-backend` API server argument. Set it to `mp` for multiprocessing or `ray` for Ray. It's not required for Ray to be installed for the multiprocessing case.

26

@@ -29,7 +29,7 @@ To run multi-GPU inference with the `LLM` class, set the `tensor_parallel_size`

29 ```python

30 from vllm import LLM

31 llm = LLM("facebook/opt-13b", tensor_parallel_size=4)

32 -output = llm.generate("San Franciso is a")

32 +output = llm.generate("San Francisco is a")

33 ```

34

35 To run multi-GPU serving, pass in the `--tensor-parallel-size` argument when starting the server. For example, to run API server on 4 GPUs:

docs/source/training/rlhf.md

+1 -1

@@ -1,6 +1,6 @@

1 # Reinforcement Learning from Human Feedback

2

3 -Reinforcement Learning from Human Feedback (RLHF) is a technique that fine-tunes language models using human-generated preference data to align model outputs with desired behaviours.

3 +Reinforcement Learning from Human Feedback (RLHF) is a technique that fine-tunes language models using human-generated preference data to align model outputs with desired behaviors.

4

5 vLLM can be used to generate the completions for RLHF. The best way to do this is with libraries like [TRL](https://github.com/huggingface/trl), [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) and [verl](https://github.com/volcengine/verl).

6

examples/other/logging_configuration.md

+1 -1

@@ -127,7 +127,7 @@ configuration for the root vLLM logger and for the logger you wish to silence:

127 "vllm": {

128 "handlers": ["vllm"],

129 "level": "DEBUG",

130 - "propagage": false

130 + "propagate": false

131 },

132 "vllm.example_noisy_logger": {

133 "propagate": false

vllm/distributed/kv_transfer/README.md

+1 -1

@@ -24,6 +24,6 @@ NOTE: If you want to not only transfer KV caches, but adjust the model execution

24

25 The example usage is in [this file](../../../examples/online_serving/disaggregated_prefill.sh).

26

27 -Here is the diagram of how we run disaggretgated prefilling.

27 +Here is the diagram of how we run disaggregated prefilling.

28

29 ![Disaggregated prefill workflow](./disagg_prefill_workflow.jpg)

[Doc] Fix typo in documentation (#14783)