aab0102a26 · tduncan/Nvidia-Optimization-Stack-VLLM

docs/contributing/model/basic.md

-2

@@ -133,8 +133,6 @@ We consider 3 different scenarios:

133 For case (1), we recommend looking at the implementation of [`MambaForCausalLM`](../../../vllm/model_executor/models/mamba.py) (for Mamba-1) or [`Mamba2ForCausalLM`](../../../vllm/model_executor/models/mamba2.py) (for Mamba-2) as a reference.

134 The model should inherit protocol `IsAttentionFree` and also implement class methods `get_mamba_state_dtype_from_config` and `get_mamba_state_shape_from_config` to calculate the state shapes and data types from the config.

135 For the mamba layers themselves, please use the [`MambaMixer`](../../../vllm/model_executor/layers/mamba/mamba_mixer.py) (for Mamba-1) or [`MambaMixer2`](../../../vllm/model_executor/layers/mamba/mamba_mixer2.py) (for Mamba-2) classes.

136 -Please *do not* use the `MambaCacheManager` (deprecated in V1) or replicate any of the V0-specific code paths in the existing model implementations.

137 -V0-only classes and code will be removed in the very near future.

136 The model should also be added to the `MODELS_CONFIG_MAP` dictionary in [vllm/model_executor/models/config.py](../../../vllm/model_executor/models/config.py) to ensure that the runtime defaults are optimized.

137

138 For case (2), we recommend using as a reference the implementation of [`JambaForCausalLM`](../../../vllm/model_executor/models/jamba.py) (for an example of a model that uses Mamba-1 and attention together) or [`BambaForCausalLM`](../../../vllm/model_executor/models/bamba.py) (for an example of a model that uses Mamba-2 and attention together).

docs/design/prefix_caching.md

-3

@@ -94,9 +94,6 @@ To improve privacy in shared environments, vLLM supports isolating prefix cache

94

95 With this setup, cache sharing is limited to users or requests that explicitly agree on a common salt, enabling cache reuse within a trust group while isolating others.

96

97 -!!! note

98 - Cache isolation is not supported in engine V0.

99 -

97 ## Data Structure

98

99 The prefix caching in vLLM v1 is implemented in the KV cache manager. The basic building block is the “Block” data class (simplified):

docs/usage/reproducibility.md

+2 -7

@@ -1,10 +1,7 @@

1 # Reproducibility

2

3 -vLLM does not guarantee the reproducibility of the results by default, for the sake of performance. You need to do the following to achieve

4 -reproducible results:

5 -

6 -- For V1: Turn off multiprocessing to make the scheduling deterministic by setting `VLLM_ENABLE_V1_MULTIPROCESSING=0`.

7 -- For V0: Set the global seed (see below).

3 +vLLM does not guarantee the reproducibility of the results by default, for the sake of performance. To achieve

4 +reproducible results, you need to turn off multiprocessing to make the scheduling deterministic by setting `VLLM_ENABLE_V1_MULTIPROCESSING=0`.

5

6 Example: [examples/offline_inference/reproducibility.py](../../examples/offline_inference/reproducibility.py)

7

@@ -30,8 +27,6 @@ However, in some cases, setting the seed will also [change the random state in u

27

28 ### Default Behavior

29

33 -In V0, the `seed` parameter defaults to `None`. When the `seed` parameter is `None`, the random states for `random`, `np.random`, and `torch.manual_seed` are not set. This means that each run of vLLM will produce different results if `temperature > 0`, as expected.

34 -

30 In V1, the `seed` parameter defaults to `0` which sets the random state for each worker, so the results will remain consistent for each vLLM run even if `temperature > 0`.

31

32 !!! note

docs/usage/v1_guide.md

+1 -1

@@ -2,7 +2,7 @@

2

3 !!! announcement

4

5 - We have started the process of deprecating V0. Please read [RFC #18571](https://github.com/vllm-project/vllm/issues/18571) for more details.

5 + We have fully deprecated V0. Please read [RFC #18571](https://github.com/vllm-project/vllm/issues/18571) for more details.

6

7 V1 is now enabled by default for all supported use cases, and we will gradually enable it for every use case we plan to support. Please share any feedback on [GitHub](https://github.com/vllm-project/vllm) or in the [vLLM Slack](https://inviter.co/vllm-slack).

8

examples/offline_inference/reproducibility.py

+2 -6

@@ -11,13 +11,9 @@ import random

11

12 from vllm import LLM, SamplingParams

13

14 -# V1 only: Turn off multiprocessing to make the scheduling deterministic.

14 +# Turn off multiprocessing to make the scheduling deterministic.

15 os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"

16

17 -# V0 only: Set the global seed. The default seed is None, which is

18 -# not reproducible.

19 -SEED = 42

20 -

17 prompts = [

18 "Hello, my name is",

19 "The president of the United States is",

@@ -28,7 +24,7 @@ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

24

25

26 def main():

31 - llm = LLM(model="facebook/opt-125m", seed=SEED)

27 + llm = LLM(model="facebook/opt-125m")

28 outputs = llm.generate(prompts, sampling_params)

29 print("-" * 50)

30 for output in outputs:

examples/offline_inference/rlhf_utils.py

+4 -4

@@ -30,8 +30,8 @@ class WorkerExtension:

30 """

31 The class for vLLM's worker to inherit from.

32 By defining an extension class, the code can work no matter what is

33 - the underlying worker class. This way, the code can be compatible

34 - with both vLLM V0 and V1.

33 + the underlying worker class.

34 +

35 NOTE: we define this class in a separate module, and the main module

36 should pass the full qualified name as `worker_extension_cls` argument.

37 """

@@ -96,8 +96,8 @@ class ColocateWorkerExtension:

96 """

97 The class for vLLM's worker to inherit from, in the colocate setting.

98 By defining an extension class, the code can work no matter what is

99 - the underlying worker class. This way, the code can be compatible

100 - with both vLLM V0 and V1.

99 + the underlying worker class.

100 +

101 NOTE: we define this class in a separate module, and the main module

102 should pass the full qualified name as `worker_extension_cls` argument.

103 """

examples/offline_inference/save_sharded_state.py

+3 -16

@@ -67,22 +67,9 @@ def main(args):

67 Path(args.output).mkdir(exist_ok=True)

68 # Dump worker states to output directory

69

70 - # Check which engine version is being used

71 - is_v1_engine = hasattr(llm.llm_engine, "engine_core")

72 -

73 - if is_v1_engine:

74 - # For V1 engine, we need to use engine_core.save_sharded_state

75 - print("Using V1 engine save path")

76 - llm.llm_engine.engine_core.save_sharded_state(

77 - path=args.output, pattern=args.file_pattern, max_size=args.max_file_size

78 - )

79 - else:

80 - # For V0 engine

81 - print("Using V0 engine save path")

82 - model_executor = llm.llm_engine.model_executor

83 - model_executor.save_sharded_state(

84 - path=args.output, pattern=args.file_pattern, max_size=args.max_file_size

85 - )

70 + llm.llm_engine.engine_core.save_sharded_state(

71 + path=args.output, pattern=args.file_pattern, max_size=args.max_file_size

72 + )

73

74 # Copy metadata files to output directory

75 for file in os.listdir(model_path):

examples/offline_inference/spec_decode.py

+1 -5

@@ -158,11 +158,7 @@ def main(args):

158 print(f"generated text: {output.outputs[0].text}")

159 print("-" * 50)

160

161 - try:

162 - metrics = llm.get_metrics()

163 - except AssertionError:

164 - print("Metrics are not supported in the V0 engine.")

165 - return

161 + metrics = llm.get_metrics()

162

163 total_num_output_tokens = sum(

164 len(output.outputs[0].token_ids) for output in outputs

tests/model_executor/model_loader/test_sharded_state_loader.py

+2 -11

@@ -60,18 +60,9 @@ def llama_3p2_1b_files():

60

61 def _run_writer(input_dir, output_dir, weights_patterns, **kwargs):

62 llm_sharded_writer = LLM(model=input_dir, **kwargs)

63 - # Check which engine version is being used

64 - is_v1_engine = hasattr(llm_sharded_writer.llm_engine, "engine_core")

63 +

64 # Dump worker states to output directory

66 - if is_v1_engine:

67 - # For V1 engine, we need to use engine_core.save_sharded_state

68 - print("Using V1 engine save path")

69 - llm_sharded_writer.llm_engine.engine_core.save_sharded_state(path=output_dir)

70 - else:

71 - # For V0 engine

72 - print("Using V0 engine save path")

73 - model_executor = llm_sharded_writer.llm_engine.model_executor

74 - model_executor.save_sharded_state(path=output_dir)

65 + llm_sharded_writer.llm_engine.engine_core.save_sharded_state(path=output_dir)

66

67 # Copy metadata files to output directory

68 for file in os.listdir(input_dir):

tests/tool_use/utils.py

+13 -12

@@ -140,21 +140,22 @@ CONFIGS: dict[str, ServerConfig] = {

140 "without calling a tool. DO NOT CALL A TOOL THAT IS IRRELEVANT "

141 "to the user's question - just respond to it normally.",

142 },

143 - # V1 Test: Passing locally but failing in CI. This runs the

144 - # V0 Engine because of CPU offloading. Need to debug why.

143 + # FIXME: This test currently fails, need to debug why.

144 # "granite20b": {

146 - # "model":

147 - # "mbayser/granite-20b-functioncalling-FP8-KV",

145 + # "model": "mbayser/granite-20b-functioncalling-FP8-KV",

146 # "arguments": [

149 - # "--tool-call-parser", "granite-20b-fc", "--chat-template",

150 - # str(VLLM_PATH /

151 - # "examples/tool_chat_template_granite_20b_fc.jinja"),

152 - # "--max_num_seqs", "1", "--enforce-eager", "--cpu-offload-gb", "20"

147 + # "--tool-call-parser",

148 + # "granite-20b-fc",

149 + # "--chat-template",

150 + # str(VLLM_PATH / "examples/tool_chat_template_granite_20b_fc.jinja"),

151 + # "--max_num_seqs",

152 + # "1",

153 + # "--enforce-eager",

154 + # "--cpu-offload-gb",

155 + # "20",

156 # ],

154 - # "supports_parallel":

155 - # False,

156 - # "supports_rocm":

157 - # False,

157 + # "supports_parallel": False,

158 + # "supports_rocm": False,

159 # },

160 "granite-3.0-8b": {

161 "model": "ibm-granite/granite-3.0-8b-instruct",

vllm/entrypoints/llm.py

-1

@@ -339,7 +339,6 @@ class LLM:

339

340 log_non_default_args(engine_args)

341

342 - # Create the Engine (autoselects V0 vs V1)

342 self.llm_engine = LLMEngine.from_engine_args(

343 engine_args=engine_args, usage_context=UsageContext.LLM_CLASS

344 )

vllm/entrypoints/openai/protocol.py

+3 -3

@@ -377,7 +377,7 @@ class ResponsesRequest(OpenAIBaseModel):

377 "environments. The salt should be random, protected from "

378 "access by 3rd parties, and long enough to be "

379 "unpredictable (e.g., 43 characters base64-encoded, corresponding "

380 - "to 256 bit). Not supported by vLLM engine V0."

380 + "to 256 bit)."

381 ),

382 )

383

@@ -763,7 +763,7 @@ class ChatCompletionRequest(OpenAIBaseModel):

763 "environments. The salt should be random, protected from "

764 "access by 3rd parties, and long enough to be "

765 "unpredictable (e.g., 43 characters base64-encoded, corresponding "

766 - "to 256 bit). Not supported by vLLM engine V0."

766 + "to 256 bit)."

767 ),

768 )

769 kv_transfer_params: dict[str, Any] | None = Field(

@@ -1249,7 +1249,7 @@ class CompletionRequest(OpenAIBaseModel):

1249 "environments. The salt should be random, protected from "

1250 "access by 3rd parties, and long enough to be "

1251 "unpredictable (e.g., 43 characters base64-encoded, corresponding "

1252 - "to 256 bit). Not supported by vLLM engine V0."

1252 + "to 256 bit)."

1253 ),

1254 )

1255

vllm/model_executor/layers/mamba/mamba_mixer2.py

-1

@@ -590,7 +590,6 @@ class MambaMixer2(MambaBase, CustomOp):

590 hidden_states, _B, _C = self.split_hidden_states_B_C_fn(hidden_states_B_C)

591 return hidden_states

592

593 - # NOTE: V0 put prefill before decode, v1 puts decode before prefill

593 num_prefills = attn_metadata.num_prefills # request count

594 num_decodes = attn_metadata.num_decode_tokens # token count (=request)

595 num_prefill_tokens = attn_metadata.num_prefill_tokens # token count

vllm/model_executor/models/interfaces.py

-2

@@ -586,13 +586,11 @@ class IsHybrid(Protocol):

586 def get_mamba_state_shape_from_config(

587 cls,

588 vllm_config: VllmConfig,

589 - use_v1: bool = True,

589 ) -> tuple[tuple[int, int], tuple[int, int, int]]:

590 """Calculate shapes for Mamba's convolutional and state caches.

591

592 Args:

593 vllm_config: vLLM config

595 - use_v1: Get shapes for V1 (or V0)

594

595 Returns:

596 Tuple containing:

vllm/model_executor/models/plamo2.py

-1

@@ -290,7 +290,6 @@ class Plamo2MambaMixer(MambaBase, CustomOp):

290 has_decode = num_decodes > 0

291 num_actual_tokens = num_prefill_tokens + num_decodes

292

293 - # NOTE: V0 put prefill before decode, v1 puts decode before prefill

293 # Separate prefill and decode by splitting varlen input

294 # Split along token dimension

295 hidden_states_d, hidden_states_p = torch.split(

[V0 deprecation] Remove more V0 references (#29088)