@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a991f7d508
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

[Doc] Basic guide for writing unit tests for new models (#11951)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Jan 11, 2025 at 21:27 UTC a991f7d5085e3e9474531f78639766eac3af607c
6 files changed +81 -3
docs/source/contributing/model/basic.md
+1 -1
@@ -1,6 +1,6 @@
1 (new-model-basic)=
2
3 -# Basic Implementation
3 +# Implementing a Basic Model
4
5 This guide walks you through the steps to implement a basic vLLM model.
6
docs/source/contributing/model/index.md
+1
@@ -10,6 +10,7 @@ This section provides more information on how to integrate a [PyTorch](https://p
10
11 basic
12 registration
13 +tests
14 multimodal
15 ```
16
docs/source/contributing/model/registration.md
+1 -2
@@ -1,6 +1,6 @@
1 (new-model-registration)=
2
3 -# Model Registration
3 +# Registering a Model to vLLM
4
5 vLLM relies on a model registry to determine how to run each model.
6 A list of pre-registered architectures can be found [here](#supported-models).
@@ -15,7 +15,6 @@ This gives you the ability to modify the codebase and test your model.
15
16 After you have implemented your model (see [tutorial](#new-model-basic)), put it into the <gh-dir:vllm/model_executor/models> directory.
17 Then, add your model class to `_VLLM_MODELS` in <gh-file:vllm/model_executor/models/registry.py> so that it is automatically registered upon importing vLLM.
18 -You should also include an example HuggingFace repository for this model in <gh-file:tests/models/registry.py> to run the unit tests.
18 Finally, update our [list of supported models](#supported-models) to promote your model!
19
20 ```{important}
docs/source/contributing/model/tests.md new
+63
@@ -0,0 +1,63 @@
1 +(new-model-tests)=
2 +
3 +# Writing Unit Tests
4 +
5 +This page explains how to write unit tests to verify the implementation of your model.
6 +
7 +## Required Tests
8 +
9 +These tests are necessary to get your PR merged into vLLM library.
10 +Without them, the CI for your PR will fail.
11 +
12 +### Model loading
13 +
14 +Include an example HuggingFace repository for your model in <gh-file:tests/models/registry.py>.
15 +This enables a unit test that loads dummy weights to ensure that the model can be initialized in vLLM.
16 +
17 +```{important}
18 +The list of models in each section should be maintained in alphabetical order.
19 +```
20 +
21 +```{tip}
22 +If your model requires a development version of HF Transformers, you can set
23 +`min_transformers_version` to skip the test in CI until the model is released.
24 +```
25 +
26 +## Optional Tests
27 +
28 +These tests are optional to get your PR merged into vLLM library.
29 +Passing these tests provides more confidence that your implementation is correct, and helps avoid future regressions.
30 +
31 +### Model correctness
32 +
33 +These tests compare the model outputs of vLLM against [HF Transformers](https://github.com/huggingface/transformers). You can add new tests under the subdirectories of <gh-dir:tests/models>.
34 +
35 +#### Generative models
36 +
37 +For [generative models](#generative-models), there are two levels of correctness tests, as defined in <gh-file:tests/models/utils.py>:
38 +
39 +- Exact correctness (`check_outputs_equal`): The text outputted by vLLM should exactly match the text outputted by HF.
40 +- Logprobs similarity (`check_logprobs_close`): The logprobs outputted by vLLM should be in the top-k logprobs outputted by HF, and vice versa.
41 +
42 +#### Pooling models
43 +
44 +For [pooling models](#pooling-models), we simply check the cosine similarity, as defined in <gh-file:tests/models/embedding/utils.py>.
45 +
46 +(mm-processing-tests)=
47 +
48 +### Multi-modal processing
49 +
50 +#### Common tests
51 +
52 +Adding your model to <gh-file:tests/models/multimodal/processing/test_common.py> verifies that the following input combinations result in the same outputs:
53 +
54 +- Text + multi-modal data
55 +- Tokens + multi-modal data
56 +- Text + cached multi-modal data
57 +- Tokens + cached multi-modal data
58 +
59 +#### Model-specific tests
60 +
61 +You can add a new file under <gh-dir:tests/models/multimodal/processing> to run tests that only apply to your model.
62 +
63 +For example, if the HF processor for your model accepts user-specified keyword arguments, you can verify that the keyword arguments are being applied correctly, such as in <gh-file:tests/models/multimodal/processing/test_phi3v.py>.
tests/models/registry.py
+5
@@ -22,6 +22,11 @@ class _HfExamplesInfo:
22 for speculative decoding.
23 """
24
25 + min_transformers_version: Optional[str] = None
26 + """
27 + The minimum version of HF Transformers that is required to run this model.
28 + """
29 +
30 is_available_online: bool = True
31 """
32 Set this to ``False`` if the name of this architecture no longer exists on
tests/models/test_initialization.py
+10
@@ -1,7 +1,9 @@
1 from unittest.mock import patch
2
3 import pytest
4 +from packaging.version import Version
5 from transformers import PretrainedConfig
6 +from transformers import __version__ as TRANSFORMERS_VERSION
7
8 from vllm import LLM
9
@@ -13,6 +15,14 @@ def test_can_initialize(model_arch):
15 model_info = HF_EXAMPLE_MODELS.get_hf_info(model_arch)
16 if not model_info.is_available_online:
17 pytest.skip("Model is not available online")
18 + if model_info.min_transformers_version is not None:
19 + current_version = TRANSFORMERS_VERSION
20 + required_version = model_info.min_transformers_version
21 + if Version(current_version) < Version(required_version):
22 + pytest.skip(
23 + f"You have `transformers=={current_version}` installed, but "
24 + f"`transformers>={required_version}` is required to run this "
25 + "model")
26
27 # Avoid OOM
28 def hf_overrides(hf_config: PretrainedConfig) -> PretrainedConfig: