A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Doc] Basic guide for writing unit tests for new models (#11951)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jan 11, 2025 at 21:27 UTC
a991f7d5085e3e9474531f78639766eac3af607c
6 files changed
+81
-3
docs/source/contributing/model/basic.md
+1
-1
@@ -1,6 +1,6 @@
1
(new-model-basic)=
2
3
-# Basic Implementation
3
+# Implementing a Basic Model
4
5
This guide walks you through the steps to implement a basic vLLM model.
6
docs/source/contributing/model/index.md
+1
@@ -10,6 +10,7 @@ This section provides more information on how to integrate a [PyTorch](https://p
10
11
basic
12
registration
13
+tests
14
multimodal
15
```
16
docs/source/contributing/model/registration.md
+1
-2
@@ -1,6 +1,6 @@
1
(new-model-registration)=
2
3
-# Model Registration
3
+# Registering a Model to vLLM
4
5
vLLM relies on a model registry to determine how to run each model.
6
A list of pre-registered architectures can be found [here](#supported-models).
@@ -15,7 +15,6 @@ This gives you the ability to modify the codebase and test your model.
15
16
After you have implemented your model (see [tutorial](#new-model-basic)), put it into the <gh-dir:vllm/model_executor/models> directory.
17
Then, add your model class to `_VLLM_MODELS` in <gh-file:vllm/model_executor/models/registry.py> so that it is automatically registered upon importing vLLM.
18
-You should also include an example HuggingFace repository for this model in <gh-file:tests/models/registry.py> to run the unit tests.
18
Finally, update our [list of supported models](#supported-models) to promote your model!
19
20
```{important}
docs/source/contributing/model/tests.md
new
+63
@@ -0,0 +1,63 @@
1
+(new-model-tests)=
2
+
3
+# Writing Unit Tests
4
+
5
+This page explains how to write unit tests to verify the implementation of your model.
6
+
7
+## Required Tests
8
+
9
+These tests are necessary to get your PR merged into vLLM library.
10
+Without them, the CI for your PR will fail.
11
+
12
+### Model loading
13
+
14
+Include an example HuggingFace repository for your model in <gh-file:tests/models/registry.py>.
15
+This enables a unit test that loads dummy weights to ensure that the model can be initialized in vLLM.
16
+
17
+```{important}
18
+The list of models in each section should be maintained in alphabetical order.
19
+```
20
+
21
+```{tip}
22
+If your model requires a development version of HF Transformers, you can set
23
+`min_transformers_version` to skip the test in CI until the model is released.
24
+```
25
+
26
+## Optional Tests
27
+
28
+These tests are optional to get your PR merged into vLLM library.
29
+Passing these tests provides more confidence that your implementation is correct, and helps avoid future regressions.
30
+
31
+### Model correctness
32
+
33
+These tests compare the model outputs of vLLM against [HF Transformers](https://github.com/huggingface/transformers). You can add new tests under the subdirectories of <gh-dir:tests/models>.
34
+
35
+#### Generative models
36
+
37
+For [generative models](#generative-models), there are two levels of correctness tests, as defined in <gh-file:tests/models/utils.py>:
38
+
39
+- Exact correctness (`check_outputs_equal`): The text outputted by vLLM should exactly match the text outputted by HF.
40
+- Logprobs similarity (`check_logprobs_close`): The logprobs outputted by vLLM should be in the top-k logprobs outputted by HF, and vice versa.
41
+
42
+#### Pooling models
43
+
44
+For [pooling models](#pooling-models), we simply check the cosine similarity, as defined in <gh-file:tests/models/embedding/utils.py>.
45
+
46
+(mm-processing-tests)=
47
+
48
+### Multi-modal processing
49
+
50
+#### Common tests
51
+
52
+Adding your model to <gh-file:tests/models/multimodal/processing/test_common.py> verifies that the following input combinations result in the same outputs:
53
+
54
+- Text + multi-modal data
55
+- Tokens + multi-modal data
56
+- Text + cached multi-modal data
57
+- Tokens + cached multi-modal data
58
+
59
+#### Model-specific tests
60
+
61
+You can add a new file under <gh-dir:tests/models/multimodal/processing> to run tests that only apply to your model.
62
+
63
+For example, if the HF processor for your model accepts user-specified keyword arguments, you can verify that the keyword arguments are being applied correctly, such as in <gh-file:tests/models/multimodal/processing/test_phi3v.py>.
tests/models/registry.py
+5
@@ -22,6 +22,11 @@ class _HfExamplesInfo:
22
for speculative decoding.
23
"""
24
25
+ min_transformers_version: Optional[str] = None
26
+ """
27
+ The minimum version of HF Transformers that is required to run this model.
28
+ """
29
+
30
is_available_online: bool = True
31
"""
32
Set this to ``False`` if the name of this architecture no longer exists on
tests/models/test_initialization.py
+10
@@ -1,7 +1,9 @@
1
from unittest.mock import patch
2
3
import pytest
4
+from packaging.version import Version
5
from transformers import PretrainedConfig
6
+from transformers import __version__ as TRANSFORMERS_VERSION
7
8
from vllm import LLM
9
@@ -13,6 +15,14 @@ def test_can_initialize(model_arch):
15
model_info = HF_EXAMPLE_MODELS.get_hf_info(model_arch)
16
if not model_info.is_available_online:
17
pytest.skip("Model is not available online")
18
+ if model_info.min_transformers_version is not None:
19
+ current_version = TRANSFORMERS_VERSION
20
+ required_version = model_info.min_transformers_version
21
+ if Version(current_version) < Version(required_version):
22
+ pytest.skip(
23
+ f"You have `transformers=={current_version}` installed, but "
24
+ f"`transformers>={required_version}` is required to run this "
25
+ "model")
26
27
# Avoid OOM
28
def hf_overrides(hf_config: PretrainedConfig) -> PretrainedConfig: