| 1 | # Unit Testing |
| 2 | |
| 3 | This page explains how to write unit tests to verify the implementation of your model. |
| 4 | |
| 5 | ## Required Tests |
| 6 | |
| 7 | These tests are necessary to get your PR merged into vLLM library. |
| 8 | Without them, the CI for your PR will fail. |
| 9 | |
| 10 | ### Model loading |
| 11 | |
| 12 | Include an example HuggingFace repository for your model in [tests/models/registry.py](../../../tests/models/registry.py). |
| 13 | This enables a unit test that loads dummy weights to ensure that the model can be initialized in vLLM. |
| 14 | |
| 15 | !!! important |
| 16 | The list of models in each section should be maintained in alphabetical order. |
| 17 | |
| 18 | !!! tip |
| 19 | If your model requires a development version of HF Transformers, you can set |
| 20 | `min_transformers_version` to skip the test in CI until the model is released. |
| 21 | |
| 22 | ## Optional Tests |
| 23 | |
| 24 | These tests are optional to get your PR merged into vLLM library. |
| 25 | Passing these tests provides more confidence that your implementation is correct, and helps avoid future regressions. |
| 26 | |
| 27 | ### Model correctness |
| 28 | |
| 29 | These tests compare the model outputs of vLLM against [HF Transformers](https://github.com/huggingface/transformers). You can add new tests under the subdirectories of [tests/models](../../../tests/models). |
| 30 | |
| 31 | #### Generative models |
| 32 | |
| 33 | For [generative models](../../models/generative_models.md), there are two levels of correctness tests, as defined in [tests/models/utils.py](../../../tests/models/utils.py): |
| 34 | |
| 35 | - Exact correctness (`check_outputs_equal`): The text outputted by vLLM should exactly match the text outputted by HF. |
| 36 | - Logprobs similarity (`check_logprobs_close`): The logprobs outputted by vLLM should be in the top-k logprobs outputted by HF, and vice versa. |
| 37 | |
| 38 | #### Pooling models |
| 39 | |
| 40 | For [pooling models](../../models/pooling_models/README.md), we simply check the cosine similarity, as defined in [tests/models/utils.py](../../../tests/models/utils.py). |
| 41 | |
| 42 | ### Multi-modal processing |
| 43 | |
| 44 | #### Common tests |
| 45 | |
| 46 | Adding your model to [tests/models/multimodal/processing/test_common.py](../../../tests/models/multimodal/processing/test_common.py) verifies that the following input combinations result in the same outputs: |
| 47 | |
| 48 | - Text + multi-modal data |
| 49 | - Tokens + multi-modal data |
| 50 | - Text + cached multi-modal data |
| 51 | - Tokens + cached multi-modal data |
| 52 | |
| 53 | #### Model-specific tests |
| 54 | |
| 55 | You can add a new file under [tests/models/multimodal/processing](../../../tests/models/multimodal/processing) to run tests that only apply to your model. |
| 56 | |
| 57 | For example, if the HF processor for your model accepts user-specified keyword arguments, you can verify that the keyword arguments are being applied correctly, such as in [tests/models/multimodal/processing/test_phi3v.py](../../../tests/models/multimodal/processing/test_phi3v.py). |