| 1 | name: 🧪 CI failure report |
| 2 | description: Report a failing test. |
| 3 | title: "[CI Failure]: " |
| 4 | labels: ["ci-failure"] |
| 5 | |
| 6 | body: |
| 7 | - type: markdown |
| 8 | attributes: |
| 9 | value: > |
| 10 | #### Include the name of the failing Buildkite step and test file in the title. |
| 11 | - type: input |
| 12 | attributes: |
| 13 | label: Name of failing test |
| 14 | description: | |
| 15 | Paste in the fully-qualified name of the failing test from the logs. |
| 16 | placeholder: | |
| 17 | `path/to/test_file.py::test_name[params]` |
| 18 | validations: |
| 19 | required: true |
| 20 | - type: checkboxes |
| 21 | attributes: |
| 22 | label: Basic information |
| 23 | description: Select all items that apply to the failing test. |
| 24 | options: |
| 25 | - label: Flaky test |
| 26 | - label: Can reproduce locally |
| 27 | - label: Caused by external libraries (e.g. bug in `transformers`) |
| 28 | - type: textarea |
| 29 | attributes: |
| 30 | label: 🧪 Describe the failing test |
| 31 | description: | |
| 32 | Please provide a clear and concise description of the failing test. |
| 33 | placeholder: | |
| 34 | A clear and concise description of the failing test. |
| 35 | |
| 36 | ``` |
| 37 | The error message you got, with the full traceback and the error logs with [dump_input.py:##] if present. |
| 38 | ``` |
| 39 | validations: |
| 40 | required: true |
| 41 | - type: textarea |
| 42 | attributes: |
| 43 | label: 📝 History of failing test |
| 44 | description: | |
| 45 | Since when did the test start to fail? |
| 46 | You can look up its history via [Buildkite Test Suites](https://buildkite.com/organizations/vllm/analytics/suites/ci-1/tests?branch=main). |
| 47 | |
| 48 | If you have time, identify the PR that caused the test to fail on main. You can do so via the following methods: |
| 49 | |
| 50 | - Use Buildkite Test Suites to find the PR where the test failure first occurred, and reproduce the failure locally. |
| 51 | |
| 52 | - Run [`git bisect`](https://git-scm.com/docs/git-bisect) locally. |
| 53 | |
| 54 | - Manually unblock Buildkite steps for suspected PRs on main and check the results. (authorized users only) |
| 55 | placeholder: | |
| 56 | Approximate timeline and/or problematic PRs |
| 57 | |
| 58 | A link to the Buildkite analytics of the failing test (if available) |
| 59 | validations: |
| 60 | required: true |
| 61 | - type: textarea |
| 62 | attributes: |
| 63 | label: CC List. |
| 64 | description: > |
| 65 | The list of people you want to CC. Usually, this includes those who worked on the PR that failed the test. |
| 66 | - type: markdown |
| 67 | attributes: |
| 68 | value: > |
| 69 | Thanks for reporting 🙏! |