| 1 | # CI Failures |
| 2 | |
| 3 | What should I do when a CI job fails on my PR, but I don't think my PR caused |
| 4 | the failure? |
| 5 | |
| 6 | - Check the dashboard of current CI test failures: |
| 7 | 👉 [CI Failures Dashboard](https://github.com/orgs/vllm-project/projects/20) |
| 8 | |
| 9 | - If your failure **is already listed**, it's likely unrelated to your PR. |
| 10 | Help fixing it is always welcome! |
| 11 | - Leave comments with links to additional instances of the failure. |
| 12 | - React with a 👍 to signal how many are affected. |
| 13 | |
| 14 | - If your failure **is not listed**, you should **file an issue**. |
| 15 | |
| 16 | ## Filing a CI Test Failure Issue |
| 17 | |
| 18 | - **File a bug report:** |
| 19 | 👉 [New CI Failure Report](https://github.com/vllm-project/vllm/issues/new?template=450-ci-failure.yml) |
| 20 | |
| 21 | - **Use this title format:** |
| 22 | |
| 23 | ```text |
| 24 | [CI Failure]: failing-test-job - regex/matching/failing:test |
| 25 | ``` |
| 26 | |
| 27 | - **For the environment field:** |
| 28 | |
| 29 | ```text |
| 30 | Still failing on main as of commit abcdef123 |
| 31 | ``` |
| 32 | |
| 33 | - **In the description, include failing tests:** |
| 34 | |
| 35 | ```text |
| 36 | FAILED failing/test.py:failing_test1 - Failure description |
| 37 | FAILED failing/test.py:failing_test2 - Failure description |
| 38 | https://github.com/orgs/vllm-project/projects/20 |
| 39 | https://github.com/vllm-project/vllm/issues/new?template=400-bug-report.yml |
| 40 | FAILED failing/test.py:failing_test3 - Failure description |
| 41 | ``` |
| 42 | |
| 43 | - **Attach logs** (collapsible section example): |
| 44 | <details> |
| 45 | <summary>Logs:</summary> |
| 46 | |
| 47 | ```text |
| 48 | ERROR 05-20 03:26:38 [dump_input.py:68] Dumping input data |
| 49 | --- Logging error --- |
| 50 | Traceback (most recent call last): |
| 51 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 203, in execute_model |
| 52 | return self.model_executor.execute_model(scheduler_output) |
| 53 | ... |
| 54 | FAILED failing/test.py:failing_test1 - Failure description |
| 55 | FAILED failing/test.py:failing_test2 - Failure description |
| 56 | FAILED failing/test.py:failing_test3 - Failure description |
| 57 | ``` |
| 58 | |
| 59 | </details> |
| 60 | |
| 61 | ## Logs Wrangling |
| 62 | |
| 63 | Logs are public; no Buildkite login needed. |
| 64 | [.buildkite/scripts/ci-fetch-log.sh](../../../.buildkite/scripts/ci-fetch-log.sh) |
| 65 | saves each log as `ci-<build>-<job-name>.log`, stripped of timestamps and |
| 66 | ANSI codes: |
| 67 | |
| 68 | ```bash |
| 69 | # All failed jobs in a PR's latest build (current branch's PR if omitted): |
| 70 | .buildkite/scripts/ci-fetch-log.sh --pr <PR> |
| 71 | |
| 72 | # All failed jobs in a build (--soft also includes soft-failed jobs; |
| 73 | # --all fetches every finished job): |
| 74 | .buildkite/scripts/ci-fetch-log.sh "https://buildkite.com/vllm/ci/builds/<N>" |
| 75 | |
| 76 | # One job — `gh pr checks` URLs (#<job_uuid>) and web UI URLs (?sid=) both |
| 77 | # work; pass "-" as a second argument to stream to stdout: |
| 78 | .buildkite/scripts/ci-fetch-log.sh "https://buildkite.com/vllm/ci/builds/<N>#<job_uuid>" |
| 79 | ``` |
| 80 | |
| 81 | To clean an already-downloaded log: |
| 82 | |
| 83 | [.buildkite/scripts/ci-clean-log.sh](../../../.buildkite/scripts/ci-clean-log.sh) |
| 84 | |
| 85 | ```bash |
| 86 | ./ci-clean-log.sh ci.log |
| 87 | ``` |
| 88 | |
| 89 | Use a tool [wl-clipboard](https://github.com/bugaevc/wl-clipboard) for quick copy-pasting: |
| 90 | |
| 91 | ```bash |
| 92 | tail -525 ci_build.log | wl-copy |
| 93 | ``` |
| 94 | |
| 95 | ## Investigating a CI Test Failure |
| 96 | |
| 97 | 1. Go to 👉 [Buildkite main branch](https://buildkite.com/vllm/ci/builds?branch=main) |
| 98 | 2. Bisect to find the first build that shows the issue. |
| 99 | 3. Add your findings to the GitHub issue. |
| 100 | 4. If you find a strong candidate PR, mention it in the issue and ping contributors. |
| 101 | |
| 102 | ## Reproducing a Failure |
| 103 | |
| 104 | CI test failures may be flaky. Use a bash loop to run repeatedly: |
| 105 | |
| 106 | [.buildkite/scripts/rerun-test.sh](../../../.buildkite/scripts/rerun-test.sh) |
| 107 | |
| 108 | ```bash |
| 109 | ./rerun-test.sh tests/v1/engine/test_engine_core_client.py::test_kv_cache_events[True-tcp] |
| 110 | ``` |
| 111 | |
| 112 | ## Submitting a PR |
| 113 | |
| 114 | If you submit a PR to fix a CI failure: |
| 115 | |
| 116 | - Link the PR to the issue: |
| 117 | Add `Closes #12345` to the PR description. |
| 118 | - Add the `ci-failure` label: |
| 119 | This helps track it in the [CI Failures GitHub Project](https://github.com/orgs/vllm-project/projects/20). |
| 120 | |
| 121 | ## Other Resources |
| 122 | |
| 123 | - 🔍 [Test Reliability on `main`](https://buildkite.com/organizations/vllm/analytics/suites/ci-1/tests?branch=main&order=ASC&sort_by=reliability) |
| 124 | - 🧪 [Latest Buildkite CI Runs](https://buildkite.com/vllm/ci/builds?branch=main) |
| 125 | |
| 126 | ## Daily Triage |
| 127 | |
| 128 | Use [Buildkite analytics (2-day view)](https://buildkite.com/organizations/vllm/analytics/suites/ci-1/tests?branch=main&period=2days) to: |
| 129 | |
| 130 | - Identify recent test failures **on `main`**. |
| 131 | - Exclude legitimate test failures on PRs. |
| 132 | - (Optional) Ignore tests with 0% reliability. |
| 133 | |
| 134 | Compare to the [CI Failures Dashboard](https://github.com/orgs/vllm-project/projects/20). |