ci: make it easier to find failed tests' logs in the GitHub workflow

When investigating a test failure, the time that matters most is the time it takes from getting aware of the failure to displaying the output of the failing test case. You currently have to know a lot of implementation details when investigating test failures in the CI runs. The first step is easy: the failed job is marked quite clearly, but when opening it, the failed step is expanded, which in our case is the one running `ci/run-build-and-tests.sh`. This step, most notably, only offers a high-level view of what went wrong: it prints the output of `prove` which merely tells the reader which test script failed. The actually interesting part is in the detailed log of said failed test script. But that log is shown in the CI run's step that runs `ci/print-test-failures.sh`. And that step is _not_ expanded in the web UI by default. It is even marked as "successful", which makes it very easy to miss that there is useful information hidden in there. Let's help the reader by showing the failed tests' detailed logs in the step that is expanded automatically, i.e. directly after the test suite failed. This also helps the situation where the _build_ failed and the `print-test-failures` step was executed under the assumption that the _test suite_ failed, and consequently failed to find any failed tests. An alternative way to implement this patch would be to source `ci/print-test-failures.sh` in the `handle_test_failures` function to show these logs. However, over the course of the next few commits, we want to introduce some grouping which would be harder to achieve that way (for example, we do want a leaner, and colored, preamble for each failed test script, and it would be trickier to accommodate the lack of nested groupings in GitHub workflows' output). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Johannes Schindelin committed May 21, 2022 at 22:18 UTC 08dccc8fc1f248c8de7e87ac6e435edac4f77ebe
4 files changed +27 -14
.github/workflows/main.yml
-12
@@ -119,10 +119,6 @@ jobs:
119 - name: test
120 shell: bash
121 run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10
122 - - name: ci/print-test-failures.sh
123 - if: failure()
124 - shell: bash
125 - run: ci/print-test-failures.sh
122 - name: Upload failed tests' directories
123 if: failure() && env.FAILED_TEST_ARTIFACTS != ''
124 uses: actions/upload-artifact@v2
@@ -204,10 +200,6 @@ jobs:
200 env:
201 NO_SVN_TESTS: 1
202 run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10
207 - - name: ci/print-test-failures.sh
208 - if: failure()
209 - shell: bash
210 - run: ci/print-test-failures.sh
203 - name: Upload failed tests' directories
204 if: failure() && env.FAILED_TEST_ARTIFACTS != ''
205 uses: actions/upload-artifact@v2
@@ -261,8 +253,6 @@ jobs:
253 - uses: actions/checkout@v2
254 - run: ci/install-dependencies.sh
255 - run: ci/run-build-and-tests.sh
264 - - run: ci/print-test-failures.sh
265 - if: failure()
256 - name: Upload failed tests' directories
257 if: failure() && env.FAILED_TEST_ARTIFACTS != ''
258 uses: actions/upload-artifact@v2
@@ -292,8 +282,6 @@ jobs:
282 - uses: actions/checkout@v1
283 - run: ci/install-docker-dependencies.sh
284 - run: ci/run-build-and-tests.sh
295 - - run: ci/print-test-failures.sh
296 - if: failure()
285 - name: Upload failed tests' directories
286 if: failure() && env.FAILED_TEST_ARTIFACTS != ''
287 uses: actions/upload-artifact@v1
ci/lib.sh
+23
@@ -78,6 +78,10 @@ check_unignored_build_artifacts () {
78 }
79 }
80
81 +handle_failed_tests () {
82 + return 1
83 +}
84 +
85 # GitHub Action doesn't set TERM, which is required by tput
86 export TERM=${TERM:-dumb}
87
@@ -123,6 +127,25 @@ then
127 CI_JOB_ID="$GITHUB_RUN_ID"
128 CC="${CC_PACKAGE:-${CC:-gcc}}"
129 DONT_SKIP_TAGS=t
130 + handle_failed_tests () {
131 + mkdir -p t/failed-test-artifacts
132 + echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV
133 +
134 + for test_exit in t/test-results/*.exit
135 + do
136 + test 0 != "$(cat "$test_exit")" || continue
137 +
138 + test_name="${test_exit%.exit}"
139 + test_name="${test_name##*/}"
140 + printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n"
141 + cat "t/test-results/$test_name.out"
142 +
143 + trash_dir="t/trash directory.$test_name"
144 + cp "t/test-results/$test_name.out" t/failed-test-artifacts/
145 + tar czf t/failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir"
146 + done
147 + return 1
148 + }
149
150 cache_dir="$HOME/none"
151
ci/run-build-and-tests.sh
+2 -1
@@ -48,7 +48,8 @@ esac
48 make
49 if test -n "$run_tests"
50 then
51 - make test
51 + make test ||
52 + handle_failed_tests
53 fi
54 check_unignored_build_artifacts
55
ci/run-test-slice.sh
+2 -1
@@ -12,6 +12,7 @@ esac
12
13 make --quiet -C t T="$(cd t &&
14 ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh |
15 - tr '\n' ' ')"
15 + tr '\n' ' ')" ||
16 +handle_failed_tests
17
18 check_unignored_build_artifacts