test-lib: make '--stress' more bisect-friendly

Let's suppose that a test somehow becomes flaky between 'master' and 'pu', and tends to fail within the first 50 repetitions when run with '--stress'. In such a case we could use 'git bisect' to find the culprit: if the test script fails with '--stress', then the commit is definitely bad, but if it survives, say, 300 repetitions, then we could consider it good with reasonable confidence. Unfortunately, all this could only be done manually, because '--stress' would run the test script repeatedly for all eternity on a good commit, and it would exit with success even when it found a failure on a bad commit. So let's make '--stress' usable with 'git bisect run': - Make it exit with failure if a failure is found. - Add the '--stress-limit=<N>' option to repeat the test script at most N times in each of the parallel jobs, and exit with success when the limit is reached. And then we could simply run something like: $ git bisect start origin/pu master $ git bisect run sh -c 'make && cd t && ./t1234-foo.sh --stress --stress-limit=300' Sure, as a brand new feature it won't be any useful right now, but in a release or three most cooking topics will already contain this, so we could automatically bisect at least newly introduced flakiness. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

SZEDER Gábor committed Feb 8, 2019 at 12:50 UTC 76e27fbfd92030a685a591cc89f63de2cc37f540
2 files changed +21 -2
t/README
+5
@@ -202,6 +202,11 @@ appropriately before running "make".
202 '.stress-<nr>' suffix, and the trash directory of the failed
203 test job is renamed to end with a '.stress-failed' suffix.
204
205 +--stress-limit=<N>::
206 + When combined with --stress run the test script repeatedly
207 + this many times in each of the parallel jobs or until one of
208 + them fails, whichever comes first.
209 +
210 You can also set the GIT_TEST_INSTALLED environment variable to
211 the bindir of an existing git installation to test that installation.
212 You still need to have built this git sandbox, from which various
t/test-lib.sh
+16 -2
@@ -152,6 +152,17 @@ do
152 ;;
153 esac
154 ;;
155 + --stress-limit=*)
156 + stress_limit=${opt#--*=}
157 + case "$stress_limit" in
158 + *[^0-9]*|0*|"")
159 + echo "error: --stress-limit=<N> requires the number of repetitions" >&2
160 + exit 1
161 + ;;
162 + *) # Good.
163 + ;;
164 + esac
165 + ;;
166 *)
167 echo "error: unknown test option '$opt'" >&2; exit 1 ;;
168 esac
@@ -237,8 +248,10 @@ then
248 exit 1
249 ' TERM INT
250
240 - cnt=0
241 - while ! test -e "$stressfail"
251 + cnt=1
252 + while ! test -e "$stressfail" &&
253 + { test -z "$stress_limit" ||
254 + test $cnt -le $stress_limit ; }
255 do
256 $TEST_SHELL_PATH "$0" "$@" >"$TEST_RESULTS_BASE.stress-$job_nr.out" 2>&1 &
257 test_pid=$!
@@ -261,6 +274,7 @@ then
274
275 if test -f "$stressfail"
276 then
277 + stress_exit=1
278 echo "Log(s) of failed test run(s):"
279 for failed_job_nr in $(sort -n "$stressfail")
280 do