test-lib: allow selecting tests by substring/glob with --run
Many of our test scripts have several "setup" tests. It's a lot easier to say ./t0050-filesystem.sh --run=setup,9 in order to run all the setup tests as well as test #9, than it is to track down what all the setup tests are and enter all their numbers in the list. Also, I often find myself wanting to run just one or a couple tests from the test file, but I don't know the numbering of any of the tests -- to get it I either have to first run the whole test file (or start counting by hand or figure out some other clever but non-obvious tricks). It's really convenient to be able to just look at the test description(s) and then run ./t6416-recursive-corner-cases.sh --run=symlink or ./t6402-merge-rename.sh --run='setup,unnecessary update' Add such an ability to test selection which relies on merely matching against the test description. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Elijah Newren committed
Oct 18, 2020 at 00:23 UTC
f21ac368f15b0a4df301c65a9a603c4e57277f57
3 files changed
+74
-47
t/README
+31
-17
@@ -258,16 +258,21 @@ For an individual test suite --run could be used to specify that
258
only some tests should be run or that some tests should be
259
excluded from a run.
260
261
-The argument for --run is a list of individual test numbers or
262
-ranges with an optional negation prefix that define what tests in
263
-a test suite to include in the run. A range is two numbers
264
-separated with a dash and matches a range of tests with both ends
265
-been included. You may omit the first or the second number to
266
-mean "from the first test" or "up to the very last test"
267
-respectively.
268
-
269
-Optional prefix of '!' means that the test or a range of tests
270
-should be excluded from the run.
261
+The argument for --run, <test-selector>, is a list of description
262
+substrings or globs or individual test numbers or ranges with an
263
+optional negation prefix (of '!') that define what tests in a test
264
+suite to include (or exclude, if negated) in the run. A range is two
265
+numbers separated with a dash and matches a range of tests with both
266
+ends been included. You may omit the first or the second number to
267
+mean "from the first test" or "up to the very last test" respectively.
268
+
269
+The argument to --run is split on commas into separate strings,
270
+numbers, and ranges, and picks all tests that match any of the
271
+individual selection criteria. If the substring of the description
272
+text that you want to match includes a comma, use the glob character
273
+'?' instead. For example --run='rebase,merge?cherry-pick' would match
274
+on all tests that match either the glob *rebase* or the glob
275
+*merge?cherry-pick*.
276
277
If --run starts with an unprefixed number or range the initial
278
set of tests to run is empty. If the first item starts with '!'
@@ -275,9 +280,6 @@ all the tests are added to the initial set. After initial set is
280
determined every test number or range is added or excluded from
281
the set one by one, from left to right.
282
278
-Individual numbers or ranges could be separated either by a space
279
-or a comma.
280
-
283
For example, to run only tests up to a specific test (21), one
284
could do this:
285
@@ -290,7 +292,7 @@ or this:
292
Common case is to run several setup tests (1, 2, 3) and then a
293
specific test (21) that relies on that setup:
294
293
- $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'
295
+ $ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'
296
297
or:
298
@@ -298,17 +300,17 @@ or:
300
301
or:
302
301
- $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'
303
+ $ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'
304
305
As noted above, the test set is built by going through the items
306
from left to right, so this:
307
306
- $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
308
+ $ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'
309
310
will run tests 1, 2, and 4. Items that come later have higher
311
precedence. It means that this:
312
311
- $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
313
+ $ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'
314
315
would just run tests from 1 to 4, including 3.
316
@@ -317,6 +319,18 @@ test in the test suite except from 7 up to 11:
319
320
$ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
321
322
+Sometimes there may be multiple tests with e.g. "setup" in their name
323
+that are needed and rather than figuring out the number for all of them
324
+we can just use "setup" as a substring/glob to match against the test
325
+description:
326
+
327
+ $ sh ./t0050-filesystem.sh --run=setup,9-11
328
+
329
+or one could select both the setup tests and the rename ones (assuming all
330
+relevant tests had those words in their descriptions):
331
+
332
+ $ sh ./t0050-filesystem.sh --run=setup,rename
333
+
334
Some tests in a test suite rely on the previous tests performing
335
certain actions, specifically some tests are designated as
336
"setup" test, so you cannot _arbitrarily_ disable one test and
t/t0000-basic.sh
+31
-22
@@ -430,7 +430,7 @@ test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' "
430
431
test_expect_success '--run basic' "
432
run_sub_test_lib_test run-basic \
433
- '--run basic' --run='1 3 5' <<-\\EOF &&
433
+ '--run basic' --run='1,3,5' <<-\\EOF &&
434
for i in 1 2 3 4 5 6
435
do
436
test_expect_success \"passing test #\$i\" 'true'
@@ -472,7 +472,7 @@ test_expect_success '--run with a range' "
472
473
test_expect_success '--run with two ranges' "
474
run_sub_test_lib_test run-two-ranges \
475
- '--run with two ranges' --run='1-2 5-6' <<-\\EOF &&
475
+ '--run with two ranges' --run='1-2,5-6' <<-\\EOF &&
476
for i in 1 2 3 4 5 6
477
do
478
test_expect_success \"passing test #\$i\" 'true'
@@ -556,7 +556,7 @@ test_expect_success '--run with basic negation' "
556
557
test_expect_success '--run with two negations' "
558
run_sub_test_lib_test run-two-neg \
559
- '--run with two negations' --run='"'!3 !6'"' <<-\\EOF &&
559
+ '--run with two negations' --run='"'!3,!6'"' <<-\\EOF &&
560
for i in 1 2 3 4 5 6
561
do
562
test_expect_success \"passing test #\$i\" 'true'
@@ -577,7 +577,7 @@ test_expect_success '--run with two negations' "
577
578
test_expect_success '--run a range and negation' "
579
run_sub_test_lib_test run-range-and-neg \
580
- '--run a range and negation' --run='"'-4 !2'"' <<-\\EOF &&
580
+ '--run a range and negation' --run='"'-4,!2'"' <<-\\EOF &&
581
for i in 1 2 3 4 5 6
582
do
583
test_expect_success \"passing test #\$i\" 'true'
@@ -620,7 +620,7 @@ test_expect_success '--run range negation' "
620
test_expect_success '--run include, exclude and include' "
621
run_sub_test_lib_test run-inc-neg-inc \
622
'--run include, exclude and include' \
623
- --run='"'1-5 !1-3 2'"' <<-\\EOF &&
623
+ --run='"'1-5,!1-3,2'"' <<-\\EOF &&
624
for i in 1 2 3 4 5 6
625
do
626
test_expect_success \"passing test #\$i\" 'true'
@@ -664,7 +664,7 @@ test_expect_success '--run include, exclude and include, comma separated' "
664
test_expect_success '--run exclude and include' "
665
run_sub_test_lib_test run-neg-inc \
666
'--run exclude and include' \
667
- --run='"'!3- 5'"' <<-\\EOF &&
667
+ --run='"'!3-,5'"' <<-\\EOF &&
668
for i in 1 2 3 4 5 6
669
do
670
test_expect_success \"passing test #\$i\" 'true'
@@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' "
705
EOF
706
"
707
708
-test_expect_success '--run invalid range start' "
708
+test_expect_success '--run substring selector' "
709
+ run_sub_test_lib_test run-substring-selector \
710
+ '--run empty selectors' \
711
+ --run='relevant' <<-\\EOF &&
712
+ test_expect_success \"relevant test\" 'true'
713
+ for i in 1 2 3 4 5 6
714
+ do
715
+ test_expect_success \"other test #\$i\" 'true'
716
+ done
717
+ test_done
718
+ EOF
719
+ check_sub_test_lib_test run-substring-selector <<-\\EOF
720
+ > ok 1 - relevant test
721
+ > ok 2 # skip other test #1 (--run)
722
+ > ok 3 # skip other test #2 (--run)
723
+ > ok 4 # skip other test #3 (--run)
724
+ > ok 5 # skip other test #4 (--run)
725
+ > ok 6 # skip other test #5 (--run)
726
+ > ok 7 # skip other test #6 (--run)
727
+ > # passed all 7 test(s)
728
+ > 1..7
729
+ EOF
730
+"
731
+
732
+test_expect_success '--run keyword selection' "
733
run_sub_test_lib_test_err run-inv-range-start \
734
'--run invalid range start' \
735
--run='a-5' <<-\\EOF &&
@@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' "
759
EOF_ERR
760
"
761
738
-test_expect_success '--run invalid selector' "
739
- run_sub_test_lib_test_err run-inv-selector \
740
- '--run invalid selector' \
741
- --run='1?' <<-\\EOF &&
742
- test_expect_success \"passing test #1\" 'true'
743
- test_done
744
- EOF
745
- check_sub_test_lib_test_err run-inv-selector \
746
- <<-\\EOF_OUT 3<<-\\EOF_ERR
747
- > FATAL: Unexpected exit with code 1
748
- EOF_OUT
749
- > error: --run: invalid non-numeric in test selector: '1?'
750
- EOF_ERR
751
-"
752
-
762
763
test_set_prereq HAVEIT
764
haveit=no
t/test-lib.sh
+12
-8
@@ -769,15 +769,17 @@ match_pattern_list () {
769
}
770
771
match_test_selector_list () {
772
+ operation="$1"
773
+ shift
774
title="$1"
775
shift
776
arg="$1"
777
shift
778
test -z "$1" && return 0
779
778
- # Both commas and whitespace are accepted as separators.
780
+ # Commas are accepted as separators.
781
OLDIFS=$IFS
780
- IFS=' ,'
782
+ IFS=','
783
set -- $1
784
IFS=$OLDIFS
785
@@ -805,13 +807,13 @@ match_test_selector_list () {
807
*-*)
808
if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
809
then
808
- echo "error: $title: invalid non-numeric in range" \
810
+ echo "error: $operation: invalid non-numeric in range" \
811
"start: '$orig_selector'" >&2
812
exit 1
813
fi
814
if expr "z${selector#*-}" : "z[0-9]*[^0-9]" >/dev/null
815
then
814
- echo "error: $title: invalid non-numeric in range" \
816
+ echo "error: $operation: invalid non-numeric in range" \
817
"end: '$orig_selector'" >&2
818
exit 1
819
fi
@@ -819,9 +821,11 @@ match_test_selector_list () {
821
*)
822
if expr "z$selector" : "z[0-9]*[^0-9]" >/dev/null
823
then
822
- echo "error: $title: invalid non-numeric in test" \
823
- "selector: '$orig_selector'" >&2
824
- exit 1
824
+ case "$title" in *${selector}*)
825
+ include=$positive
826
+ ;;
827
+ esac
828
+ continue
829
fi
830
esac
831
@@ -1031,7 +1035,7 @@ test_skip () {
1035
skipped_reason="GIT_SKIP_TESTS"
1036
fi
1037
if test -z "$to_skip" && test -n "$run_list" &&
1034
- ! match_test_selector_list '--run' $test_count "$run_list"
1038
+ ! match_test_selector_list '--run' "$1" $test_count "$run_list"
1039
then
1040
to_skip=t
1041
skipped_reason="--run"