@matheusbsbs / matheus-bagliano / commits / 5099b99d25
Fork 0

test-lib: Adjust output to be valid TAP format

TAP, the Test Anything Protocol, is a simple text-based interface between testing modules in a test harness. test-lib.sh's output was already very close to being valid TAP. This change brings it all the way there. Before: $ ./t0005-signals.sh * ok 1: sigchain works * passed all 1 test(s) And after: $ ./t0005-signals.sh ok 1 - sigchain works # passed all 1 test(s) 1..1 The advantage of using TAP is that any program that reads the format (a "test harness") can run the tests. The most popular of these is the prove(1) utility that comes with Perl. It can run tests in parallel, display colored output, format the output to console, file, HTML etc., and much more. An example: $ prove ./t0005-signals.sh ./t0005-signals.sh .. ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.01 cusr 0.02 csys = 0.06 CPU) Result: PASS prove(1) gives you human readable output without being too verbose. Running the test suite in parallel with `make test -j15` produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh` makes it easy to follow what's going on. All this patch does is re-arrange the output a bit so that it conforms with the TAP spec, everything that the test suite did before continues to work. That includes aggregating results in t/test-results/, the --verbose, --debug and other options for tests, and the test color output. TAP harnesses ignore everything that they don't know about, so running the tests with --verbose works: $ prove ./t0005-signals.sh :: --verbose --debug ./t0005-signals.sh .. Terminated ./t0005-signals.sh .. ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.01 cusr 0.01 csys = 0.05 CPU) Result: PASS Just supply the -v option to prove itself to get all the verbose output that it suppresses: $ prove -v ./t0005-signals.sh :: --verbose --debug ./t0005-signals.sh .. Initialized empty Git repository in /home/avar/g/git/t/trash directory.t0005-signals/.git/ expecting success: test-sigchain >actual case "$?" in 143) true ;; # POSIX w/ SIGTERM=15 3) true ;; # Windows *) false ;; esac && test_cmp expect actual Terminated ok 1 - sigchain works # passed all 1 test(s) 1..1 ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.04 CPU) Result: PASS As a further example, consider this test script that uses a lot of test-lib.sh features by Jakub Narebski: #!/bin/sh test_description='this is a sample test. This test is here to see various test outputs.' . ./test-lib.sh say 'diagnostic message' test_expect_success 'true test' 'true' test_expect_success 'false test' 'false' test_expect_failure 'true test (todo)' 'true' test_expect_failure 'false test (todo)' 'false' test_debug 'echo "debug message"' test_done The output of that was previously: * diagnostic message # yellow * ok 1: true test * FAIL 2: false test # bold red false * FIXED 3: true test (todo) * still broken 4: false test (todo) # bold green * fixed 1 known breakage(s) # green * still have 1 known breakage(s) # bold red * failed 1 among remaining 3 test(s) # bold red But is now: diagnostic message # yellow ok 1 - true test not ok - 2 false test # bold red # false ok 3 - true test (todo) # TODO known breakage not ok 4 - false test (todo) # TODO known breakage # bold green # fixed 1 known breakage(s) # green # still have 1 known breakage(s) # bold red # failed 1 among remaining 3 test(s) # bold red 1..4 All the coloring is preserved when the test is run manually. Under prove(1) the test performs as expected, even with --debug and --verbose options: $ prove ./example.sh :: --debug --verbose ./example.sh .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/4 subtests (1 TODO test unexpectedly succeeded) Test Summary Report ------------------- ./example.sh (Wstat: 256 Tests: 4 Failed: 1) Failed test: 2 TODO passed: 3 Non-zero exit status: 1 Files=1, Tests=4, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.01 csys = 0.03 CPU) Result: FAIL The TAP harness itself doesn't get confused by the color output, they aren't used by test-lib.sh stdout isn't open to a terminal (test -t 1). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Ævar Arnfjörð Bjarmason committed Jun 24, 2010 at 21:52 UTC 5099b99d25f134c558edc356ad20209da679dd3b
2 files changed +59 -30
t/README
+41 -18
@@ -18,25 +18,48 @@ The easiest way to run tests is to say "make". This runs all
18 the tests.
19
20 *** t0000-basic.sh ***
21 - * ok 1: .git/objects should be empty after git-init in an empty repo.
22 - * ok 2: .git/objects should have 256 subdirectories.
23 - * ok 3: git-update-index without --add should fail adding.
21 + ok 1 - .git/objects should be empty after git init in an empty repo.
22 + ok 2 - .git/objects should have 3 subdirectories.
23 + ok 3 - success is reported like this
24 ...
25 - * ok 23: no diff after checkout and git-update-index --refresh.
26 - * passed all 23 test(s)
27 - *** t0100-environment-names.sh ***
28 - * ok 1: using old names should issue warnings.
29 - * ok 2: using old names but having new names should not issue warnings.
30 - ...
31 -
32 -Or you can run each test individually from command line, like
33 -this:
34 -
35 - $ sh ./t3001-ls-files-killed.sh
36 - * ok 1: git-update-index --add to add various paths.
37 - * ok 2: git-ls-files -k to show killed files.
38 - * ok 3: validate git-ls-files -k output.
39 - * passed all 3 test(s)
25 + ok 43 - very long name in the index handled sanely
26 + # fixed 1 known breakage(s)
27 + # still have 1 known breakage(s)
28 + # passed all remaining 42 test(s)
29 + 1..43
30 + *** t0001-init.sh ***
31 + ok 1 - plain
32 + ok 2 - plain with GIT_WORK_TREE
33 + ok 3 - plain bare
34 +
35 +Since the tests all output TAP (see http://testanything.org) they can
36 +be run with any TAP harness. Here's an example of paralell testing
37 +powered by a recent version of prove(1):
38 +
39 + $ prove --timer --jobs 15 ./t[0-9]*.sh
40 + [19:17:33] ./t0005-signals.sh ................................... ok 36 ms
41 + [19:17:33] ./t0022-crlf-rename.sh ............................... ok 69 ms
42 + [19:17:33] ./t0024-crlf-archive.sh .............................. ok 154 ms
43 + [19:17:33] ./t0004-unwritable.sh ................................ ok 289 ms
44 + [19:17:33] ./t0002-gitfile.sh ................................... ok 480 ms
45 + ===( 102;0 25/? 6/? 5/? 16/? 1/? 4/? 2/? 1/? 3/? 1... )===
46 +
47 +prove and other harnesses come with a lot of useful options. The
48 +--state option in particular is very useful:
49 +
50 + # Repeat until no more failures
51 + $ prove -j 15 --state=failed,save ./t[0-9]*.sh
52 +
53 +You can also run each test individually from command line, like this:
54 +
55 + $ sh ./t3010-ls-files-killed-modified.sh
56 + ok 1 - git update-index --add to add various paths.
57 + ok 2 - git ls-files -k to show killed files.
58 + ok 3 - validate git ls-files -k output.
59 + ok 4 - git ls-files -m to show modified files.
60 + ok 5 - validate git ls-files -m output.
61 + # passed all 5 test(s)
62 + 1..5
63
64 You can pass --verbose (or -v), --debug (or -d), and --immediate
65 (or -i) command line argument to the test, or by setting GIT_TEST_OPTS
t/test-lib.sh
+18 -12
@@ -160,7 +160,7 @@ if test -n "$color"; then
160 *) test -n "$quiet" && return;;
161 esac
162 shift
163 - printf "* %s" "$*"
163 + printf "%s" "$*"
164 tput sgr0
165 echo
166 )
@@ -169,7 +169,7 @@ else
169 say_color() {
170 test -z "$1" && test -n "$quiet" && return
171 shift
172 - echo "* $*"
172 + echo "$*"
173 }
174 fi
175
@@ -339,25 +339,25 @@ test_have_prereq () {
339
340 test_ok_ () {
341 test_success=$(($test_success + 1))
342 - say_color "" " ok $test_count: $@"
342 + say_color "" "ok $test_count - $@"
343 }
344
345 test_failure_ () {
346 test_failure=$(($test_failure + 1))
347 - say_color error "FAIL $test_count: $1"
347 + say_color error "not ok - $test_count $1"
348 shift
349 - echo "$@" | sed -e 's/^/ /'
349 + echo "$@" | sed -e 's/^/# /'
350 test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
351 }
352
353 test_known_broken_ok_ () {
354 test_fixed=$(($test_fixed+1))
355 - say_color "" " FIXED $test_count: $@"
355 + say_color "" "ok $test_count - $@ # TODO known breakage"
356 }
357
358 test_known_broken_failure_ () {
359 test_broken=$(($test_broken+1))
360 - say_color skip " still broken $test_count: $@"
360 + say_color skip "not ok $test_count - $@ # TODO known breakage"
361 }
362
363 test_debug () {
@@ -390,7 +390,7 @@ test_skip () {
390 case "$to_skip" in
391 t)
392 say_color skip >&3 "skipping test: $@"
393 - say_color skip "skip $test_count: $1"
393 + say_color skip "ok $test_count: # skip $1"
394 : true
395 ;;
396 *)
@@ -620,18 +620,22 @@ test_done () {
620
621 if test "$test_fixed" != 0
622 then
623 - say_color pass "fixed $test_fixed known breakage(s)"
623 + say_color pass "# fixed $test_fixed known breakage(s)"
624 fi
625 if test "$test_broken" != 0
626 then
627 - say_color error "still have $test_broken known breakage(s)"
627 + say_color error "# still have $test_broken known breakage(s)"
628 msg="remaining $(($test_count-$test_broken)) test(s)"
629 else
630 msg="$test_count test(s)"
631 fi
632 case "$test_failure" in
633 0)
634 - say_color pass "passed all $msg"
634 + # Maybe print SKIP message
635 + [ -z "$skip_all" ] || skip_all=" # SKIP $skip_all"
636 +
637 + say_color pass "# passed all $msg"
638 + say "1..$test_count$skip_all"
639
640 test -d "$remove_trash" &&
641 cd "$(dirname "$remove_trash")" &&
@@ -640,7 +644,9 @@ test_done () {
644 exit 0 ;;
645
646 *)
643 - say_color error "failed $test_failure among $msg"
647 + say_color error "# failed $test_failure among $msg"
648 + say "1..$test_count"
649 +
650 exit 1 ;;
651
652 esac