chainlint: annotate original test definition rather than token stream
When chainlint detects problems in a test, such as a broken &&-chain, it
prints out the test with "?!FOO?!" annotations inserted at each problem
location. However, rather than annotating the original test definition,
it instead dumps out a parsed token representation of the test. Since it
lacks comments, indentations, here-doc bodies, and so forth, this
tokenized representation can be difficult for the test author to digest
and relate back to the original test definition.
However, now that each parsed token carries positional information, the
location of a detected problem can be pinpointed precisely in the
original test definition. Therefore, take advantage of this information
to annotate the test definition itself rather than annotating the parsed
token stream, thus making it easier for a test author to relate a
problem back to the source.
Maintaining the positional meta-information associated with each
detected problem requires a slight change in how the problems are
managed internally. In particular, shell syntax such as:
msg="total: $(cd data; wc -w *.txt) words"
requires the lexical analyzer to recursively invoke the parser in order
to detect problems within the $(...) expression inside the double-quoted
string. In this case, the recursive parse context will detect the broken
&&-chain between the `cd` and `wc` commands, returning the token stream:
cd data ; ?!AMP?! wc -w *.txt
However, the parent parse context will see everything inside the
double-quotes as a single string token:
"total: $(cd data ; ?!AMP?! wc -w *.txt) words"
losing whatever positional information was attached to the ";" token
where the problem was detected.
One way to preserve the positional information of a detected problem in
a recursive parse context within a string would be to attach the
positional information to the annotation textually; for instance:
"total: $(cd data ; ?!AMP:21:22?! wc -w *.txt) words"
and then extract the positional information when annotating the original
test definition.
However, a cleaner and much simpler approach is to maintain the list of
detected problems separately rather than embedding the problems as
annotations directly in the parsed token stream. Not only does this
ensure that positional information within recursive parse contexts is
not lost, but it keeps the token stream free from non-token pollution,
which may simplify implementation of validations added in the future
since they won't have to handle non-token "?!FOO!?" items specially.
Finally, the chainlint self-test "expect" files need a few mechanical
adjustments now that the original test definitions are emitted rather
than the parsed token stream. In particular, the following items missing
from the historic parsed-token output are now preserved verbatim:
* indentation (and whitespace, in general)
* comments
* here-doc bodies
* here-doc tag quoting (i.e. "\EOF")
* line-splices (i.e. "\" at the end of a line)
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Eric Sunshine committedNov 8, 2022 at 19:08 UTC73c768dae9ea4838736693965b25ba34e941ac88
22 files changed+163-33
t/chainlint.pl
+24-7
index 59aa79babc..7972c5bbe6 100755--- a/t/chainlint.pl+++ b/t/chainlint.pl@@ -459,6 +459,13 @@ package TestParser; use base 'ShellParser';+sub new {+ my $class = shift @_;+ my $self = $class->SUPER::new(@_);+ $self->{problems} = [];+ return $self;+}+ sub find_non_nl { my $tokens = shift @_; my $n = shift @_;@@ -498,7 +505,7 @@ sub parse_loop_body { return @tokens if ends_with(\@tokens, [qr/^\|\|$/, "\n", qr/^echo$/, qr/^.+$/]); # flag missing "return/exit" handling explicit failure in loop body my $n = find_non_nl(\@tokens);- splice(@tokens, $n + 1, 0, ['?!LOOP?!', $tokens[$n]->[1], $tokens[$n]->[2]]);+ push(@{$self->{problems}}, ['LOOP', $tokens[$n]]); return @tokens; }@@ -511,6 +518,7 @@ my @safe_endings = ( sub accumulate { my ($self, $tokens, $cmd) = @_;+ my $problems = $self->{problems}; # no previous command to check for missing "&&" goto DONE unless @$tokens;@@ -531,13 +539,13 @@ sub accumulate { # failure explicitly), then okay for all preceding commands to be # missing "&&" if ($$cmd[0]->[0] =~ /^(?:false|return|exit)$/) {- @$tokens = grep {$_->[0] !~ /^\?!AMP\?!$/} @$tokens;+ @$problems = grep {$_->[0] ne 'AMP'} @$problems; goto DONE; } # flag missing "&&" at end of previous command my $n = find_non_nl($tokens);- splice(@$tokens, $n + 1, 0, ['?!AMP?!', $$tokens[$n]->[1], $$tokens[$n]->[2]]) unless $n < 0;+ push(@$problems, ['AMP', $tokens->[$n]]) unless $n < 0; DONE: $self->SUPER::accumulate($tokens, $cmd);@@ -594,12 +602,21 @@ sub check_test { $self->{ntests}++; my $parser = TestParser->new(\$body); my @tokens = $parser->parse();- return unless $emit_all || grep {$_->[0] =~ /\?![^?]+\?!/} @tokens;+ my $problems = $parser->{problems};+ return unless $emit_all || @$problems; my $c = main::fd_colors(1);- my $checked = join(' ', map {$_->[0]} @tokens);+ my $start = 0;+ my $checked = '';+ for (sort {$a->[1]->[2] <=> $b->[1]->[2]} @$problems) {+ my ($label, $token) = @$_;+ my $pos = $token->[2];+ $checked .= substr($body, $start, $pos - $start) . " ?!$label?! ";+ $start = $pos;+ }+ $checked .= substr($body, $start); $checked =~ s/^\n//;- $checked =~ s/^ //mg;- $checked =~ s/ $//mg;+ $checked =~ s/(\s) \?!/$1?!/mg;+ $checked =~ s/\?! (\s)/?!$1/mg; $checked =~ s/(\?![^?]+\?!)/$c->{rev}$c->{red}$1$c->{reset}/mg; $checked .= "\n" unless $checked =~ /\n$/; push(@{$self->{output}}, "$c->{blue}# chainlint: $title$c->{reset}\n$checked");
t/chainlint/block-comment.expect
+2
index d10b2eeaf2..df2beea888 100644--- a/t/chainlint/block-comment.expect+++ b/t/chainlint/block-comment.expect@@ -1,6 +1,8 @@ ( {+ # show a echo a &&+ # show b echo b } )
t/chainlint/case-comment.expect
+3
index 1e4b054bda..641c157b98 100644--- a/t/chainlint/case-comment.expect+++ b/t/chainlint/case-comment.expect@@ -1,7 +1,10 @@ ( case "$x" in+ # found foo x) foo ;;+ # found other *)+ # treat it as bar bar ;; esac
index a5810c9bdd..d65c82129a 100644--- a/t/chainlint/for-loop.expect+++ b/t/chainlint/for-loop.expect@@ -2,7 +2,9 @@ for i in a b c do echo $i ?!AMP?!- cat <<-EOF ?!LOOP?!+ cat <<-\EOF ?!LOOP?!+ bar+ EOF done ?!AMP?! for i in a b c; do echo $i &&
index ffac8f9018..134d3a14f5 100644--- a/t/chainlint/incomplete-line.expect+++ b/t/chainlint/incomplete-line.expect@@ -1,4 +1,10 @@-line 1 line 2 line 3 line 4 &&+line 1 \+line 2 \+line 3 \+line 4 && (- line 5 line 6 line 7 line 8+ line 5 \+ line 6 \+ line 7 \+ line 8 )
t/chainlint/inline-comment.expect
+2-2
index dd0dace077..6bad218530 100644--- a/t/chainlint/inline-comment.expect+++ b/t/chainlint/inline-comment.expect@@ -1,6 +1,6 @@ (- foobar &&- barfoo ?!AMP?!+ foobar && # comment 1+ barfoo ?!AMP?! # wrong position for && flibble "not a # comment" ) &&
index e3bef63f75..29b3832a98 100644--- a/t/chainlint/nested-here-doc.expect+++ b/t/chainlint/nested-here-doc.expect@@ -1,7 +1,30 @@ cat <<ARBITRARY >foop &&+naddle+fub <<EOF+ nozzle+ noodle+EOF+formp+ARBITRARY (- cat <<-INPUT_END &&- cat <<-EOT ?!AMP?!+ cat <<-\INPUT_END &&+ fish are mice+ but geese go slow+ data <<EOF+ perl is lerp+ and nothing else+ EOF+ toink+ INPUT_END++ cat <<-\EOT ?!AMP?!+ text goes here+ data <<EOF+ data goes here+ EOF+ more test here+ EOT+ foobar )
t/chainlint/nested-subshell-comment.expect
+2
index be4b27a305..9138cf386d 100644--- a/t/chainlint/nested-subshell-comment.expect+++ b/t/chainlint/nested-subshell-comment.expect@@ -2,6 +2,8 @@ foo && ( bar &&+ # bottles wobble while fiddles gobble+ # minor numbers of cows (or do they?) baz && snaff ) ?!AMP?!