@matheusbsbs / matheus-bagliano / commits / 029f37217c

gitweb: apply fallback encoding before highlight

Some multi-byte character encodings (such as Shift_JIS and GBK) have characters whose final bytes is an ASCII '\' (0x5c), and they will be displayed as funny-characters even if $fallback_encoding is correct. This is because `highlight` command always expects UTF-8 encoded strings from STDIN. $ echo 'my $v = "申";' | highlight --syntax perl | w3m -T text/html -dump my $v = "申"; $ echo 'my $v = "申";' | iconv -f UTF-8 -t Shift_JIS | highlight \ --syntax perl | iconv -f Shift_JIS -t UTF-8 | w3m -T text/html -dump iconv: (stdin):9:135: cannot convert my $v = " This patch prepare git blob objects to be encoded into UTF-8 before highlighting in the manner of `to_utf8` subroutine. Signed-off-by: Shin Kojima <shin@kojima.org> Reviewed-by: Jakub Narębski <jnareb@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Shin Kojima committed May 3, 2016 at 22:00 UTC 029f37217c01fc6437357a30cd1b084b03ca872b

1 file changed +3

gitweb/gitweb.perl

@@ -3935,6 +3935,9 @@ sub run_highlighter {

3935

3936 close $fd;

3937 open $fd, quote_command(git_cmd(), "cat-file", "blob", $hash)." | ".

3938 + quote_command($^X, '-CO', '-MEncode=decode,FB_DEFAULT', '-pse',

3939 + '$_ = decode($fe, $_, FB_DEFAULT) if !utf8::decode($_);',

3940 + '--', "-fe=$fallback_encoding")." | ".

3941 quote_command($highlight_bin).

3942 " --replace-tabs=8 --fragment --syntax $syntax |"

3943 or die_error(500, "Couldn't open file or run syntax highlighter");