@matheusbsbs / matheus-bagliano / commits / b59d398bea
Fork 0

Do a better job at guessing unknown character sets

At least in the kernel development community, we're generally slowly converting to UTF-8 everywhere, and the old default of Latin1 in emails is being supplanted by UTF-8, and it doesn't necessarily show up as such in the mail headers (because, quite frankly, when people send patches around, they want the email client to do as little as humanly possible about the patch) Despite that, it's often the case that email addresses etc still have Latin1, so I've seen emails where this is a mixed bag, with Signed-off parts being copied from email (and containing Latin1 characters), and the rest of the email being a patch in UTF-8. So this suggests a very natural change: if the target character set is utf-8 (the default), and if the source already looks like utf-8, just assume that it doesn't need any conversion at all. Only assume that it needs conversion if it isn't already valid utf-8, in which case we (for historical reasons) will assume it's Latin1. Basically no really _valid_ latin1 will ever look like utf-8, so while this changes our historical behaviour, it doesn't do so in practice, and makes the default behaviour saner for the case where the input was already in proper format. We could do a more fancy guess, of course, but this correctly handled a series of patches I just got from Andrew that had a mixture of Latin1 and UTF-8 (in different emails, but without any character set indication). Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Linus Torvalds committed Jul 17, 2007 at 10:34 UTC b59d398beab604e577846ef8393735478c1ca3c2
1 file changed +29 -4
builtin-mailinfo.c
+29 -4
@@ -499,15 +499,40 @@ static int decode_b_segment(char *in, char *ot, char *ep)
499 return 0;
500 }
501
502 +/*
503 + * When there is no known charset, guess.
504 + *
505 + * Right now we assume that if the target is UTF-8 (the default),
506 + * and it already looks like UTF-8 (which includes US-ASCII as its
507 + * subset, of course) then that is what it is and there is nothing
508 + * to do.
509 + *
510 + * Otherwise, we default to assuming it is Latin1 for historical
511 + * reasons.
512 + */
513 +static const char *guess_charset(const char *line, const char *target_charset)
514 +{
515 + if (is_encoding_utf8(target_charset)) {
516 + if (is_utf8(line))
517 + return NULL;
518 + }
519 + return "latin1";
520 +}
521 +
522 static void convert_to_utf8(char *line, const char *charset)
523 {
504 - static const char latin_one[] = "latin1";
505 - const char *input_charset = *charset ? charset : latin_one;
506 - char *out = reencode_string(line, metainfo_charset, input_charset);
524 + char *out;
525 +
526 + if (!charset || !*charset) {
527 + charset = guess_charset(line, metainfo_charset);
528 + if (!charset)
529 + return;
530 + }
531
532 + out = reencode_string(line, metainfo_charset, charset);
533 if (!out)
534 die("cannot convert from %s to %s\n",
510 - input_charset, metainfo_charset);
535 + charset, metainfo_charset);
536 strcpy(line, out);
537 free(out);
538 }