@matheusbsbs / matheus-bagliano / commits / c6e48652f6

utf8: add function to detect a missing UTF-16/32 BOM

If the endianness is not defined in the encoding name, then let's be strict and require a BOM to avoid any encoding confusion. The is_missing_required_utf_bom() function returns true if a required BOM is missing. The Unicode standard instructs to assume big-endian if there in no BOM for UTF-16/32 [1][2]. However, the W3C/WHATWG encoding standard used in HTML5 recommends to assume little-endian to "deal with deployed content" [3]. Strictly requiring a BOM seems to be the safest option for content in Git. This function is used in a subsequent commit. [1] http://unicode.org/faq/utf_bom.html#gen6 [2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf Section 3.10, D98, page 132 [3] https://encoding.spec.whatwg.org/#utf-16le Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Lars Schneider committed Apr 15, 2018 at 20:16 UTC c6e48652f69f6955bbbb423100e0df2a49467db8
2 files changed +32
utf8.c
+13
@@ -586,6 +586,19 @@ int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)
586 );
587 }
588
589 +int is_missing_required_utf_bom(const char *enc, const char *data, size_t len)
590 +{
591 + return (
592 + (same_utf_encoding(enc, "UTF-16")) &&
593 + !(has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||
594 + has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))
595 + ) || (
596 + (same_utf_encoding(enc, "UTF-32")) &&
597 + !(has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||
598 + has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))
599 + );
600 +}
601 +
602 /*
603 * Returns first character length in bytes for multi-byte `text` according to
604 * `encoding`.
utf8.h
+19
@@ -79,4 +79,23 @@ void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int wid
79 */
80 int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);
81
82 +/*
83 + * If the endianness is not defined in the encoding name, then we
84 + * require a BOM. The function returns true if a required BOM is missing.
85 + *
86 + * The Unicode standard instructs to assume big-endian if there in no
87 + * BOM for UTF-16/32 [1][2]. However, the W3C/WHATWG encoding standard
88 + * used in HTML5 recommends to assume little-endian to "deal with
89 + * deployed content" [3].
90 + *
91 + * Therefore, strictly requiring a BOM seems to be the safest option for
92 + * content in Git.
93 + *
94 + * [1] http://unicode.org/faq/utf_bom.html#gen6
95 + * [2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf
96 + * Section 3.10, D98, page 132
97 + * [3] https://encoding.spec.whatwg.org/#utf-16le
98 + */
99 +int is_missing_required_utf_bom(const char *enc, const char *data, size_t len);
100 +
101 #endif