@matheusbsbs / matheus-bagliano / commits / 3a59e5954e

Documentation/i18n.txt: clarify character encoding support

As a "distributed" VCS, git should better define the encodings of its core textual data structures, in particular those that are part of the network protocol. That git is encoding agnostic is only really true for blob objects. E.g. the 'non-NUL bytes' requirement of tree and commit objects excludes UTF-16/32, and the special meaning of '/' in the index file as well as space and linefeed in commit objects eliminates EBCDIC and other non-ASCII encodings. Git expects bytes < 0x80 to be pure ASCII, thus CJK encodings that partly overlap with the ASCII range are problematic as well. E.g. fmt_ident() removes trailing 0x5C from user names on the assumption that it is ASCII '\'. However, there are over 200 GBK double byte codes that end in 0x5C. UTF-8 as default encoding on Linux and respective path translations in the Mac and Windows versions have established UTF-8 NFC as de-facto standard for path names. Update the documentation in i18n.txt to reflect the current status-quo. Signed-off-by: Karsten Blees <blees@dcon.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Karsten Blees committed Jul 1, 2015 at 21:10 UTC 3a59e5954ef19ac94522219c2f29d49a187d31d8
1 file changed +23 -10
Documentation/i18n.txt
+23 -10
@@ -1,18 +1,31 @@
1 -At the core level, Git is character encoding agnostic.
2 -
3 - - The pathnames recorded in the index and in the tree objects
4 - are treated as uninterpreted sequences of non-NUL bytes.
5 - What readdir(2) returns are what are recorded and compared
6 - with the data Git keeps track of, which in turn are expected
7 - to be what lstat(2) and creat(2) accepts. There is no such
8 - thing as pathname encoding translation.
1 +Git is to some extent character encoding agnostic.
2
3 - The contents of the blob objects are uninterpreted sequences
4 of bytes. There is no encoding translation at the core
5 level.
6
14 - - The commit log messages are uninterpreted sequences of non-NUL
15 - bytes.
7 + - Path names are encoded in UTF-8 normalization form C. This
8 + applies to tree objects, the index file, ref names, as well as
9 + path names in command line arguments, environment variables
10 + and config files (`.git/config` (see linkgit:git-config[1]),
11 + linkgit:gitignore[5], linkgit:gitattributes[5] and
12 + linkgit:gitmodules[5]).
13 ++
14 +Note that Git at the core level treats path names simply as
15 +sequences of non-NUL bytes, there are no path name encoding
16 +conversions (except on Mac and Windows). Therefore, using
17 +non-ASCII path names will mostly work even on platforms and file
18 +systems that use legacy extended ASCII encodings. However,
19 +repositories created on such systems will not work properly on
20 +UTF-8-based systems (e.g. Linux, Mac, Windows) and vice versa.
21 +Additionally, many Git-based tools simply assume path names to
22 +be UTF-8 and will fail to display other encodings correctly.
23 +
24 + - Commit log messages are typically encoded in UTF-8, but other
25 + extended ASCII encodings are also supported. This includes
26 + ISO-8859-x, CP125x and many others, but _not_ UTF-16/32,
27 + EBCDIC and CJK multi-byte encodings (GBK, Shift-JIS, Big5,
28 + EUC-x, CP9xx etc.).
29
30 Although we encourage that the commit log messages are encoded
31 in UTF-8, both the core and Git Porcelain are designed not to