@matheusbsbs / matheus-bagliano / commits / fbe5f6b804

git-p4: preserve utf8 BOM when importing from p4 to git

Perforce has a file type "utf8" which represents a text file with explicit BOM. utf8-encoded files *without* BOM are stored as regular file type "text". The "utf8" file type behaves like text in all but one important way: it is stored, internally, without the leading 3 BOM bytes. git-p4 has historically imported utf8-with-BOM files (files stored, in Perforce, as type "utf8") the same way as regular text files - losing the BOM in the process. Under most circumstances this issue has little functional impact, as most systems consider the BOM to be optional and redundant, but this *is* a correctness failure, and can have lead to practical issues for example when BOMs are explicitly included in test files, for example in a file encoding test suite. Fix the handling of utf8-with-BOM files when importing changes from p4 to git, and introduce a test that checks it is working correctly. Signed-off-by: Tao Klerks <tao@klerks.biz> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Tao Klerks committed Apr 4, 2022 at 05:50 UTC fbe5f6b80437adbcd58af1b3751b830910a2ddaa
2 files changed +44
git-p4.py
+10
@@ -2885,6 +2885,16 @@ class P4Sync(Command, P4UserMap):
2885 print("\nIgnoring apple filetype file %s" % file['depotFile'])
2886 return
2887
2888 + if type_base == "utf8":
2889 + # The type utf8 explicitly means utf8 *with BOM*. These are
2890 + # streamed just like regular text files, however, without
2891 + # the BOM in the stream.
2892 + # Therefore, to accurately import these files into git, we
2893 + # need to explicitly re-add the BOM before writing.
2894 + # 'contents' is a set of bytes in this case, so create the
2895 + # BOM prefix as a b'' literal.
2896 + contents = [b'\xef\xbb\xbf' + contents[0]] + contents[1:]
2897 +
2898 # Note that we do not try to de-mangle keywords on utf16 files,
2899 # even though in theory somebody may want that.
2900 regexp = p4_keywords_regexp_for_type(type_base, type_mods)
t/t9802-git-p4-filetype.sh
+34
@@ -333,4 +333,38 @@ test_expect_success SYMLINKS 'empty symlink target' '
333 )
334 '
335
336 +test_expect_success SYMLINKS 'utf-8 with and without BOM in text file' '
337 + (
338 + cd "$cli" &&
339 +
340 + # some utf8 content
341 + echo some tǣxt >utf8-nobom-test &&
342 +
343 + # same utf8 content as before but with bom
344 + echo some tǣxt | sed '\''s/^/\xef\xbb\xbf/'\'' >utf8-bom-test &&
345 +
346 + # bom only
347 + dd bs=1 count=3 if=utf8-bom-test of=utf8-bom-empty-test &&
348 +
349 + p4 add utf8-nobom-test utf8-bom-test utf8-bom-empty-test &&
350 + p4 submit -d "add utf8 test files"
351 + ) &&
352 + test_when_finished cleanup_git &&
353 +
354 + git p4 clone --dest="$git" //depot@all &&
355 + (
356 + cd "$git" &&
357 + git checkout refs/remotes/p4/master &&
358 +
359 + echo some tǣxt >utf8-nobom-check &&
360 + test_cmp utf8-nobom-check utf8-nobom-test &&
361 +
362 + echo some tǣxt | sed '\''s/^/\xef\xbb\xbf/'\'' >utf8-bom-check &&
363 + test_cmp utf8-bom-check utf8-bom-test &&
364 +
365 + dd bs=1 count=3 if=utf8-bom-check of=utf8-bom-empty-check &&
366 + test_cmp utf8-bom-empty-check utf8-bom-empty-test
367 + )
368 +'
369 +
370 test_done