midx: disable replace objects

We observed a series of clone failures arose in a specific set of repositories after we fully enabled the MIDX bitmap feature within our Codebase service. These failures were accompanied with error messages such as: Cloning into bare repository 'clone.git'... remote: Enumerating objects: 8, done. remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1) Receiving objects: 100% (8/8), done. fatal: did not receive expected object ... fatal: fetch-pack: invalid index-pack output Temporarily disabling the MIDX feature eliminated the reported issues. After some investigation we found that all repositories experiencing failures contain replace references, which seem to be improperly acknowledged by the MIDX bitmap generation logic. A more thorough explanation about the root cause from Taylor Blau says: Indeed, the pack-bitmap-write machinery does not itself call disable_replace_refs(). So when it generates a reachability bitmap, it is doing so with the replace refs in mind. You can see that this is indeed the cause of the problem by looking at the output of an instrumented version of Git that indicates what bits are being set during the bitmap generation phase. With replace refs (incorrectly) enabled, we get: [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8] and doing the same after calling disable_replace_refs(), we instead get: [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8] Single pack bitmaps are unaffected by this issue because we generate them from within pack-objects, which does call disable_replace_refs(). This patch updates the MIDX logic to disable replace objects within the multi-pack-index builtin, and a test showing a clone (which would fail with MIDX bitmap) is added to demonstrate the bug. Helped-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Xing Xin <xingxin.xx@bytedance.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Xing Xin committed Apr 8, 2024 at 05:26 UTC 93e2ae1c95e2685f51fb6320508bbde20fa7949f
2 files changed +24
builtin/multi-pack-index.c
+3
@@ -8,6 +8,7 @@
8 #include "strbuf.h"
9 #include "trace2.h"
10 #include "object-store-ll.h"
11 +#include "replace-object.h"
12
13 #define BUILTIN_MIDX_WRITE_USAGE \
14 N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]" \
@@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv,
274 };
275 struct option *options = parse_options_concat(builtin_multi_pack_index_options, common_opts);
276
277 + disable_replace_refs();
278 +
279 git_config(git_default_config, NULL);
280
281 if (the_repository &&
t/t5326-multi-pack-bitmaps.sh
+21
@@ -434,6 +434,27 @@ test_expect_success 'tagged commits are selected for bitmapping' '
434 )
435 '
436
437 +test_expect_success 'do not follow replace objects for MIDX bitmap' '
438 + rm -fr repo &&
439 + git init repo &&
440 + test_when_finished "rm -fr repo" &&
441 + (
442 + cd repo &&
443 +
444 + test_commit A &&
445 + test_commit B &&
446 + git checkout --orphan=orphan A &&
447 + test_commit orphan &&
448 +
449 + git replace A HEAD &&
450 + git repack -ad --write-midx --write-bitmap-index &&
451 +
452 + # generating reachability bitmaps with replace refs
453 + # enabled will result in broken clones
454 + git clone --no-local --bare . clone.git
455 + )
456 +'
457 +
458 corrupt_file () {
459 chmod a+w "$1" &&
460 printf "bogus" | dd of="$1" bs=1 seek="12" conv=notrunc