@matheusbsbs / matheus-bagliano / commits / b2ec8e90c2

midx: do not require packs to be sorted in lexicographic order

The MIDX file format currently requires that pack files be identified by the lexicographic ordering of their names (that is, a pack having a checksum beginning with "abc" would have a numeric pack_int_id which is smaller than the same value for a pack beginning with "bcd"). As a result, it is impossible to combine adjacent MIDX layers together without permuting bits from bitmaps that are in more recent layer(s). To see why, consider the following example: | packs | preferred pack --------+-------------+--------------- MIDX #0 | { X, Y, Z } | Y MIDX #1 | { A, B, C } | B MIDX #2 | { D, E, F } | D , where MIDX #2's base MIDX is MIDX #1, and so on. Suppose that we want to combine MIDX layers #0 and #1, to create a new layer #0' containing the packs from both layers. With the original three MIDX layers, objects are laid out in the bitmap in the order they appear in their source pack, and the packs themselves are arranged according to the pseudo-pack order. In this case, that ordering is Y, X, Z, B, A, C. But recall that the pseudo-pack ordering is defined by the order that packs appear in the MIDX, with the exception of the preferred pack, which sorts ahead of all other packs regardless of its position within the MIDX. In the above example, that means that pack 'Y' could be placed anywhere (so long as it is designated as preferred), however, all other packs must be placed in the location listed above. Because that ordering isn't sorted lexicographically, it is impossible to compact MIDX layers in the above configuration without permuting the object-to-bit-position mapping. Changing this mapping would affect all bitmaps belonging to newer layers, rendering the bitmaps associated with MIDX #2 unreadable. One of the goals of MIDX compaction is that we are able to shrink the length of the MIDX chain *without* invalidating bitmaps that belong to newer layers, and the lexicographic ordering constraint is at odds with this goal. However, packs do not *need* to be lexicographically ordered within the MIDX. As far as I can gather, the only reason they are sorted lexically is to make it possible to perform a binary search over the pack names in a MIDX, necessary to make `midx_contains_pack()`'s performance logarithmic in the number of packs rather than linear. Relax this constraint by allowing MIDX writes to proceed with packs that are not arranged in lexicographic order. `midx_contains_pack()` will lazily instantiate a `pack_names_sorted` array on the MIDX, which will be used to implement the binary search over pack names. This change produces MIDXs which may not be correctly read with external tools or older versions of Git. Though older versions of Git know how to gracefully degrade and ignore any MIDX(s) they consider corrupt, external tools may not be as robust. To avoid unintentionally breaking any such tools, guard this change behind a version bump in the MIDX's on-disk format. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Taylor Blau committed Feb 24, 2026 at 14:00 UTC b2ec8e90c201d2395b67f89f3bf38bb472e43460

5 files changed +69 -16

Documentation/gitformat-pack.adoc

+6 -2

@@ -374,7 +374,9 @@ HEADER:

374 The signature is: {'M', 'I', 'D', 'X'}

375

376 1-byte version number:

377 - Git only writes or recognizes version 1.

377 + Git writes the version specified by the "midx.version"

378 + configuration option, which defaults to 2. It recognizes

379 + both versions 1 and 2.

380

381 1-byte Object Id Version

382 We infer the length of object IDs (OIDs) from this value:

@@ -413,7 +415,9 @@ CHUNK DATA:

415 strings. There is no extra padding between the filenames,

416 and they are listed in lexicographic order. The chunk itself

417 is padded at the end with between 0 and 3 NUL bytes to make the

416 - chunk size a multiple of 4 bytes.

418 + chunk size a multiple of 4 bytes. Version 1 MIDXs are required to

419 + list their packs in lexicographic order, but version 2 MIDXs may

420 + list their packs in any arbitrary order.

421

422 Bitmapped Packfiles (ID: {'B', 'T', 'M', 'P'})

423 Stores a table of two 4-byte unsigned integers in network order.

midx-write.c

+22 -4

@@ -36,10 +36,13 @@ extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,

37 static size_t write_midx_header(const struct git_hash_algo *hash_algo,

38 struct hashfile *f, unsigned char num_chunks,

39 - uint32_t num_packs)

39 + uint32_t num_packs, int version)

40 {

41 + if (version != MIDX_VERSION_V1 && version != MIDX_VERSION_V2)

42 + BUG("unexpected MIDX version: %d", version);

43 +

44 hashwrite_be32(f, MIDX_SIGNATURE);

42 - hashwrite_u8(f, MIDX_VERSION);

45 + hashwrite_u8(f, version);

46 hashwrite_u8(f, oid_version(hash_algo));

47 hashwrite_u8(f, num_chunks);

48 hashwrite_u8(f, 0); /* unused */

@@ -105,6 +108,8 @@ struct write_midx_context {

108

109 uint32_t preferred_pack_idx;

110

111 + int version; /* must be MIDX_VERSION_V1 or _V2 */

112 +

113 int incremental;

114 uint32_t num_multi_pack_indexes_before;

115

@@ -410,7 +415,9 @@ static int write_midx_pack_names(struct hashfile *f, void *data)

415 if (ctx->info[i].expired)

416 continue;

417

413 - if (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)

418 + if (ctx->version == MIDX_VERSION_V1 &&

419 + i && strcmp(ctx->info[i].pack_name,

420 + ctx->info[i - 1].pack_name) <= 0)

421 BUG("incorrect pack-file order: %s before %s",

422 ctx->info[i - 1].pack_name,

423 ctx->info[i].pack_name);

@@ -1025,6 +1032,12 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c

1032 if (!midx_checksum_valid(midx))

1033 goto out;

1034

1035 + /*

1036 + * If the version differs, we need to update.

1037 + */

1038 + if (midx->version != ctx->version)

1039 + goto out;

1040 +

1041 /*

1042 * Ignore incremental updates for now. The assumption is that any

1043 * incremental update would be either empty (in which case we will bail

@@ -1100,6 +1113,7 @@ static int write_midx_internal(struct write_midx_opts *opts)

1113 struct tempfile *incr;

1114 struct write_midx_context ctx = {

1115 .preferred_pack_idx = NO_PREFERRED_PACK,

1116 + .version = MIDX_VERSION_V2,

1117 };

1118 struct multi_pack_index *midx_to_free = NULL;

1119 int bitmapped_packs_concat_len = 0;

@@ -1114,6 +1128,10 @@ static int write_midx_internal(struct write_midx_opts *opts)

1128 ctx.repo = r;

1129 ctx.source = opts->source;

1130

1131 + repo_config_get_int(ctx.repo, "midx.version", &ctx.version);

1132 + if (ctx.version != MIDX_VERSION_V1 && ctx.version != MIDX_VERSION_V2)

1133 + die(_("unknown MIDX version: %d"), ctx.version);

1134 +

1135 ctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);

1136

1137 if (ctx.incremental)

@@ -1445,7 +1463,7 @@ static int write_midx_internal(struct write_midx_opts *opts)

1463 }

1464

1465 write_midx_header(r->hash_algo, f, get_num_chunks(cf),

1448 - ctx.nr - dropped_packs);

1466 + ctx.nr - dropped_packs, ctx.version);

1467 write_chunkfile(cf, &ctx);

1468

1469 finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,

midx.c

+28 -3

@@ -149,7 +149,7 @@ static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou

149 m->signature, MIDX_SIGNATURE);

150

151 m->version = m->data[MIDX_BYTE_FILE_VERSION];

152 - if (m->version != MIDX_VERSION)

152 + if (m->version != MIDX_VERSION_V1 && m->version != MIDX_VERSION_V2)

153 die(_("multi-pack-index version %d not recognized"),

154 m->version);

155

@@ -210,7 +210,8 @@ static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou

210 die(_("multi-pack-index pack-name chunk is too short"));

211 cur_pack_name = end + 1;

212

213 - if (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)

213 + if (m->version == MIDX_VERSION_V1 &&

214 + i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)

215 die(_("multi-pack-index pack names out of order: '%s' before '%s'"),

216 m->pack_names[i - 1],

217 m->pack_names[i]);

@@ -411,6 +412,7 @@ void close_midx(struct multi_pack_index *m)

412 }

413 FREE_AND_NULL(m->packs);

414 FREE_AND_NULL(m->pack_names);

415 + FREE_AND_NULL(m->pack_names_sorted);

416 free(m);

417 }

418

@@ -655,17 +657,40 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,

657 return strcmp(idx_or_pack_name, idx_name);

658 }

659

660 +

661 +static int midx_pack_names_cmp(const void *a, const void *b, void *m_)

662 +{

663 + struct multi_pack_index *m = m_;

664 + return strcmp(m->pack_names[*(const size_t *)a],

665 + m->pack_names[*(const size_t *)b]);

666 +}

667 +

668 static int midx_contains_pack_1(struct multi_pack_index *m,

669 const char *idx_or_pack_name)

670 {

671 uint32_t first = 0, last = m->num_packs;

672

673 + if (m->version == MIDX_VERSION_V2 && !m->pack_names_sorted) {

674 + uint32_t i;

675 +

676 + ALLOC_ARRAY(m->pack_names_sorted, m->num_packs);

677 +

678 + for (i = 0; i < m->num_packs; i++)

679 + m->pack_names_sorted[i] = i;

680 +

681 + QSORT_S(m->pack_names_sorted, m->num_packs, midx_pack_names_cmp,

682 + m);

683 + }

684 +

685 while (first < last) {

686 uint32_t mid = first + (last - first) / 2;

687 const char *current;

688 int cmp;

689

668 - current = m->pack_names[mid];

690 + if (m->pack_names_sorted)

691 + current = m->pack_names[m->pack_names_sorted[mid]];

692 + else

693 + current = m->pack_names[mid];

694 cmp = cmp_idx_or_pack_name(idx_or_pack_name, current);

695 if (!cmp)

696 return 1;

midx.h

+3 -1

@@ -11,7 +11,8 @@ struct git_hash_algo;

11 struct odb_source;

13 #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */

14 -#define MIDX_VERSION 1

14 +#define MIDX_VERSION_V1 1

15 +#define MIDX_VERSION_V2 2

16 #define MIDX_BYTE_FILE_VERSION 4

17 #define MIDX_BYTE_HASH_VERSION 5

18 #define MIDX_BYTE_NUM_CHUNKS 6

@@ -71,6 +72,7 @@ struct multi_pack_index {

72 uint32_t num_packs_in_base;

74 const char **pack_names;

75 + size_t *pack_names_sorted;

76 struct packed_git **packs;

77 };

t/t5319-multi-pack-index.sh

+10 -6

@@ -21,7 +21,7 @@ midx_read_expect () {

21 EXTRA_CHUNKS="$5"

22 {

23 cat <<-EOF &&

24 - header: 4d494458 1 $HASH_LEN $NUM_CHUNKS $NUM_PACKS

24 + header: 4d494458 2 $HASH_LEN $NUM_CHUNKS $NUM_PACKS

25 chunks: pack-names oid-fanout oid-lookup object-offsets$EXTRA_CHUNKS

26 num_objects: $NUM_OBJECTS

27 packs:

@@ -512,11 +512,6 @@ test_expect_success 'verify invalid chunk offset' '

512 "improper chunk offset(s)"

513 '

514

515 -test_expect_success 'verify packnames out of order' '

516 - corrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER "z" $objdir \

517 - "pack names out of order"

518 -'

519 -

515 test_expect_success 'verify missing pack' '

516 corrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER "a" $objdir \

517 "failed to load pack"

@@ -578,6 +573,15 @@ test_expect_success 'verify incorrect checksum' '

573 $objdir "incorrect checksum"

574 '

575

576 +test_expect_success 'setup for v1-specific fsck tests' '

577 + git -c midx.version=1 multi-pack-index write

578 +'

579 +

580 +test_expect_success 'verify packnames out of order (v1)' '

581 + corrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER "z" $objdir \

582 + "pack names out of order"

583 +'

584 +

585 test_expect_success 'repack progress off for redirected stderr' '

586 GIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir repack 2>err &&

587 test_line_count = 0 err