@matheusbsbs / matheus-bagliano / commits / 9df44a97f1

midx: implement MIDX compaction

When managing a MIDX chain with many layers, it is convenient to combine a sequence of adjacent layers into a single layer to prevent the chain from growing too long. While it is conceptually possible to "compact" a sequence of MIDX layers together by running "git multi-pack-index write --stdin-packs", there are a few drawbacks that make this less than desirable: - Preserving the MIDX chain is impossible, since there is no way to write a MIDX layer that contains objects or packs found in an earlier MIDX layer already part of the chain. So callers would have to write an entirely new (non-incremental) MIDX containing only the compacted layers, discarding all other objects/packs from the MIDX. - There is (currently) no way to write a MIDX layer outside of the MIDX chain to work around the above, such that the MIDX chain could be reassembled substituting the compacted layers with the MIDX that was written. - The `--stdin-packs` command-line option does not allow us to specify the order of packs as they appear in the MIDX. Therefore, even if there were workarounds for the previous two challenges, any bitmaps belonging to layers which come after the compacted layer(s) would no longer be valid. This commit introduces a way to compact a sequence of adjacent MIDX layers into a single layer while preserving the MIDX chain, as well as any bitmap(s) in layers which are newer than the compacted ones. Implementing MIDX compaction does not require a significant number of changes to how MIDX layers are written. The main changes are as follows: - Instead of calling `fill_packs_from_midx()`, we call a new function `fill_packs_from_midx_range()`, which walks backwards along the portion of the MIDX chain which we are compacting, and adds packs one layer a time. In order to preserve the pseudo-pack order, the concatenated pack order is preserved, with the exception of preferred packs which are always added first. - After adding entries from the set of packs in the compaction range, `compute_sorted_entries()` must adjust the `pack_int_id`'s for all objects added in each fanout layer to match their original `pack_int_id`'s (as opposed to the index at which each pack appears in `ctx.info`). Note that we cannot reuse `midx_fanout_add_midx_fanout()` directly here, as it unconditionally recurs through the `->base_midx`. Factor out a `_1()` variant that operates on a single layer, reimplement the existing function in terms of it, and use the new variant from `midx_fanout_add_compact()`. Since we are sorting the list of objects ourselves, the order we add them in does not matter. - When writing out the new 'multi-pack-index-chain' file, discard any layers in the compaction range, replacing them with the newly written layer, instead of keeping them and placing the new layer at the end of the chain. This ends up being sufficient to implement MIDX compaction in such a way that preserves bitmaps corresponding to more recent layers in the MIDX chain. The tests for MIDX compaction are so far fairly spartan, since the main interesting behavior here is ensuring that the right packs/objects are selected from each layer, and that the pack order is preserved despite whether or not they are sorted in lexicographic order in the original MIDX chain. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Taylor Blau committed Feb 24, 2026 at 14:00 UTC 9df44a97f165bbdd8d591c6e99c8894ae036b5bd

6 files changed +518 -27

Documentation/git-multi-pack-index.adoc

+13

@@ -12,6 +12,8 @@ SYNOPSIS

12 'git multi-pack-index' [<options>] write [--preferred-pack=<pack>]

13 [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]

14 [--refs-snapshot=<path>]

15 +'git multi-pack-index' [<options>] compact [--[no-]incremental]

16 + <from> <to>

17 'git multi-pack-index' [<options>] verify

18 'git multi-pack-index' [<options>] expire

19 'git multi-pack-index' [<options>] repack [--batch-size=<size>]

@@ -83,6 +85,17 @@ marker).

85 necessary.

86 --

88 +compact::

89 + Write a new MIDX layer containing only objects and packs present

90 + in the range `<from>` to `<to>`, where both arguments are

91 + checksums of existing layers in the MIDX chain.

92 ++

93 +--

94 + --incremental::

95 + Write the result to a MIDX chain instead of writing a

96 + stand-alone MIDX.

97 +--

98 +

99 verify::

100 Verify the contents of the MIDX file.

101

builtin/multi-pack-index.c

+74

@@ -17,6 +17,10 @@

17 " [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n" \

18 " [--refs-snapshot=<path>]")

20 +#define BUILTIN_MIDX_COMPACT_USAGE \

21 + N_("git multi-pack-index [<options>] compact [--[no-]incremental]\n" \

22 + " <from> <to>")

23 +

24 #define BUILTIN_MIDX_VERIFY_USAGE \

25 N_("git multi-pack-index [<options>] verify")

@@ -30,6 +34,10 @@ static char const * const builtin_multi_pack_index_write_usage[] = {

34 BUILTIN_MIDX_WRITE_USAGE,

35 NULL

36 };

37 +static char const * const builtin_multi_pack_index_compact_usage[] = {

38 + BUILTIN_MIDX_COMPACT_USAGE,

39 + NULL

40 +};

41 static char const * const builtin_multi_pack_index_verify_usage[] = {

42 BUILTIN_MIDX_VERIFY_USAGE,

43 NULL

@@ -44,6 +52,7 @@ static char const * const builtin_multi_pack_index_repack_usage[] = {

52 };

53 static char const * const builtin_multi_pack_index_usage[] = {

54 BUILTIN_MIDX_WRITE_USAGE,

55 + BUILTIN_MIDX_COMPACT_USAGE,

56 BUILTIN_MIDX_VERIFY_USAGE,

57 BUILTIN_MIDX_EXPIRE_USAGE,

58 BUILTIN_MIDX_REPACK_USAGE,

@@ -195,6 +204,70 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,

204 return ret;

205 }

206

207 +static int cmd_multi_pack_index_compact(int argc, const char **argv,

208 + const char *prefix,

209 + struct repository *repo)

210 +{

211 + struct multi_pack_index *m, *cur;

212 + struct multi_pack_index *from_midx = NULL;

213 + struct multi_pack_index *to_midx = NULL;

214 + struct odb_source *source;

215 + int ret;

216 +

217 + struct option *options;

218 + static struct option builtin_multi_pack_index_compact_options[] = {

219 + OPT_BIT(0, "incremental", &opts.flags,

220 + N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL),

221 + OPT_END(),

222 + };

223 +

224 + repo_config(repo, git_multi_pack_index_write_config, NULL);

225 +

226 + options = add_common_options(builtin_multi_pack_index_compact_options);

227 +

228 + trace2_cmd_mode(argv[0]);

229 +

230 + if (isatty(2))

231 + opts.flags |= MIDX_PROGRESS;

232 + argc = parse_options(argc, argv, prefix,

233 + options, builtin_multi_pack_index_compact_usage,

234 + 0);

235 +

236 + if (argc != 2)

237 + usage_with_options(builtin_multi_pack_index_compact_usage,

238 + options);

239 + source = handle_object_dir_option(the_repository);

240 +

241 + FREE_AND_NULL(options);

242 +

243 + m = get_multi_pack_index(source);

244 +

245 + for (cur = m; cur && !(from_midx && to_midx); cur = cur->base_midx) {

246 + const char *midx_csum = midx_get_checksum_hex(cur);

247 +

248 + if (!from_midx && !strcmp(midx_csum, argv[0]))

249 + from_midx = cur;

250 + if (!to_midx && !strcmp(midx_csum, argv[1]))

251 + to_midx = cur;

252 + }

253 +

254 + if (!from_midx)

255 + die(_("could not find MIDX: %s"), argv[0]);

256 + if (!to_midx)

257 + die(_("could not find MIDX: %s"), argv[1]);

258 + if (from_midx == to_midx)

259 + die(_("MIDX compaction endpoints must be unique"));

260 +

261 + for (m = from_midx; m; m = m->base_midx) {

262 + if (m == to_midx)

263 + die(_("MIDX %s must be an ancestor of %s"), argv[0], argv[1]);

264 + }

265 +

266 + ret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);

267 +

268 + return ret;

269 +}

270 +

271 static int cmd_multi_pack_index_verify(int argc, const char **argv,

272 const char *prefix,

273 struct repository *repo UNUSED)

@@ -295,6 +368,7 @@ int cmd_multi_pack_index(int argc,

368 struct option builtin_multi_pack_index_options[] = {

369 OPT_SUBCOMMAND("repack", &fn, cmd_multi_pack_index_repack),

370 OPT_SUBCOMMAND("write", &fn, cmd_multi_pack_index_write),

371 + OPT_SUBCOMMAND("compact", &fn, cmd_multi_pack_index_compact),

372 OPT_SUBCOMMAND("verify", &fn, cmd_multi_pack_index_verify),

373 OPT_SUBCOMMAND("expire", &fn, cmd_multi_pack_index_expire),

374 OPT_END(),

midx-write.c

+250 -27

@@ -113,6 +113,10 @@ struct write_midx_context {

113 int incremental;

114 uint32_t num_multi_pack_indexes_before;

115

116 + struct multi_pack_index *compact_from;

117 + struct multi_pack_index *compact_to;

118 + int compact;

119 +

120 struct string_list *to_include;

121

122 struct repository *repo;

@@ -122,6 +126,8 @@ struct write_midx_context {

126 static uint32_t midx_pack_perm(struct write_midx_context *ctx,

127 uint32_t orig_pack_int_id)

128 {

129 + if (ctx->compact)

130 + orig_pack_int_id -= ctx->compact_from->num_packs_in_base;

131 return ctx->pack_perm[orig_pack_int_id];

132 }

133

@@ -268,18 +274,14 @@ static void midx_fanout_sort(struct midx_fanout *fanout)

274 QSORT(fanout->entries, fanout->nr, midx_oid_compare);

275 }

276

271 -static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,

272 - struct multi_pack_index *m,

273 - uint32_t cur_fanout,

274 - uint32_t preferred_pack)

277 +static void midx_fanout_add_midx_fanout_1(struct midx_fanout *fanout,

278 + struct multi_pack_index *m,

279 + uint32_t cur_fanout,

280 + uint32_t preferred_pack)

281 {

282 uint32_t start = m->num_objects_in_base, end;

283 uint32_t cur_object;

284

279 - if (m->base_midx)

280 - midx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,

281 - preferred_pack);

282 -

285 if (cur_fanout)

286 start += ntohl(m->chunk_oid_fanout[cur_fanout - 1]);

287 end = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);

@@ -303,6 +305,17 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,

305 }

306 }

307

308 +static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,

309 + struct multi_pack_index *m,

310 + uint32_t cur_fanout,

311 + uint32_t preferred_pack)

312 +{

313 + if (m->base_midx)

314 + midx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,

315 + preferred_pack);

316 + midx_fanout_add_midx_fanout_1(fanout, m, cur_fanout, preferred_pack);

317 +}

318 +

319 static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,

320 struct pack_info *info,

321 uint32_t cur_pack,

@@ -352,6 +365,21 @@ static void midx_fanout_add(struct midx_fanout *fanout,

365 cur_fanout);

366 }

367

368 +static void midx_fanout_add_compact(struct midx_fanout *fanout,

369 + struct write_midx_context *ctx,

370 + uint32_t cur_fanout)

371 +{

372 + struct multi_pack_index *m = ctx->compact_to;

373 +

374 + ASSERT(ctx->compact);

375 +

376 + while (m && m != ctx->compact_from->base_midx) {

377 + midx_fanout_add_midx_fanout_1(fanout, m, cur_fanout,

378 + NO_PREFERRED_PACK);

379 + m = m->base_midx;

380 + }

381 +}

382 +

383 /*

384 * It is possible to artificially get into a state where there are many

385 * duplicate copies of objects. That can create high memory pressure if

@@ -370,6 +398,9 @@ static void compute_sorted_entries(struct write_midx_context *ctx,

398 size_t alloc_objects, total_objects = 0;

399 struct midx_fanout fanout = { 0 };

400

401 + if (ctx->compact)

402 + ASSERT(!start_pack);

403 +

404 for (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++)

405 total_objects = st_add(total_objects,

406 ctx->info[cur_pack].p->num_objects);

@@ -388,7 +419,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx,

419 for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {

420 fanout.nr = 0;

421

391 - midx_fanout_add(&fanout, ctx, start_pack, cur_fanout);

422 + if (ctx->compact)

423 + midx_fanout_add_compact(&fanout, ctx, cur_fanout);

424 + else

425 + midx_fanout_add(&fanout, ctx, start_pack, cur_fanout);

426 midx_fanout_sort(&fanout);

427

428 /*

@@ -956,6 +990,75 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)

990 return 0;

991 }

992

993 +static uint32_t compactible_packs_between(const struct multi_pack_index *from,

994 + const struct multi_pack_index *to)

995 +{

996 + uint32_t nr;

997 +

998 + ASSERT(from && to);

999 +

1000 + if (unsigned_add_overflows(to->num_packs, to->num_packs_in_base))

1001 + die(_("too many packs, unable to compact"));

1002 +

1003 + nr = to->num_packs + to->num_packs_in_base;

1004 + if (nr < from->num_packs_in_base)

1005 + BUG("unexpected number of packs in base during compaction: "

1006 + "%"PRIu32" < %"PRIu32, nr, from->num_packs_in_base);

1007 +

1008 + return nr - from->num_packs_in_base;

1009 +}

1010 +

1011 +static int fill_packs_from_midx_range(struct write_midx_context *ctx,

1012 + int bitmap_order)

1013 +{

1014 + struct multi_pack_index *m = ctx->compact_to;

1015 + uint32_t packs_nr;

1016 +

1017 + ASSERT(ctx->compact && !ctx->nr);

1018 + ASSERT(ctx->compact_from);

1019 + ASSERT(ctx->compact_to);

1020 +

1021 + packs_nr = compactible_packs_between(ctx->compact_from,

1022 + ctx->compact_to);

1023 +

1024 + ALLOC_GROW(ctx->info, packs_nr, ctx->alloc);

1025 +

1026 + while (m != ctx->compact_from->base_midx) {

1027 + uint32_t pack_int_id, preferred_pack_id;

1028 + uint32_t i;

1029 +

1030 + if (bitmap_order) {

1031 + if (midx_preferred_pack(m, &preferred_pack_id) < 0)

1032 + die(_("could not determine preferred pack"));

1033 + } else {

1034 + preferred_pack_id = m->num_packs_in_base;

1035 + }

1036 +

1037 + pack_int_id = m->num_packs_in_base - ctx->compact_from->num_packs_in_base;

1038 +

1039 + if (fill_pack_from_midx(&ctx->info[pack_int_id++], m,

1040 + preferred_pack_id) < 0)

1041 + return -1;

1042 +

1043 + for (i = m->num_packs_in_base;

1044 + i < m->num_packs_in_base + m->num_packs; i++) {

1045 + if (preferred_pack_id == i)

1046 + continue;

1047 +

1048 + if (fill_pack_from_midx(&ctx->info[pack_int_id++], m,

1049 + i) < 0)

1050 + return -1;

1051 + }

1052 +

1053 + ctx->nr += m->num_packs;

1054 + m = m->base_midx;

1055 + }

1056 +

1057 + ASSERT(ctx->nr == packs_nr);

1058 +

1059 + return 0;

1060 +}

1061 +

1062 static struct {

1063 const char *non_split;

1064 const char *split;

@@ -1075,6 +1178,9 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c

1178 if (ctx->incremental)

1179 goto out;

1180

1181 + if (ctx->compact)

1182 + goto out; /* Compaction always requires an update. */

1183 +

1184 /*

1185 * Otherwise, we need to verify that the packs covered by the existing

1186 * MIDX match the packs that we already have. The logic to do so is way

@@ -1120,12 +1226,23 @@ out:

1226 return needed;

1227 }

1228

1229 +static int midx_hashcmp(const struct multi_pack_index *a,

1230 + const struct multi_pack_index *b,

1231 + const struct git_hash_algo *algop)

1232 +{

1233 + return hashcmp(midx_get_checksum_hash(a), midx_get_checksum_hash(b),

1234 + algop);

1235 +}

1236 +

1237 struct write_midx_opts {

1238 struct odb_source *source; /* non-optional */

1239

1240 struct string_list *packs_to_include;

1241 struct string_list *packs_to_drop;

1242

1243 + struct multi_pack_index *compact_from;

1244 + struct multi_pack_index *compact_to;

1245 +

1246 const char *preferred_pack_name;

1247 const char *refs_snapshot;

1248 unsigned flags;

@@ -1150,6 +1267,7 @@ static int write_midx_internal(struct write_midx_opts *opts)

1267 int dropped_packs = 0;

1268 int result = -1;

1269 const char **keep_hashes = NULL;

1270 + size_t keep_hashes_nr = 0;

1271 struct chunkfile *cf;

1272

1273 trace2_region_enter("midx", "write_midx_internal", r);

@@ -1162,6 +1280,19 @@ static int write_midx_internal(struct write_midx_opts *opts)

1280 die(_("unknown MIDX version: %d"), ctx.version);

1281

1282 ctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);

1283 + ctx.compact = !!(opts->flags & MIDX_WRITE_COMPACT);

1284 +

1285 + if (ctx.compact) {

1286 + if (ctx.version != MIDX_VERSION_V2)

1287 + die(_("cannot perform MIDX compaction with v1 format"));

1288 + if (!opts->compact_from)

1289 + BUG("expected non-NULL 'from' MIDX during compaction");

1290 + if (!opts->compact_to)

1291 + BUG("expected non-NULL 'to' MIDX during compaction");

1292 +

1293 + ctx.compact_from = opts->compact_from;

1294 + ctx.compact_to = opts->compact_to;

1295 + }

1296

1297 if (ctx.incremental)

1298 strbuf_addf(&midx_name,

@@ -1189,11 +1320,18 @@ static int write_midx_internal(struct write_midx_opts *opts)

1320 */

1321 if (ctx.incremental)

1322 ctx.base_midx = m;

1192 - else if (!opts->packs_to_include)

1323 + if (!opts->packs_to_include)

1324 ctx.m = m;

1325 }

1326 }

1327

1328 + /*

1329 + * If compacting MIDX layer(s) in the range [from, to], then the

1330 + * compacted MIDX will share the same base MIDX as 'from'.

1331 + */

1332 + if (ctx.compact)

1333 + ctx.base_midx = ctx.compact_from->base_midx;

1334 +

1335 ctx.nr = 0;

1336 ctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;

1337 ctx.info = NULL;

@@ -1210,7 +1348,7 @@ static int write_midx_internal(struct write_midx_opts *opts)

1348 ctx.num_multi_pack_indexes_before++;

1349 m = m->base_midx;

1350 }

1213 - } else if (ctx.m && fill_packs_from_midx(&ctx)) {

1351 + } else if (ctx.m && !ctx.compact && fill_packs_from_midx(&ctx)) {

1352 goto cleanup;

1353 }

1354

@@ -1223,9 +1361,18 @@ static int write_midx_internal(struct write_midx_opts *opts)

1361 else

1362 ctx.progress = NULL;

1363

1226 - ctx.to_include = opts->packs_to_include;

1364 + if (ctx.compact) {

1365 + int bitmap_order = 0;

1366 + if (opts->preferred_pack_name)

1367 + bitmap_order |= 1;

1368 + else if (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))

1369 + bitmap_order |= 1;

1370

1228 - for_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);

1371 + fill_packs_from_midx_range(&ctx, bitmap_order);

1372 + } else {

1373 + ctx.to_include = opts->packs_to_include;

1374 + for_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);

1375 + }

1376 stop_progress(&ctx.progress);

1377

1378 if (!opts->packs_to_drop) {

@@ -1354,12 +1501,19 @@ static int write_midx_internal(struct write_midx_opts *opts)

1501 ctx.large_offsets_needed = 1;

1502 }

1503

1357 - QSORT(ctx.info, ctx.nr, pack_info_compare);

1504 + if (ctx.compact) {

1505 + if (ctx.version != MIDX_VERSION_V2)

1506 + BUG("performing MIDX compaction with v1 MIDX");

1507 + } else {

1508 + QSORT(ctx.info, ctx.nr, pack_info_compare);

1509 + }

1510

1511 if (opts->packs_to_drop && opts->packs_to_drop->nr) {

1512 size_t drop_index = 0;

1513 int missing_drops = 0;

1514

1515 + ASSERT(!ctx.compact);

1516 +

1517 for (size_t i = 0;

1518 i < ctx.nr && drop_index < opts->packs_to_drop->nr; i++) {

1519 int cmp = strcmp(ctx.info[i].pack_name,

@@ -1391,12 +1545,20 @@ static int write_midx_internal(struct write_midx_opts *opts)

1545 */

1546 ALLOC_ARRAY(ctx.pack_perm, ctx.nr);

1547 for (size_t i = 0; i < ctx.nr; i++) {

1548 + uint32_t from = ctx.info[i].orig_pack_int_id;

1549 + uint32_t to;

1550 +

1551 if (ctx.info[i].expired) {

1552 + to = PACK_EXPIRED;

1553 dropped_packs++;

1396 - ctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;

1554 } else {

1398 - ctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;

1555 + to = i - dropped_packs;

1556 }

1557 +

1558 + if (ctx.compact)

1559 + from -= ctx.compact_from->num_packs_in_base;

1560 +

1561 + ctx.pack_perm[from] = to;

1562 }

1563

1564 for (size_t i = 0; i < ctx.nr; i++) {

@@ -1542,7 +1704,24 @@ static int write_midx_internal(struct write_midx_opts *opts)

1704 if (ctx.num_multi_pack_indexes_before == UINT32_MAX)

1705 die(_("too many multi-pack-indexes"));

1706

1545 - CALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);

1707 + if (ctx.compact) {

1708 + struct multi_pack_index *m;

1709 +

1710 + /*

1711 + * Keep all MIDX layers excluding those in the range [from, to].

1712 + */

1713 + for (m = ctx.base_midx; m; m = m->base_midx)

1714 + keep_hashes_nr++;

1715 + for (m = ctx.m;

1716 + m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);

1717 + m = m->base_midx)

1718 + keep_hashes_nr++;

1719 +

1720 + keep_hashes_nr++; /* include the compacted layer */

1721 + } else {

1722 + keep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;

1723 + }

1724 + CALLOC_ARRAY(keep_hashes, keep_hashes_nr);

1725

1726 if (ctx.incremental) {

1727 FILE *chainf = fdopen_lock_file(&lk, "w");

@@ -1567,17 +1746,47 @@ static int write_midx_internal(struct write_midx_opts *opts)

1746

1747 strbuf_release(&final_midx_name);

1748

1570 - keep_hashes[ctx.num_multi_pack_indexes_before] =

1571 - xstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));

1749 + if (ctx.compact) {

1750 + struct multi_pack_index *m;

1751 + uint32_t num_layers_before_from = 0;

1752 + uint32_t i;

1753

1573 - for (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {

1574 - uint32_t j = ctx.num_multi_pack_indexes_before - i - 1;

1754 + for (m = ctx.base_midx; m; m = m->base_midx)

1755 + num_layers_before_from++;

1756

1576 - keep_hashes[j] = xstrdup(midx_get_checksum_hex(m));

1577 - m = m->base_midx;

1757 + m = ctx.base_midx;

1758 + for (i = 0; i < num_layers_before_from; i++) {

1759 + uint32_t j = num_layers_before_from - i - 1;

1760 +

1761 + keep_hashes[j] = xstrdup(midx_get_checksum_hex(m));

1762 + m = m->base_midx;

1763 + }

1764 +

1765 + keep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,

1766 + r->hash_algo));

1767 +

1768 + i = 0;

1769 + for (m = ctx.m;

1770 + m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);

1771 + m = m->base_midx) {

1772 + keep_hashes[keep_hashes_nr - i - 1] =

1773 + xstrdup(midx_get_checksum_hex(m));

1774 + i++;

1775 + }

1776 + } else {

1777 + keep_hashes[ctx.num_multi_pack_indexes_before] =

1778 + xstrdup(hash_to_hex_algop(midx_hash,

1779 + r->hash_algo));

1780 +

1781 + for (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {

1782 + uint32_t j = ctx.num_multi_pack_indexes_before - i - 1;

1783 +

1784 + keep_hashes[j] = xstrdup(midx_get_checksum_hex(m));

1785 + m = m->base_midx;

1786 + }

1787 }

1788

1580 - for (uint32_t i = 0; i <= ctx.num_multi_pack_indexes_before; i++)

1789 + for (uint32_t i = 0; i < keep_hashes_nr; i++)

1790 fprintf(get_lock_file_fp(&lk), "%s\n", keep_hashes[i]);

1791 } else {

1792 keep_hashes[ctx.num_multi_pack_indexes_before] =

@@ -1590,8 +1799,7 @@ static int write_midx_internal(struct write_midx_opts *opts)

1799 if (commit_lock_file(&lk) < 0)

1800 die_errno(_("could not write multi-pack-index"));

1801

1593 - clear_midx_files(opts->source, keep_hashes,

1594 - ctx.num_multi_pack_indexes_before + 1,

1802 + clear_midx_files(opts->source, keep_hashes, keep_hashes_nr,

1803 ctx.incremental);

1804 result = 0;

1805

@@ -1609,7 +1817,7 @@ cleanup:

1817 free(ctx.pack_perm);

1818 free(ctx.pack_order);

1819 if (keep_hashes) {

1612 - for (uint32_t i = 0; i <= ctx.num_multi_pack_indexes_before; i++)

1820 + for (uint32_t i = 0; i < keep_hashes_nr; i++)

1821 free((char *)keep_hashes[i]);

1822 free(keep_hashes);

1823 }

@@ -1651,6 +1859,21 @@ int write_midx_file_only(struct odb_source *source,

1859 return write_midx_internal(&opts);

1860 }

1861

1862 +int write_midx_file_compact(struct odb_source *source,

1863 + struct multi_pack_index *from,

1864 + struct multi_pack_index *to,

1865 + unsigned flags)

1866 +{

1867 + struct write_midx_opts opts = {

1868 + .source = source,

1869 + .compact_from = from,

1870 + .compact_to = to,

1871 + .flags = flags | MIDX_WRITE_COMPACT,

1872 + };

1873 +

1874 + return write_midx_internal(&opts);

1875 +}

1876 +

1877 int expire_midx_packs(struct odb_source *source, unsigned flags)

1878 {

1879 uint32_t i, *count, result = 0;

midx.h

@@ -82,6 +82,7 @@ struct multi_pack_index {

82 #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)

83 #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)

84 #define MIDX_WRITE_INCREMENTAL (1 << 5)

85 +#define MIDX_WRITE_COMPACT (1 << 6)

87 #define MIDX_EXT_REV "rev"

88 #define MIDX_EXT_BITMAP "bitmap"

@@ -131,6 +132,10 @@ int write_midx_file_only(struct odb_source *source,

132 struct string_list *packs_to_include,

133 const char *preferred_pack_name,

134 const char *refs_snapshot, unsigned flags);

135 +int write_midx_file_compact(struct odb_source *source,

136 + struct multi_pack_index *from,

137 + struct multi_pack_index *to,

138 + unsigned flags);

139 void clear_midx_file(struct repository *r);

140 int verify_midx_file(struct odb_source *source, unsigned flags);

141 int expire_midx_packs(struct odb_source *source, unsigned flags);

t/meson.build

@@ -618,6 +618,7 @@ integration_tests = [

618 't5332-multi-pack-reuse.sh',

619 't5333-pseudo-merge-bitmaps.sh',

620 't5334-incremental-multi-pack-index.sh',

621 + 't5335-compact-multi-pack-index.sh',

622 't5351-unpack-large-objects.sh',

623 't5400-send-pack.sh',

624 't5401-update-hooks.sh',

t/t5335-compact-multi-pack-index.sh new

+175

@@ -0,0 +1,175 @@

1 +#!/bin/sh

2 +

3 +test_description='multi-pack-index compaction'

4 +

5 +. ./test-lib.sh

6 +

7 +GIT_TEST_MULTI_PACK_INDEX=0

8 +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0

9 +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0

10 +

11 +objdir=.git/objects

12 +packdir=$objdir/pack

13 +midxdir=$packdir/multi-pack-index.d

14 +midx_chain=$midxdir/multi-pack-index-chain

15 +

16 +nth_line() {

17 + local n="$1"

18 + shift

19 + awk "NR==$n" "$@"

20 +}

21 +

22 +write_packs () {

23 + for c in "$@"

24 + do

25 + test_commit "$c" &&

26 +

27 + git pack-objects --all --unpacked $packdir/pack-$c &&

28 + git prune-packed &&

29 +

30 + git multi-pack-index write --incremental --bitmap || return 1

31 + done

32 +}

33 +

34 +test_midx_layer_packs () {

35 + local checksum="$1" &&

36 + shift &&

37 +

38 + test-tool read-midx $objdir "$checksum" >out &&

39 +

40 + printf "%s\n" "$@" >expect &&

41 + # NOTE: do *not* pipe through sort here, we want to ensure the

42 + # order of packs is preserved during compaction.

43 + grep "^pack-" out | cut -d"-" -f2 >actual &&

44 +

45 + test_cmp expect actual

46 +}

47 +

48 +test_midx_layer_object_uniqueness () {

49 + : >objs.all

50 + while read layer

51 + do

52 + test-tool read-midx --show-objects $objdir "$layer" >out &&

53 + grep "\.pack$" out | cut -d" " -f1 | sort >objs.layer &&

54 + test_stdout_line_count = 0 comm -12 objs.all objs.layer &&

55 + cat objs.all objs.layer | sort >objs.tmp &&

56 + mv objs.tmp objs.all || return 1

57 + done <$midx_chain

58 +}

59 +

60 +test_expect_success 'MIDX compaction with lex-ordered pack names' '

61 + git init midx-compact-lex-order &&

62 + (

63 + cd midx-compact-lex-order &&

64 +

65 + git config maintenance.auto false &&

66 +

67 + write_packs A B C D E &&

68 + test_line_count = 5 $midx_chain &&

69 +

70 + git multi-pack-index compact --incremental \

71 + "$(nth_line 2 "$midx_chain")" \

72 + "$(nth_line 4 "$midx_chain")" &&

73 + test_line_count = 3 $midx_chain &&

74 +

75 + test_midx_layer_packs "$(nth_line 1 "$midx_chain")" A &&

76 + test_midx_layer_packs "$(nth_line 2 "$midx_chain")" B C D &&

77 + test_midx_layer_packs "$(nth_line 3 "$midx_chain")" E &&

78 +

79 + test_midx_layer_object_uniqueness

80 + )

81 +'

82 +

83 +test_expect_success 'MIDX compaction with non-lex-ordered pack names' '

84 + git init midx-compact-non-lex-order &&

85 + (

86 + cd midx-compact-non-lex-order &&

87 +

88 + git config maintenance.auto false &&

89 +

90 + write_packs D C A B E &&

91 + test_line_count = 5 $midx_chain &&

92 +

93 + git multi-pack-index compact --incremental \

94 + "$(nth_line 2 "$midx_chain")" \

95 + "$(nth_line 4 "$midx_chain")" &&

96 + test_line_count = 3 $midx_chain &&

97 +

98 + test_midx_layer_packs "$(nth_line 1 "$midx_chain")" D &&

99 + test_midx_layer_packs "$(nth_line 2 "$midx_chain")" C A B &&

100 + test_midx_layer_packs "$(nth_line 3 "$midx_chain")" E &&

101 +

102 + test_midx_layer_object_uniqueness

103 + )

104 +'

105 +

106 +test_expect_success 'setup for bogus MIDX compaction scenarios' '

107 + git init midx-compact-bogus &&

108 + (

109 + cd midx-compact-bogus &&

110 +

111 + git config maintenance.auto false &&

112 +

113 + write_packs A B C

114 + )

115 +'

116 +

117 +test_expect_success 'MIDX compaction with missing endpoints' '

118 + (

119 + cd midx-compact-bogus &&

120 +

121 + test_must_fail git multi-pack-index compact --incremental \

122 + "<missing>" "<missing>" 2>err &&

123 + test_grep "could not find MIDX: <missing>" err &&

124 +

125 + test_must_fail git multi-pack-index compact --incremental \

126 + "<missing>" "$(nth_line 2 "$midx_chain")" 2>err &&

127 + test_grep "could not find MIDX: <missing>" err &&

128 +

129 + test_must_fail git multi-pack-index compact --incremental \

130 + "$(nth_line 2 "$midx_chain")" "<missing>" 2>err &&

131 + test_grep "could not find MIDX: <missing>" err

132 + )

133 +'

134 +

135 +test_expect_success 'MIDX compaction with reversed endpoints' '

136 + (

137 + cd midx-compact-bogus &&

138 +

139 + from="$(nth_line 3 "$midx_chain")" &&

140 + to="$(nth_line 1 "$midx_chain")" &&

141 +

142 + test_must_fail git multi-pack-index compact --incremental \

143 + "$from" "$to" 2>err &&

144 +

145 + test_grep "MIDX $from must be an ancestor of $to" err

146 + )

147 +'

148 +

149 +test_expect_success 'MIDX compaction with identical endpoints' '

150 + (

151 + cd midx-compact-bogus &&

152 +

153 + from="$(nth_line 3 "$midx_chain")" &&

154 + to="$(nth_line 3 "$midx_chain")" &&

155 +

156 + test_must_fail git multi-pack-index compact --incremental \

157 + "$from" "$to" 2>err &&

158 +

159 + test_grep "MIDX compaction endpoints must be unique" err

160 + )

161 +'

162 +

163 +test_expect_success 'MIDX compaction with midx.version=1' '

164 + (

165 + cd midx-compact-bogus &&

166 +

167 + test_must_fail git -c midx.version=1 multi-pack-index compact \

168 + "$(nth_line 1 "$midx_chain")" \

169 + "$(nth_line 2 "$midx_chain")" 2>err &&

170 +

171 + test_grep "fatal: cannot perform MIDX compaction with v1 format" err

172 + )

173 +'

174 +

175 +test_done