@matheusbsbs / matheus-bagliano / commits / 938af89260

repack: introduce `--write-midx=incremental`

Expose the incremental MIDX repacking mode (implemented in an earlier commit) via a new --write-midx=incremental option for `git repack`. Add "incremental" as a recognized argument to the --write-midx OPT_CALLBACK, mapping it to REPACK_WRITE_MIDX_INCREMENTAL. When this mode is active and --geometric is in use, set the midx_layer_threshold on the pack geometry so that only packs in sufficiently large tip layers are considered for repacking. Two new configuration options control the compaction behavior: - repack.midxSplitFactor (default: 2): the factor used in the geometric merging condition for MIDX layers. - repack.midxNewLayerThreshold (default: 8): the minimum number of packs in the tip MIDX layer before its packs are considered as candidates for geometric repacking. Add tests exercising the new mode across a variety of scenarios including basic geometric violations, multi-round chain integrity, branching and merging histories, cross-layer object uniqueness, and threshold-based compaction. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Taylor Blau committed May 19, 2026 at 11:58 UTC 938af8926099882ff87f8ffa4115c34ed63d9e8b
11 files changed +669 -24
Documentation/config/repack.adoc
+18
@@ -46,3 +46,21 @@ repack.midxMustContainCruft::
46 `--write-midx`. When false, cruft packs are only included in the MIDX
47 when necessary (e.g., because they might be required to form a
48 reachability closure with MIDX bitmaps). Defaults to true.
49 +
50 +repack.midxSplitFactor::
51 + The factor used in the geometric merging condition when
52 + compacting incremental MIDX layers during `git repack` when
53 + invoked with the `--write-midx=incremental` option.
54 ++
55 +Adjacent layers are merged when the accumulated object count of the
56 +newer layer exceeds `1/<N>` of the object count of the next deeper
57 +layer. Must be at least 2. Defaults to 2.
58 +
59 +repack.midxNewLayerThreshold::
60 + The minimum number of packs in the tip MIDX layer before those
61 + packs are considered as candidates for geometric repacking
62 + during `git repack --write-midx=incremental`.
63 ++
64 +When the tip layer has fewer packs than this threshold, those packs are
65 +excluded from the geometric repack entirely, and are thus left
66 +unmodified. Must be at least 1. Defaults to 8.
Documentation/git-repack.adoc
+36 -3
@@ -11,7 +11,7 @@ SYNOPSIS
11 [verse]
12 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]
13 [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]
14 - [--write-midx] [--name-hash-version=<n>] [--path-walk]
14 + [--write-midx[=<mode>]] [--name-hash-version=<n>] [--path-walk]
15
16 DESCRIPTION
17 -----------
@@ -250,9 +250,42 @@ pack as the preferred pack for object selection by the MIDX (see
250 linkgit:git-multi-pack-index[1]).
251
252 -m::
253 ---write-midx::
253 +--write-midx[=<mode>]::
254 Write a multi-pack index (see linkgit:git-multi-pack-index[1])
255 - containing the non-redundant packs.
255 + containing the non-redundant packs. The following modes are
256 + available:
257 ++
258 +--
259 + `default`;;
260 + Write a single MIDX covering all packs. This is the
261 + default when `--write-midx` is given without an
262 + explicit mode.
263 +
264 + `incremental`;;
265 + Write an incremental MIDX chain instead of a single
266 + flat MIDX. This mode requires `--geometric`.
267 ++
268 +The incremental mode maintains a chain of MIDX layers that is compacted
269 +over time using a geometric merging strategy. Each repack creates a new
270 +tip layer containing the newly written pack(s). Adjacent layers are then
271 +merged whenever the newer layer's object count exceeds
272 +`1/repack.midxSplitFactor` of the next deeper layer's count. Layers
273 +that do not meet this condition are retained as-is.
274 ++
275 +The result is that newer (tip) layers tend to contain many small packs
276 +with relatively few objects, while older (deeper) layers contain fewer,
277 +larger packs covering more objects. Because compaction is driven by the
278 +tip of the chain, newer layers are also rewritten more frequently than
279 +older ones, which are only touched when enough objects have accumulated
280 +to justify merging into them. This keeps the total number of layers
281 +logarithmic relative to the total number of objects.
282 ++
283 +Only packs in the tip MIDX layer are considered as candidates for the
284 +geometric repack; packs in deeper layers are left untouched. If the tip
285 +layer contains fewer packs than `repack.midxNewLayerThreshold`, those
286 +packs are excluded from the geometry entirely, and a new layer is
287 +created for any new pack(s) without disturbing the existing chain.
288 +--
289
290 --name-hash-version=<n>::
291 Provide this argument to the underlying `git pack-objects` process.
builtin/repack.c
+43 -6
@@ -33,7 +33,7 @@ static int midx_must_contain_cruft = 1;
33 static const char *const git_repack_usage[] = {
34 N_("git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n"
35 "[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n"
36 - "[--write-midx] [--name-hash-version=<n>] [--path-walk]"),
36 + "[--write-midx[=<mode>]] [--name-hash-version=<n>] [--path-walk]"),
37 NULL
38 };
39
@@ -48,6 +48,8 @@ static const char incremental_bitmap_conflict_error[] = N_(
48 struct repack_config_ctx {
49 struct pack_objects_args *po_args;
50 struct pack_objects_args *cruft_po_args;
51 + int midx_split_factor;
52 + int midx_new_layer_threshold;
53 };
54
55 static int repack_config(const char *var, const char *value,
@@ -97,6 +99,16 @@ static int repack_config(const char *var, const char *value,
99 midx_must_contain_cruft = git_config_bool(var, value);
100 return 0;
101 }
102 + if (!strcmp(var, "repack.midxsplitfactor")) {
103 + repack_ctx->midx_split_factor = git_config_int(var, value,
104 + ctx->kvi);
105 + return 0;
106 + }
107 + if (!strcmp(var, "repack.midxnewlayerthreshold")) {
108 + repack_ctx->midx_new_layer_threshold = git_config_int(var, value,
109 + ctx->kvi);
110 + return 0;
111 + }
112 return git_default_config(var, value, ctx, cb);
113 }
114
@@ -112,6 +124,8 @@ static int option_parse_write_midx(const struct option *opt, const char *arg,
124
125 if (!arg || !*arg)
126 *cfg = REPACK_WRITE_MIDX_DEFAULT;
127 + else if (!strcmp(arg, "incremental"))
128 + *cfg = REPACK_WRITE_MIDX_INCREMENTAL;
129 else
130 return error(_("unknown value for %s: %s"), opt->long_name, arg);
131
@@ -226,6 +240,8 @@ int cmd_repack(int argc,
240 memset(&config_ctx, 0, sizeof(config_ctx));
241 config_ctx.po_args = &po_args;
242 config_ctx.cruft_po_args = &cruft_po_args;
243 + config_ctx.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR;
244 + config_ctx.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD;
245
246 repo_config(repo, repack_config, &config_ctx);
247
@@ -247,6 +263,9 @@ int cmd_repack(int argc,
263 if (pack_everything & PACK_CRUFT)
264 pack_everything |= ALL_INTO_ONE;
265
266 + if (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)
267 + die(_("--write-midx=incremental requires --geometric"));
268 +
269 if (write_bitmaps < 0) {
270 if (write_midx == REPACK_WRITE_MIDX_NONE &&
271 (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))
@@ -273,6 +292,13 @@ int cmd_repack(int argc,
292 write_bitmaps = 0;
293 }
294
295 + if (config_ctx.midx_split_factor < 2)
296 + die(_("invalid value for %s: %d"), "--midx-split-factor",
297 + config_ctx.midx_split_factor);
298 + if (config_ctx.midx_new_layer_threshold < 1)
299 + die(_("invalid value for %s: %d"), "--midx-new-layer-threshold",
300 + config_ctx.midx_new_layer_threshold);
301 +
302 if (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {
303 struct strbuf path = STRBUF_INIT;
304
@@ -296,6 +322,10 @@ int cmd_repack(int argc,
322 if (geometry.split_factor) {
323 if (pack_everything)
324 die(_("options '%s' and '%s' cannot be used together"), "--geometric", "-A/-a");
325 + if (write_midx == REPACK_WRITE_MIDX_INCREMENTAL) {
326 + geometry.midx_layer_threshold = config_ctx.midx_new_layer_threshold;
327 + geometry.midx_layer_threshold_set = true;
328 + }
329 pack_geometry_init(&geometry, &existing, &po_args);
330 pack_geometry_split(&geometry);
331 }
@@ -545,8 +575,11 @@ int cmd_repack(int argc,
575 packtmp);
576 /* End of pack replacement. */
577
548 - if (delete_redundant && pack_everything & ALL_INTO_ONE)
578 + if (delete_redundant && pack_everything & ALL_INTO_ONE) {
579 + if (write_midx == REPACK_WRITE_MIDX_INCREMENTAL)
580 + existing_packs_retain_midx_packs(&existing);
581 existing_packs_mark_for_deletion(&existing, &names);
582 + }
583
584 if (write_midx != REPACK_WRITE_MIDX_NONE) {
585 struct repack_write_midx_opts opts = {
@@ -558,8 +591,8 @@ int cmd_repack(int argc,
591 .show_progress = show_progress,
592 .write_bitmaps = write_bitmaps > 0,
593 .midx_must_contain_cruft = midx_must_contain_cruft,
561 - .midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,
562 - .midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,
594 + .midx_split_factor = config_ctx.midx_split_factor,
595 + .midx_new_layer_threshold = config_ctx.midx_new_layer_threshold,
596 .mode = write_midx,
597 };
598
@@ -572,11 +605,15 @@ int cmd_repack(int argc,
605
606 if (delete_redundant) {
607 int opts = 0;
575 - existing_packs_remove_redundant(&existing, packdir);
608 + bool wrote_incremental_midx = write_midx == REPACK_WRITE_MIDX_INCREMENTAL;
609 +
610 + existing_packs_remove_redundant(&existing, packdir,
611 + wrote_incremental_midx);
612
613 if (geometry.split_factor)
614 pack_geometry_remove_redundant(&geometry, &names,
579 - &existing, packdir);
615 + &existing, packdir,
616 + wrote_incremental_midx);
617 if (show_progress)
618 opts |= PRUNE_PACKED_VERBOSE;
619 prune_packed_objects(opts);
midx.c
+29
@@ -850,6 +850,35 @@ void clear_midx_file(struct repository *r)
850 strbuf_release(&midx);
851 }
852
853 +void clear_incremental_midx_files(struct repository *r,
854 + const struct strvec *keep_hashes)
855 +{
856 + struct odb_source *source = r->objects->sources;
857 + struct strbuf chain = STRBUF_INIT;
858 +
859 + get_midx_chain_filename(source, &chain);
860 +
861 + for (; source; source = source->next) {
862 + struct odb_source_files *files = odb_source_files_downcast(source);
863 + if (files->packed->midx)
864 + close_midx(files->packed->midx);
865 + files->packed->midx = NULL;
866 + }
867 +
868 + if (!keep_hashes && remove_path(chain.buf))
869 + die(_("failed to clear multi-pack-index chain at %s"),
870 + chain.buf);
871 +
872 + clear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_BITMAP,
873 + keep_hashes);
874 + clear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_REV,
875 + keep_hashes);
876 + clear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_MIDX,
877 + keep_hashes);
878 +
879 + strbuf_release(&chain);
880 +}
881 +
882 static int verify_midx_error;
883
884 __attribute__((format (printf, 1, 2)))
midx.h
+3
@@ -9,6 +9,7 @@ struct repository;
9 struct bitmapped_pack;
10 struct git_hash_algo;
11 struct odb_source;
12 +struct strvec;
13
14 #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
15 #define MIDX_VERSION_V1 1
@@ -143,6 +144,8 @@ int write_midx_file_compact(struct odb_source *source,
144 const char *incremental_base,
145 unsigned flags);
146 void clear_midx_file(struct repository *r);
147 +void clear_incremental_midx_files(struct repository *r,
148 + const struct strvec *keep_hashes);
149 int verify_midx_file(struct odb_source *source, unsigned flags);
150 int expire_midx_packs(struct odb_source *source, unsigned flags);
151 int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags);
repack-geometry.c
+8 -5
@@ -249,7 +249,8 @@ static void remove_redundant_packs(struct packed_git **pack,
249 uint32_t pack_nr,
250 struct string_list *names,
251 struct existing_packs *existing,
252 - const char *packdir)
252 + const char *packdir,
253 + bool wrote_incremental_midx)
254 {
255 const struct git_hash_algo *algop = existing->repo->hash_algo;
256 struct strbuf buf = STRBUF_INIT;
@@ -269,7 +270,8 @@ static void remove_redundant_packs(struct packed_git **pack,
270 (string_list_has_string(&existing->kept_packs, buf.buf)))
271 continue;
272
272 - repack_remove_redundant_pack(existing->repo, packdir, buf.buf);
273 + repack_remove_redundant_pack(existing->repo, packdir, buf.buf,
274 + wrote_incremental_midx);
275 }
276
277 strbuf_release(&buf);
@@ -278,12 +280,13 @@ static void remove_redundant_packs(struct packed_git **pack,
280 void pack_geometry_remove_redundant(struct pack_geometry *geometry,
281 struct string_list *names,
282 struct existing_packs *existing,
281 - const char *packdir)
283 + const char *packdir,
284 + bool wrote_incremental_midx)
285 {
286 remove_redundant_packs(geometry->pack, geometry->split,
284 - names, existing, packdir);
287 + names, existing, packdir, wrote_incremental_midx);
288 remove_redundant_packs(geometry->promisor_pack, geometry->promisor_split,
286 - names, existing, packdir);
289 + names, existing, packdir, wrote_incremental_midx);
290 }
291
292 void pack_geometry_release(struct pack_geometry *geometry)
repack-midx.c
+5
@@ -887,6 +887,7 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)
887 struct midx_compaction_step *steps = NULL;
888 struct strbuf lock_name = STRBUF_INIT;
889 struct lock_file lf;
890 + struct strvec keep_hashes = STRVEC_INIT;
891 size_t steps_nr = 0;
892 size_t i;
893 int ret = 0;
@@ -932,11 +933,15 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)
933 BUG("missing result for compaction step %"PRIuMAX,
934 (uintmax_t)i);
935 fprintf(get_lock_file_fp(&lf), "%s\n", step->csum);
936 + strvec_push(&keep_hashes, step->csum);
937 }
938
939 commit_lock_file(&lf);
940
941 + clear_incremental_midx_files(opts->existing->repo, &keep_hashes);
942 +
943 done:
944 + strvec_clear(&keep_hashes);
945 strbuf_release(&lock_name);
946 for (i = 0; i < steps_nr; i++)
947 midx_compaction_step_release(&steps[i]);
repack.c
+49 -7
@@ -55,14 +55,18 @@ void pack_objects_args_release(struct pack_objects_args *args)
55 }
56
57 void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,
58 - const char *base_name)
58 + const char *base_name,
59 + bool wrote_incremental_midx)
60 {
61 struct strbuf buf = STRBUF_INIT;
62 struct odb_source *source = repo->objects->sources;
63 struct multi_pack_index *m = get_multi_pack_index(source);
64 strbuf_addf(&buf, "%s.pack", base_name);
64 - if (m && source->local && midx_contains_pack(m, buf.buf))
65 + if (m && source->local && midx_contains_pack(m, buf.buf)) {
66 clear_midx_file(repo);
67 + if (!wrote_incremental_midx)
68 + clear_incremental_midx_files(repo, NULL);
69 + }
70 strbuf_insertf(&buf, 0, "%s/", dir_name);
71 unlink_pack_path(buf.buf, 1);
72 strbuf_release(&buf);
@@ -250,25 +254,63 @@ void existing_packs_mark_for_deletion(struct existing_packs *existing,
254 &existing->cruft_packs);
255 }
256
257 +/*
258 + * Mark every pack that is referenced by the existing MIDX chain as
259 + * retained, so that a subsequent call to
260 + * existing_packs_mark_for_deletion() will not mark them for deletion.
261 + *
262 + * This is used when writing an incremental MIDX layer on top of an
263 + * existing chain: retained layers continue to reference the same
264 + * packs on disk, so those packs must not be unlinked even if the
265 + * freshly-written pack supersedes them.
266 + */
267 +void existing_packs_retain_midx_packs(struct existing_packs *existing)
268 +{
269 + struct string_list_item *item;
270 + struct strbuf buf = STRBUF_INIT;
271 +
272 + for_each_string_list_item(item, &existing->midx_packs) {
273 + struct string_list_item *found;
274 +
275 + strbuf_reset(&buf);
276 + strbuf_addstr(&buf, item->string);
277 + strbuf_strip_suffix(&buf, ".pack");
278 + strbuf_strip_suffix(&buf, ".idx");
279 +
280 + found = string_list_lookup(&existing->non_kept_packs, buf.buf);
281 + if (found)
282 + existing_packs_mark_retained(found);
283 +
284 + found = string_list_lookup(&existing->cruft_packs, buf.buf);
285 + if (found)
286 + existing_packs_mark_retained(found);
287 + }
288 +
289 + strbuf_release(&buf);
290 +}
291 +
292 static void remove_redundant_packs_1(struct repository *repo,
293 struct string_list *packs,
255 - const char *packdir)
294 + const char *packdir,
295 + bool wrote_incremental_midx)
296 {
297 struct string_list_item *item;
298 for_each_string_list_item(item, packs) {
299 if (!existing_pack_is_marked_for_deletion(item))
300 continue;
261 - repack_remove_redundant_pack(repo, packdir, item->string);
301 + repack_remove_redundant_pack(repo, packdir, item->string,
302 + wrote_incremental_midx);
303 }
304 }
305
306 void existing_packs_remove_redundant(struct existing_packs *existing,
266 - const char *packdir)
307 + const char *packdir,
308 + bool wrote_incremental_midx)
309 {
310 remove_redundant_packs_1(existing->repo, &existing->non_kept_packs,
269 - packdir);
311 + packdir, wrote_incremental_midx);
312 remove_redundant_packs_1(existing->repo, &existing->cruft_packs,
271 - packdir);
313 + packdir, wrote_incremental_midx);
314 }
315
316 void existing_packs_release(struct existing_packs *existing)
repack.h
+7 -3
@@ -34,7 +34,8 @@ void prepare_pack_objects(struct child_process *cmd,
34 void pack_objects_args_release(struct pack_objects_args *args);
35
36 void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,
37 - const char *base_name);
37 + const char *base_name,
38 + bool wrote_incremental_midx);
39
40 struct write_pack_opts {
41 struct pack_objects_args *po_args;
@@ -83,8 +84,10 @@ void existing_packs_retain_cruft(struct existing_packs *existing,
84 struct packed_git *cruft);
85 void existing_packs_mark_for_deletion(struct existing_packs *existing,
86 struct string_list *names);
87 +void existing_packs_retain_midx_packs(struct existing_packs *existing);
88 void existing_packs_remove_redundant(struct existing_packs *existing,
87 - const char *packdir);
89 + const char *packdir,
90 + bool wrote_incremental_midx);
91 void existing_packs_release(struct existing_packs *existing);
92
93 struct generated_pack;
@@ -129,7 +132,8 @@ struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry);
132 void pack_geometry_remove_redundant(struct pack_geometry *geometry,
133 struct string_list *names,
134 struct existing_packs *existing,
132 - const char *packdir);
135 + const char *packdir,
136 + bool wrote_incremental_midx);
137 void pack_geometry_release(struct pack_geometry *geometry);
138
139 struct tempfile;
t/meson.build
+1
@@ -951,6 +951,7 @@ integration_tests = [
951 't7702-repack-cyclic-alternate.sh',
952 't7703-repack-geometric.sh',
953 't7704-repack-cruft.sh',
954 + 't7705-repack-incremental-midx.sh',
955 't7800-difftool.sh',
956 't7810-grep.sh',
957 't7811-grep-open.sh',
t/t7705-repack-incremental-midx.sh new
+470
@@ -0,0 +1,470 @@
1 +#!/bin/sh
2 +
3 +test_description='git repack --write-midx=incremental'
4 +
5 +. ./test-lib.sh
6 +
7 +GIT_TEST_MULTI_PACK_INDEX=0
8 +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
9 +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0
10 +
11 +objdir=.git/objects
12 +packdir=$objdir/pack
13 +midxdir=$packdir/multi-pack-index.d
14 +midx_chain=$midxdir/multi-pack-index-chain
15 +
16 +# incrementally_repack N
17 +#
18 +# Make "N" new commits, each stored in their own pack, and then repacked
19 +# with the --write-midx=incremental strategy.
20 +incrementally_repack () {
21 + for i in $(test_seq 1 "$1")
22 + do
23 + test_commit "$i" &&
24 +
25 + git repack --geometric=2 -d --write-midx=incremental \
26 + --write-bitmap-index &&
27 + git multi-pack-index verify || return 1
28 + done
29 +}
30 +
31 +# Create packs with geometrically increasing sizes so that they
32 +# satisfy the geometric progression and survive a --geometric=2
33 +# repack without being rolled up. Creates 3 packs containing 1,
34 +# 2, and 6 commits (3, 6, and 18 objects) respectively.
35 +create_geometric_packs () {
36 + test_commit "small" &&
37 + git repack -d &&
38 +
39 + test_commit_bulk --message="medium" 2 &&
40 + test_commit_bulk --message="large" 6 &&
41 +
42 + git repack --geometric=2 -d --write-midx=incremental \
43 + --write-bitmap-index
44 +}
45 +
46 +# create_layer <test_commit_bulk args>
47 +#
48 +# Creates a new MIDX layer with the contents of "test_commit_bulk $@".
49 +create_layer () {
50 + test_commit_bulk "$@" &&
51 +
52 + git multi-pack-index write --incremental --bitmap
53 +}
54 +
55 +# create_layers
56 +#
57 +# Reads lines of "<message> <nr>" from stdin and creates a new MIDX
58 +# layer for each line. See create_layer above for more.
59 +create_layers () {
60 + while read msg nr
61 + do
62 + create_layer --message="$msg" "$nr" || return 1
63 + done
64 +}
65 +
66 +test_expect_success '--write-midx=incremental requires --geometric' '
67 + test_must_fail git repack --write-midx=incremental 2>err &&
68 +
69 + test_grep -- "--write-midx=incremental requires --geometric" err
70 +'
71 +
72 +test_expect_success 'below layer threshold, tip packs excluded' '
73 + git init below-layer-threshold-tip-packs-excluded &&
74 + (
75 + cd below-layer-threshold-tip-packs-excluded &&
76 +
77 + git config maintenance.auto false &&
78 + git config repack.midxnewlayerthreshold 4 &&
79 + git config repack.midxsplitfactor 2 &&
80 +
81 + # Create 3 packs forming a geometric progression by
82 + # object count such that they are unmodified by the
83 + # initial repack. The MIDX chain thusly contains a
84 + # single layer with three packs.
85 + create_geometric_packs &&
86 + ls $packdir/pack-*.idx | sort >packs.before &&
87 + test_line_count = 1 $midx_chain &&
88 + cp $midx_chain $midx_chain.before &&
89 +
90 + # Repack a new commit. Since the layer threshold is
91 + # unmet, a new MIDX layer is added on top of the
92 + # existing one.
93 + test_commit extra &&
94 + git repack --geometric=2 -d --write-midx=incremental \
95 + --write-bitmap-index &&
96 + git multi-pack-index verify &&
97 +
98 + ls $packdir/pack-*.idx | sort >packs.after &&
99 + comm -13 packs.before packs.after >packs.new &&
100 + test_line_count = 1 packs.new &&
101 +
102 + test_line_count = 2 "$midx_chain" &&
103 + head -n 1 "$midx_chain.before" >expect &&
104 + head -n 1 "$midx_chain" >actual &&
105 + test_cmp expect actual
106 + )
107 +'
108 +
109 +test_expect_success 'above layer threshold, tip packs repacked' '
110 + git init above-layer-threshold-tip-packs-repacked &&
111 + (
112 + cd above-layer-threshold-tip-packs-repacked &&
113 +
114 + git config maintenance.auto false &&
115 + git config repack.midxnewlayerthreshold 2 &&
116 + git config repack.midxsplitfactor 2 &&
117 +
118 + # Same setup, but with the layer threshold set to 2.
119 + # Since the tip MIDX layer meets that threshold, its
120 + # packs are considered repack candidates.
121 + create_geometric_packs &&
122 + cp $midx_chain $midx_chain.before &&
123 +
124 + # Perturb the existing progression such that it is
125 + # rolled up into a single new pack, invalidating the
126 + # existing MIDX layer and replacing it with a new one.
127 + test_commit extra &&
128 + git repack -d &&
129 + git repack --geometric=2 -d --write-midx=incremental \
130 + --write-bitmap-index &&
131 +
132 + ! test_cmp $midx_chain.before $midx_chain &&
133 + test_line_count = 1 $midx_chain &&
134 +
135 + git multi-pack-index verify
136 + )
137 +'
138 +
139 +test_expect_success 'above layer threshold, tip layer preserved' '
140 + git init above-layer-threshold-tip-layer-preserved &&
141 + (
142 + cd above-layer-threshold-tip-layer-preserved &&
143 +
144 + git config maintenance.auto false &&
145 + git config repack.midxnewlayerthreshold 2 &&
146 + git config repack.midxsplitfactor 2 &&
147 +
148 + test_commit_bulk --message="medium" 2 &&
149 + test_commit_bulk --message="large" 6 &&
150 +
151 + git repack --geometric=2 -d --write-midx=incremental \
152 + --write-bitmap-index &&
153 +
154 + test_line_count = 1 "$midx_chain" &&
155 + ls $packdir/pack-*.idx | sort >packs.before &&
156 + cp $midx_chain $midx_chain.before &&
157 +
158 + # Create objects to form a pack satisfying the geometric
159 + # progression (thus preserving the tip layer), but not
160 + # so large that it meets the layer merging condition.
161 + test_commit_bulk --message="small" 1 &&
162 + git repack --geometric=2 -d --write-midx=incremental \
163 + --write-bitmap-index &&
164 +
165 + ls $packdir/pack-*.idx | sort >packs.after &&
166 + comm -13 packs.before packs.after >packs.new &&
167 +
168 + test_line_count = 1 packs.new &&
169 + test_line_count = 3 packs.after &&
170 + test_line_count = 2 "$midx_chain" &&
171 + head -n 1 "$midx_chain.before" >expect &&
172 + head -n 1 "$midx_chain" >actual &&
173 + test_cmp expect actual &&
174 +
175 + git multi-pack-index verify
176 + )
177 +'
178 +
179 +test_expect_success 'above layer threshold, tip packs preserved' '
180 + git init above-layer-threshold-tip-packs-preserved &&
181 + (
182 + cd above-layer-threshold-tip-packs-preserved &&
183 +
184 + git config maintenance.auto false &&
185 + git config repack.midxnewlayerthreshold 2 &&
186 + git config repack.midxsplitfactor 2 &&
187 +
188 + create_geometric_packs &&
189 + ls $packdir/pack-*.idx | sort >packs.before &&
190 + cp $midx_chain $midx_chain.before &&
191 +
192 + # Same setup as above, but this time the new objects do
193 + # not satisfy the new layer merging condition, resulting
194 + # in a new tip layer.
195 + test_commit_bulk --message="huge" 18 &&
196 + git repack --geometric=2 -d --write-midx=incremental \
197 + --write-bitmap-index &&
198 +
199 + ls $packdir/pack-*.idx | sort >packs.after &&
200 + comm -13 packs.before packs.after >packs.new &&
201 +
202 + ! test_cmp $midx_chain.before $midx_chain &&
203 + test_line_count = 1 $midx_chain &&
204 + test_line_count = 1 packs.new &&
205 +
206 + git multi-pack-index verify
207 + )
208 +'
209 +
210 +test_expect_success 'new tip absorbs multiple layers' '
211 + git init new-tip-absorbs-multiple-layers &&
212 + (
213 + cd new-tip-absorbs-multiple-layers &&
214 +
215 + git config maintenance.auto false &&
216 + git config repack.midxnewlayerthreshold 1 &&
217 + git config repack.midxsplitfactor 2 &&
218 +
219 + # Build a 4-layer chain where each layer is too small to
220 + # absorb the one below it. The sizes must satisfy L(n) <
221 + # L(n-1)/2 for each adjacent pair:
222 + #
223 + # L0 (oldest): 75 obj (25 commits)
224 + # L1: 21 obj (7 commits, 21 < 75/2)
225 + # L2: 9 obj (3 commits, 9 < 21/2)
226 + # L3 (tip): 3 obj (1 commit, 3 < 9/2)
227 + create_layers <<-\EOF &&
228 + L0 25
229 + L1 7
230 + L2 3
231 + L3 1
232 + EOF
233 +
234 + test_line_count = 4 "$midx_chain" &&
235 + cp $midx_chain $midx_chain.before &&
236 +
237 + # Now add a new commit. The merging condition is
238 + # satisfied between L3-L1, but violated at L0, which is
239 + # too large relative to the accumulated size.
240 + #
241 + # As a result, the chain shrinks from 4 to 2 layers.
242 + test_commit new &&
243 + git repack --geometric=2 -d --write-midx=incremental \
244 + --write-bitmap-index &&
245 +
246 + ! test_cmp $midx_chain.before $midx_chain &&
247 + test_line_count = 2 "$midx_chain" &&
248 + git multi-pack-index verify
249 + )
250 +'
251 +
252 +test_expect_success 'compaction of older layers' '
253 + git init compaction-of-older-layers &&
254 + (
255 + cd compaction-of-older-layers &&
256 +
257 + git config maintenance.auto false &&
258 + git config repack.midxnewlayerthreshold 1 &&
259 + git config repack.midxsplitfactor 2 &&
260 +
261 + # Build a chain with two small layers at the bottom
262 + # and a larger barrier layer on top, producing a
263 + # chain that violates the compaction invariant, since
264 + # the two small layers would normally have been merged.
265 + create_layers <<-\EOF &&
266 + one 2
267 + two 4
268 + barrier 54
269 + EOF
270 +
271 + cp $midx_chain $midx_chain.before &&
272 +
273 + # Running an incremental repack compacts the two
274 + # small layers at the bottom of the chain as a
275 + # separate step in the compaction plan.
276 + test_commit another &&
277 + git repack --geometric=2 -d --write-midx=incremental \
278 + --write-bitmap-index &&
279 +
280 + test_line_count = 2 "$midx_chain" &&
281 + git multi-pack-index verify
282 + )
283 +'
284 +
285 +test_expect_success 'geometric rollup with surviving tip packs' '
286 + git init geometric-rollup-with-surviving-tip-packs &&
287 + (
288 + cd geometric-rollup-with-surviving-tip-packs &&
289 +
290 + git config maintenance.auto false &&
291 + git config repack.midxnewlayerthreshold 1 &&
292 + git config repack.midxsplitfactor 2 &&
293 +
294 + # Create a pack large enough to anchor the geometric
295 + # progression when small packs are added alongside it.
296 + create_layer --message="big" 5 &&
297 +
298 + test_line_count = 1 "$midx_chain" &&
299 + cp $midx_chain $midx_chain.before &&
300 +
301 + # Repack a small number of objects such that the
302 + # progression is unbothered. Note that the existing pack
303 + # is considered a repack candidate as the new layer
304 + # threshold is set to 1.
305 + test_commit small-1 &&
306 + git repack -d &&
307 + git repack --geometric=2 -d --write-midx=incremental \
308 + --write-bitmap-index &&
309 +
310 + ! test_cmp $midx_chain.before $midx_chain &&
311 + cp $midx_chain $midx_chain.before
312 + )
313 +'
314 +
315 +test_expect_success 'kept packs are excluded from repack' '
316 + git init kept-packs-excluded-from-repack &&
317 + (
318 + cd kept-packs-excluded-from-repack &&
319 +
320 + git config maintenance.auto false &&
321 + git config repack.midxnewlayerthreshold 1 &&
322 + git config repack.midxsplitfactor 2 &&
323 +
324 + # Create two equal-sized packs, marking one as kept.
325 + for i in A B
326 + do
327 + test_commit "$i" && git repack -d || return 1
328 + done &&
329 +
330 + keep=$(ls $packdir/pack-*.idx | head -n 1) &&
331 + touch "${keep%.idx}.keep" &&
332 +
333 + # The kept pack is excluded as a repacking candidate
334 + # entirely, so no rollup occurs as there is only one
335 + # non-kept pack. A new MIDX layer is written containing
336 + # that pack.
337 + git repack --geometric=2 -d --write-midx=incremental \
338 + --write-bitmap-index &&
339 +
340 + test-tool read-midx $objdir >actual &&
341 + grep "^pack-.*\.idx$" actual >actual.packs &&
342 + test_line_count = 1 actual.packs &&
343 + test_grep ! "$keep" actual.packs &&
344 +
345 + git multi-pack-index verify &&
346 +
347 + # All objects (from both kept and non-kept packs)
348 + # must still be accessible.
349 + git fsck
350 + )
351 +'
352 +
353 +test_expect_success 'incremental MIDX with --max-pack-size' '
354 + git init incremental-midx-with--max-pack-size &&
355 + (
356 + cd incremental-midx-with--max-pack-size &&
357 +
358 + git config maintenance.auto false &&
359 + git config repack.midxnewlayerthreshold 1 &&
360 + git config repack.midxsplitfactor 2 &&
361 +
362 + create_layer --message="base" 1 &&
363 +
364 + # Now add enough data that a small --max-pack-size will
365 + # cause pack-objects to split its output. Create objects
366 + # large enough to fill multiple packs.
367 + test-tool genrandom foo 1M >big1 &&
368 + test-tool genrandom bar 1M >big2 &&
369 + git add big1 big2 &&
370 + test_tick &&
371 + git commit -a -m "big blobs" &&
372 + git repack -d &&
373 +
374 + git repack --geometric=2 -d --write-midx=incremental \
375 + --write-bitmap-index --max-pack-size=1M &&
376 +
377 + test_line_count = 1 "$midx_chain" &&
378 + test-tool read-midx $objdir >actual &&
379 + grep "^pack-.*\.idx$" actual >actual.packs &&
380 + test_line_count -gt 1 actual.packs &&
381 +
382 + git multi-pack-index verify
383 + )
384 +'
385 +
386 +test_expect_success 'noop repack preserves valid MIDX chain' '
387 + git init noop-repack-preserves-valid-midx-chain &&
388 + (
389 + cd noop-repack-preserves-valid-midx-chain &&
390 +
391 + git config maintenance.auto false &&
392 + git config repack.midxnewlayerthreshold 1 &&
393 + git config repack.midxsplitfactor 2 &&
394 +
395 + create_layer --message="base" 1 &&
396 +
397 + git multi-pack-index verify &&
398 + cp $midx_chain $midx_chain.before &&
399 +
400 + # Running again with no new objects should not break
401 + # the MIDX chain. It produces "Nothing new to pack."
402 + git repack --geometric=2 -d --write-midx=incremental \
403 + --write-bitmap-index &&
404 +
405 + test_cmp $midx_chain.before $midx_chain &&
406 +
407 + git multi-pack-index verify &&
408 + git fsck
409 + )
410 +'
411 +
412 +test_expect_success 'repack -ad removes stale incremental chain' '
413 + git init repack--ad-removes-stale-incremental-chain &&
414 + (
415 + cd repack--ad-removes-stale-incremental-chain &&
416 +
417 + git config maintenance.auto false &&
418 + git config repack.midxnewlayerthreshold 1 &&
419 + git config repack.midxsplitfactor 2 &&
420 +
421 + create_layers <<-\EOF &&
422 + one 1
423 + two 1
424 + EOF
425 +
426 + test_path_is_file $midx_chain &&
427 + test_line_count = 2 $midx_chain &&
428 +
429 + git repack -ad &&
430 +
431 + test_path_is_missing $packdir/multi-pack-index &&
432 + test_dir_is_empty $midxdir
433 + )
434 +'
435 +
436 +test_expect_success 'repack rejects invalid midxSplitFactor' '
437 + test_when_finished "rm -fr bad-split-factor" &&
438 + git init bad-split-factor &&
439 + (
440 + cd bad-split-factor &&
441 + test_commit base &&
442 +
443 + for v in 0 1 -1
444 + do
445 + test_must_fail git -c repack.midxSplitFactor=$v \
446 + repack -d --geometric=2 --write-midx=incremental 2>err &&
447 + test_grep "invalid value for --midx-split-factor" err ||
448 + return 1
449 + done
450 + )
451 +'
452 +
453 +test_expect_success 'repack rejects invalid midxNewLayerThreshold' '
454 + test_when_finished "rm -fr bad-layer-threshold" &&
455 + git init bad-layer-threshold &&
456 + (
457 + cd bad-layer-threshold &&
458 + test_commit base &&
459 +
460 + for v in 0 -1
461 + do
462 + test_must_fail git -c repack.midxNewLayerThreshold=$v \
463 + repack -d --geometric=2 --write-midx=incremental 2>err &&
464 + test_grep "invalid value for --midx-new-layer-threshold" err ||
465 + return 1
466 + done
467 + )
468 +'
469 +
470 +test_done