pack-objects: support excluded-open packs with --stdin-packs
In cd846bacc7d (pack-objects: introduce '--stdin-packs=follow', 2025-06-23), pack-objects learned to traverse through commits in included packs when using '--stdin-packs=follow', rescuing reachable objects from unlisted packs into the output. When we encounter a commit in an excluded pack during this rescuing phase we will traverse through its parents. But because we set `revs.no_kept_objects = 1`, commit simplification will prevent us from showing it via `get_revision()`. (In practice, `--stdin-packs=follow` walks commits down to the roots, but only opens up trees for ones that do not appear in an excluded pack.) But there are certain cases where we *do* need to see the parents of an object in an excluded pack. Namely, if an object is rescue-able, but only reachable from object(s) which appear in excluded packs, then commit simplification will exclude those commits from the object traversal, and we will never see a copy of that object, and thus not rescue it. This is what causes the failure in the previous commit during repacking. When performing a geometric repack, packs above the geometric split that weren't part of the previous MIDX (e.g., packs pushed directly into `$GIT_DIR/objects/pack`) may not have full object closure. When those packs are listed as excluded via the '^' marker, the reachability traversal encounters the sequence described above, and may miss objects which we expect to rescue with `--stdin-packs=follow`. Introduce a new "excluded-open" pack prefix, '!'. Like '^'-prefixed packs, objects from '!'-prefixed packs are excluded from the resulting pack. But unlike '^', commits in '!'-prefixed packs *are* used as starting points for the follow traversal, and the traversal does not treat them as a closure boundary. In order to distinguish excluded-closed from excluded-open packs during the traversal, introduce a new `pack_keep_in_core_open` bit on `struct packed_git`, along with a corresponding `KEPT_PACK_IN_CORE_OPEN` flag for the kept-pack cache. In `add_object_entry_from_pack()`, move the `want_object_in_pack()` check to *after* `add_pending_oid()`. This is necessary so that commits from excluded-open packs are added as traversal tips even though their objects won't appear in the output. As a consequence, the caller `for_each_object_in_pack()` will always provide a non-NULL 'p', hence we are able to drop the "if (p)" conditional. The `include_check` and `include_check_obj` callbacks on `rev_info` are used to halt the walk at closed-excluded packs, since objects behind a '^' boundary are guaranteed to have closure and need not be rescued. The following commit will make use of this new functionality within the repack layer to resolve the test failure demonstrated in the previous commit. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Taylor Blau committed
Mar 27, 2026 at 16:06 UTC
3f7c0e722e2733aede32b1e531caf83e7043d1bd
5 files changed
+218
-33
Documentation/git-pack-objects.adoc
+18
-7
@@ -94,13 +94,24 @@ base-name::
94
included packs (those not beginning with `^`), excluding any
95
objects listed in the excluded packs (beginning with `^`).
96
+
97
-When `mode` is "follow", objects from packs not listed on stdin receive
98
-special treatment. Objects within unlisted packs will be included if
99
-those objects are (1) reachable from the included packs, and (2) not
100
-found in any excluded packs. This mode is useful, for example, to
101
-resurrect once-unreachable objects found in cruft packs to generate
102
-packs which are closed under reachability up to the boundary set by the
103
-excluded packs.
97
+When `mode` is "follow" packs may additionally be prefixed with `!`,
98
+indicating that they are excluded but not necessarily closed under
99
+reachability. In addition to objects in included packs, the resulting
100
+pack may include additional objects based on the following:
101
++
102
+--
103
+* If any packs are marked with `!`, then objects reachable from such
104
+ packs or included ones via objects outside of excluded-closed packs
105
+ will be included. In this case, all `^` packs are treated as closed
106
+ under reachability.
107
+* Otherwise (if there are no `!` packs), objects within unlisted packs
108
+ will be included if those objects are (1) reachable from the
109
+ included packs, and (2) not found in any excluded packs.
110
+--
111
++
112
+This mode is useful, for example, to resurrect once-unreachable
113
+objects found in cruft packs to generate packs which are closed under
114
+reachability up to the boundary set by the excluded packs.
115
+
116
Incompatible with `--revs`, or options that imply `--revs` (such as
117
`--all`), with the exception of `--unpacked`, which is compatible.
builtin/pack-objects.c
+91
-25
@@ -217,6 +217,7 @@ static int have_non_local_packs;
217
static int incremental;
218
static int ignore_packed_keep_on_disk;
219
static int ignore_packed_keep_in_core;
220
+static int ignore_packed_keep_in_core_open;
221
static int ignore_packed_keep_in_core_has_cruft;
222
static int allow_ofs_delta;
223
static struct pack_idx_option pack_idx_opts;
@@ -1618,7 +1619,8 @@ static int want_found_object(const struct object_id *oid, int exclude,
1619
/*
1620
* Then handle .keep first, as we have a fast(er) path there.
1621
*/
1621
- if (ignore_packed_keep_on_disk || ignore_packed_keep_in_core) {
1622
+ if (ignore_packed_keep_on_disk || ignore_packed_keep_in_core ||
1623
+ ignore_packed_keep_in_core_open) {
1624
/*
1625
* Set the flags for the kept-pack cache to be the ones we want
1626
* to ignore.
@@ -1632,6 +1634,8 @@ static int want_found_object(const struct object_id *oid, int exclude,
1634
flags |= KEPT_PACK_ON_DISK;
1635
if (ignore_packed_keep_in_core)
1636
flags |= KEPT_PACK_IN_CORE;
1637
+ if (ignore_packed_keep_in_core_open)
1638
+ flags |= KEPT_PACK_IN_CORE_OPEN;
1639
1640
/*
1641
* If the object is in a pack that we want to ignore, *and* we
@@ -1643,6 +1647,8 @@ static int want_found_object(const struct object_id *oid, int exclude,
1647
return 0;
1648
if (ignore_packed_keep_in_core && p->pack_keep_in_core)
1649
return 0;
1650
+ if (ignore_packed_keep_in_core_open && p->pack_keep_in_core_open)
1651
+ return 0;
1652
if (has_object_kept_pack(p->repo, oid, flags))
1653
return 0;
1654
} else {
@@ -3742,6 +3748,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,
3748
void *_data)
3749
{
3750
off_t ofs;
3751
+ struct object_info oi = OBJECT_INFO_INIT;
3752
enum object_type type = OBJ_NONE;
3753
3754
display_progress(progress_state, ++nr_seen);
@@ -3749,29 +3756,34 @@ static int add_object_entry_from_pack(const struct object_id *oid,
3756
if (have_duplicate_entry(oid, 0))
3757
return 0;
3758
3752
- ofs = nth_packed_object_offset(p, pos);
3753
- if (!want_object_in_pack(oid, 0, &p, &ofs))
3754
- return 0;
3759
+ stdin_packs_found_nr++;
3760
3756
- if (p) {
3757
- struct object_info oi = OBJECT_INFO_INIT;
3758
-
3759
- oi.typep = &type;
3760
- if (packed_object_info(p, ofs, &oi) < 0) {
3761
- die(_("could not get type of object %s in pack %s"),
3762
- oid_to_hex(oid), p->pack_name);
3763
- } else if (type == OBJ_COMMIT) {
3764
- struct rev_info *revs = _data;
3765
- /*
3766
- * commits in included packs are used as starting points for the
3767
- * subsequent revision walk
3768
- */
3769
- add_pending_oid(revs, NULL, oid, 0);
3770
- }
3761
+ ofs = nth_packed_object_offset(p, pos);
3762
3772
- stdin_packs_found_nr++;
3763
+ oi.typep = &type;
3764
+ if (packed_object_info(p, ofs, &oi) < 0) {
3765
+ die(_("could not get type of object %s in pack %s"),
3766
+ oid_to_hex(oid), p->pack_name);
3767
+ } else if (type == OBJ_COMMIT) {
3768
+ struct rev_info *revs = _data;
3769
+ /*
3770
+ * commits in included packs are used as starting points
3771
+ * for the subsequent revision walk
3772
+ *
3773
+ * Note that we do want to walk through commits that are
3774
+ * present in excluded-open ('!') packs to pick up any
3775
+ * objects reachable from them not present in the
3776
+ * excluded-closed ('^') packs.
3777
+ *
3778
+ * However, we'll only add those objects to the packing
3779
+ * list after checking `want_object_in_pack()` below.
3780
+ */
3781
+ add_pending_oid(revs, NULL, oid, 0);
3782
}
3783
3784
+ if (!want_object_in_pack(oid, 0, &p, &ofs))
3785
+ return 0;
3786
+
3787
create_object_entry(oid, type, 0, 0, 0, p, ofs);
3788
3789
return 0;
@@ -3832,12 +3844,18 @@ static void show_commit_pack_hint(struct commit *commit, void *data)
3844
* - STDIN_PACK_EXCLUDE_CLOSED: objects in any packs with this flag
3845
* bit set should be excluded from the output pack.
3846
*
3835
- * Objects in packs whose 'kind' bits include STDIN_PACK_INCLUDE are
3836
- * used as traversal tips when invoked with --stdin-packs=follow.
3847
+ * - STDIN_PACK_EXCLUDE_OPEN: objects in any packs with this flag
3848
+ * bit set should be excluded from the output pack, but are not
3849
+ * guaranteed to be closed under reachability.
3850
+ *
3851
+ * Objects in packs whose 'kind' bits include STDIN_PACK_INCLUDE or
3852
+ * STDIN_PACK_EXCLUDE_OPEN are used as traversal tips when invoked
3853
+ * with --stdin-packs=follow.
3854
*/
3855
enum stdin_pack_info_kind {
3856
STDIN_PACK_INCLUDE = (1<<0),
3857
STDIN_PACK_EXCLUDE_CLOSED = (1<<1),
3858
+ STDIN_PACK_EXCLUDE_OPEN = (1<<2),
3859
};
3860
3861
struct stdin_pack_info {
@@ -3862,6 +3880,17 @@ static int pack_mtime_cmp(const void *_a, const void *_b)
3880
return 0;
3881
}
3882
3883
+static int stdin_packs_include_check_obj(struct object *obj, void *data UNUSED)
3884
+{
3885
+ return !has_object_kept_pack(to_pack.repo, &obj->oid,
3886
+ KEPT_PACK_IN_CORE);
3887
+}
3888
+
3889
+static int stdin_packs_include_check(struct commit *commit, void *data)
3890
+{
3891
+ return stdin_packs_include_check_obj((struct object *)commit, data);
3892
+}
3893
+
3894
static void stdin_packs_add_pack_entries(struct strmap *packs,
3895
struct rev_info *revs)
3896
{
@@ -3888,7 +3917,19 @@ static void stdin_packs_add_pack_entries(struct strmap *packs,
3917
for_each_string_list_item(item, &keys) {
3918
struct stdin_pack_info *info = item->util;
3919
3891
- if (info->kind & STDIN_PACK_INCLUDE)
3920
+ if (info->kind & STDIN_PACK_EXCLUDE_OPEN) {
3921
+ /*
3922
+ * When open-excluded packs ("!") are present, stop
3923
+ * the parent walk at closed-excluded ("^") packs.
3924
+ * Objects behind a "^" boundary are guaranteed to
3925
+ * have closure and should not be rescued.
3926
+ */
3927
+ revs->include_check = stdin_packs_include_check;
3928
+ revs->include_check_obj = stdin_packs_include_check_obj;
3929
+ }
3930
+
3931
+ if ((info->kind & STDIN_PACK_INCLUDE) ||
3932
+ (info->kind & STDIN_PACK_EXCLUDE_OPEN))
3933
for_each_object_in_pack(info->p,
3934
add_object_entry_from_pack,
3935
revs,
@@ -3898,7 +3939,8 @@ static void stdin_packs_add_pack_entries(struct strmap *packs,
3939
string_list_clear(&keys, 0);
3940
}
3941
3901
-static void stdin_packs_read_input(struct rev_info *revs)
3942
+static void stdin_packs_read_input(struct rev_info *revs,
3943
+ enum stdin_packs_mode mode)
3944
{
3945
struct strbuf buf = STRBUF_INIT;
3946
struct strmap packs = STRMAP_INIT;
@@ -3913,6 +3955,8 @@ static void stdin_packs_read_input(struct rev_info *revs)
3955
continue;
3956
else if (*key == '^')
3957
kind = STDIN_PACK_EXCLUDE_CLOSED;
3958
+ else if (*key == '!' && mode == STDIN_PACKS_MODE_FOLLOW)
3959
+ kind = STDIN_PACK_EXCLUDE_OPEN;
3960
3961
if (kind != STDIN_PACK_INCLUDE)
3962
key++;
@@ -3959,6 +4003,20 @@ static void stdin_packs_read_input(struct rev_info *revs)
4003
p->pack_keep_in_core = 1;
4004
}
4005
4006
+ if (info->kind & STDIN_PACK_EXCLUDE_OPEN) {
4007
+ /*
4008
+ * Marking excluded open packs as kept in-core
4009
+ * (open) for the same reason as we marked
4010
+ * exclude closed packs as kept in-core.
4011
+ *
4012
+ * Use a separate flag here to ensure we don't
4013
+ * halt our traversal at these packs, since they
4014
+ * are not guaranteed to have closure.
4015
+ *
4016
+ */
4017
+ p->pack_keep_in_core_open = 1;
4018
+ }
4019
+
4020
info->p = p;
4021
}
4022
@@ -4002,7 +4060,15 @@ static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)
4060
4061
/* avoids adding objects in excluded packs */
4062
ignore_packed_keep_in_core = 1;
4005
- stdin_packs_read_input(&revs);
4063
+ if (mode == STDIN_PACKS_MODE_FOLLOW) {
4064
+ /*
4065
+ * In '--stdin-packs=follow' mode, additionally ignore
4066
+ * objects in excluded-open packs to prevent them from
4067
+ * appearing in the resulting pack.
4068
+ */
4069
+ ignore_packed_keep_in_core_open = 1;
4070
+ }
4071
+ stdin_packs_read_input(&revs, mode);
4072
if (rev_list_unpacked)
4073
add_unreachable_loose_objects(&revs);
4074
packfile.c
+2
-1
@@ -2246,7 +2246,8 @@ struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *st
2246
struct packed_git *p = e->pack;
2247
2248
if ((p->pack_keep && (flags & KEPT_PACK_ON_DISK)) ||
2249
- (p->pack_keep_in_core && (flags & KEPT_PACK_IN_CORE))) {
2249
+ (p->pack_keep_in_core && (flags & KEPT_PACK_IN_CORE)) ||
2250
+ (p->pack_keep_in_core_open && (flags & KEPT_PACK_IN_CORE_OPEN))) {
2251
ALLOC_GROW(packs, nr + 1, alloc);
2252
packs[nr++] = p;
2253
}
packfile.h
+2
@@ -28,6 +28,7 @@ struct packed_git {
28
unsigned pack_local:1,
29
pack_keep:1,
30
pack_keep_in_core:1,
31
+ pack_keep_in_core_open:1,
32
freshened:1,
33
do_not_close:1,
34
pack_promisor:1,
@@ -266,6 +267,7 @@ int packfile_store_freshen_object(struct packfile_store *store,
267
enum kept_pack_type {
268
KEPT_PACK_ON_DISK = (1 << 0),
269
KEPT_PACK_IN_CORE = (1 << 1),
270
+ KEPT_PACK_IN_CORE_OPEN = (1 << 2),
271
};
272
273
/*
t/t5331-pack-objects-stdin.sh
+105
@@ -415,4 +415,109 @@ test_expect_success '--stdin-packs=follow tolerates missing commits' '
415
stdin_packs__follow_with_only HEAD HEAD^{tree}
416
'
417
418
+test_expect_success '--stdin-packs=follow with open-excluded packs' '
419
+ test_when_finished "rm -fr repo" &&
420
+
421
+ git init repo &&
422
+ (
423
+ cd repo &&
424
+ git config set maintenance.auto false &&
425
+
426
+ git branch -M main &&
427
+
428
+ # Create the following commit structure:
429
+ #
430
+ # A <-- B <-- D (main)
431
+ # ^
432
+ # \
433
+ # C (other)
434
+ test_commit A &&
435
+ test_commit B &&
436
+ git checkout -B other &&
437
+ test_commit C &&
438
+ git checkout main &&
439
+ test_commit D &&
440
+
441
+ A="$(echo A | git pack-objects --revs $packdir/pack)" &&
442
+ B="$(echo A..B | git pack-objects --revs $packdir/pack)" &&
443
+ C="$(echo B..C | git pack-objects --revs $packdir/pack)" &&
444
+ D="$(echo B..D | git pack-objects --revs $packdir/pack)" &&
445
+
446
+ C_ONLY="$(git rev-parse other | git pack-objects $packdir/pack)" &&
447
+
448
+ git prune-packed &&
449
+
450
+ # Create a pack using --stdin-packs=follow where:
451
+ #
452
+ # - pack D is included,
453
+ # - pack C_ONLY is excluded, but open,
454
+ # - pack B is excluded, but closed, and
455
+ # - packs A and C are unknown
456
+ #
457
+ # The resulting pack should therefore contain:
458
+ #
459
+ # - objects from the included pack D,
460
+ # - A.t (rescued via D^{tree}), and
461
+ # - C^{tree} and C.t (rescued via pack C_ONLY)
462
+ #
463
+ # , but should omit:
464
+ #
465
+ # - C (excluded via C_ONLY),
466
+ # - objects from pack B (trivially excluded-closed)
467
+ # - A and A^{tree} (ancestors of B)
468
+ P=$(git pack-objects --stdin-packs=follow $packdir/pack <<-EOF
469
+ pack-$D.pack
470
+ !pack-$C_ONLY.pack
471
+ ^pack-$B.pack
472
+ EOF
473
+ ) &&
474
+
475
+ {
476
+ objects_in_packs $D &&
477
+ git rev-parse A:A.t "C^{tree}" C:C.t
478
+ } >expect.raw &&
479
+ sort expect.raw >expect &&
480
+
481
+ objects_in_packs $P >actual &&
482
+ test_cmp expect actual
483
+ )
484
+'
485
+
486
+test_expect_success '--stdin-packs with !-delimited pack without follow' '
487
+ test_when_finished "rm -fr repo" &&
488
+
489
+ git init repo &&
490
+ (
491
+ test_commit A &&
492
+ test_commit B &&
493
+ test_commit C &&
494
+
495
+ A="$(echo A | git pack-objects --revs $packdir/pack)" &&
496
+ B="$(echo A..B | git pack-objects --revs $packdir/pack)" &&
497
+ C="$(echo B..C | git pack-objects --revs $packdir/pack)" &&
498
+
499
+ cat >in <<-EOF &&
500
+ !pack-$A.pack
501
+ pack-$B.pack
502
+ pack-$C.pack
503
+ EOF
504
+
505
+ # Without --stdin-packs=follow, we treat the first
506
+ # line of input as a literal packfile name, and thus
507
+ # expect pack-objects to complain of a missing pack
508
+ test_must_fail git pack-objects --stdin-packs --stdout \
509
+ >/dev/null <in 2>err &&
510
+ test_grep "could not find pack .!pack-$A.pack." err &&
511
+
512
+ # With --stdin-packs=follow, we treat the second line
513
+ # of input as indicating pack-$A.pack is an excluded
514
+ # open pack, and thus expect pack-objects to succeed
515
+ P=$(git pack-objects --stdin-packs=follow $packdir/pack <in) &&
516
+
517
+ objects_in_packs $B $C >expect &&
518
+ objects_in_packs $P >actual &&
519
+ test_cmp expect actual
520
+ )
521
+'
522
+
523
test_done