clone, submodule: pass partial clone filters to submodules
When cloning a repo with a --filter and with --recurse-submodules enabled, the partial clone filter only applies to the top-level repo. This can lead to unexpected bandwidth and disk usage for projects which include large submodules. For example, a user might wish to make a partial clone of Gerrit and would run: `git clone --recurse-submodules --filter=blob:5k https://gerrit.googlesource.com/gerrit`. However, only the superproject would be a partial clone; all the submodules would have all blobs downloaded regardless of their size. With this change, the same filter can also be applied to submodules, meaning the expected bandwidth and disk savings apply consistently. To avoid changing default behavior, add a new clone flag, `--also-filter-submodules`. When this is set along with `--filter` and `--recurse-submodules`, the filter spec is passed along to git-submodule and git-submodule--helper, such that submodule clones also have the filter applied. This applies the same filter to the superproject and all submodules. Users who need to customize the filter per-submodule would need to clone with `--no-recurse-submodules` and then manually initialize each submodule with the proper filter. Applying filters to submodules should be safe thanks to Jonathan Tan's recent work [1, 2, 3] eliminating the use of alternates as a method of accessing submodule objects, so any submodule object access now triggers a lazy fetch from the submodule's promisor remote if the accessed object is missing. This patch is a reworked version of [4], which was created prior to Jonathan Tan's work. [1]: 8721e2e (Merge branch 'jt/partial-clone-submodule-1', 2021-07-16) [2]: 11e5d0a (Merge branch 'jt/grep-wo-submodule-odb-as-alternate', 2021-09-20) [3]: 162a13b (Merge branch 'jt/no-abuse-alternate-odb-for-submodules', 2021-10-25) [4]: https://lore.kernel.org/git/52bf9d45b8e2b72ff32aa773f2415bf7b2b86da2.1563322192.git.steadmon@google.com/ Signed-off-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Josh Steadmon committed
Feb 4, 2022 at 21:00 UTC
f05da2b48b48a46db65fc768b3ffecaf996dd655
8 files changed
+175
-8
Documentation/config/clone.txt
+5
@@ -6,3 +6,8 @@ clone.defaultRemoteName::
6
clone.rejectShallow::
7
Reject to clone a repository if it is a shallow one, can be overridden by
8
passing option `--reject-shallow` in command line. See linkgit:git-clone[1]
9
+
10
+clone.filterSubmodules::
11
+ If a partial clone filter is provided (see `--filter` in
12
+ linkgit:git-rev-list[1]) and `--recurse-submodules` is used, also apply
13
+ the filter to submodules.
Documentation/git-clone.txt
+6
-1
@@ -16,7 +16,7 @@ SYNOPSIS
16
[--depth <depth>] [--[no-]single-branch] [--no-tags]
17
[--recurse-submodules[=<pathspec>]] [--[no-]shallow-submodules]
18
[--[no-]remote-submodules] [--jobs <n>] [--sparse] [--[no-]reject-shallow]
19
- [--filter=<filter>] [--] <repository>
19
+ [--filter=<filter> [--also-filter-submodules]] [--] <repository>
20
[<directory>]
21
22
DESCRIPTION
@@ -182,6 +182,11 @@ objects from the source repository into a pack in the cloned repository.
182
at least `<size>`. For more details on filter specifications, see
183
the `--filter` option in linkgit:git-rev-list[1].
184
185
+--also-filter-submodules::
186
+ Also apply the partial clone filter to any submodules in the repository.
187
+ Requires `--filter` and `--recurse-submodules`. This can be turned on by
188
+ default by setting the `clone.filterSubmodules` config option.
189
+
190
--mirror::
191
Set up a mirror of the source repository. This implies `--bare`.
192
Compared to `--bare`, `--mirror` not only maps local branches of the
Documentation/git-submodule.txt
+5
-1
@@ -133,7 +133,7 @@ If you really want to remove a submodule from the repository and commit
133
that use linkgit:git-rm[1] instead. See linkgit:gitsubmodules[7] for removal
134
options.
135
136
-update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference <repository>] [--depth <depth>] [--recursive] [--jobs <n>] [--[no-]single-branch] [--] [<path>...]::
136
+update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference <repository>] [--depth <depth>] [--recursive] [--jobs <n>] [--[no-]single-branch] [--filter <filter spec>] [--] [<path>...]::
137
+
138
--
139
Update the registered submodules to match what the superproject
@@ -177,6 +177,10 @@ submodule with the `--init` option.
177
178
If `--recursive` is specified, this command will recurse into the
179
registered submodules, and update any nested submodules within.
180
+
181
+If `--filter <filter spec>` is specified, the given partial clone filter will be
182
+applied to the submodule. See linkgit:git-rev-list[1] for details on filter
183
+specifications.
184
--
185
set-branch (-b|--branch) <branch> [--] <path>::
186
set-branch (-d|--default) [--] <path>::
builtin/clone.c
+34
-2
@@ -71,6 +71,8 @@ static int option_dissociate;
71
static int max_jobs = -1;
72
static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;
73
static struct list_objects_filter_options filter_options;
74
+static int option_filter_submodules = -1; /* unspecified */
75
+static int config_filter_submodules = -1; /* unspecified */
76
static struct string_list server_options = STRING_LIST_INIT_NODUP;
77
static int option_remote_submodules;
78
@@ -150,6 +152,8 @@ static struct option builtin_clone_options[] = {
152
OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
153
TRANSPORT_FAMILY_IPV6),
154
OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
155
+ OPT_BOOL(0, "also-filter-submodules", &option_filter_submodules,
156
+ N_("apply partial clone filters to submodules")),
157
OPT_BOOL(0, "remote-submodules", &option_remote_submodules,
158
N_("any cloned submodules will use their remote-tracking branch")),
159
OPT_BOOL(0, "sparse", &option_sparse_checkout,
@@ -650,7 +654,7 @@ static int git_sparse_checkout_init(const char *repo)
654
return result;
655
}
656
653
-static int checkout(int submodule_progress)
657
+static int checkout(int submodule_progress, int filter_submodules)
658
{
659
struct object_id oid;
660
char *head;
@@ -729,6 +733,10 @@ static int checkout(int submodule_progress)
733
strvec_push(&args, "--no-fetch");
734
}
735
736
+ if (filter_submodules && filter_options.choice)
737
+ strvec_pushf(&args, "--filter=%s",
738
+ expand_list_objects_filter_spec(&filter_options));
739
+
740
if (option_single_branch >= 0)
741
strvec_push(&args, option_single_branch ?
742
"--single-branch" :
@@ -749,6 +757,8 @@ static int git_clone_config(const char *k, const char *v, void *cb)
757
}
758
if (!strcmp(k, "clone.rejectshallow"))
759
config_reject_shallow = git_config_bool(k, v);
760
+ if (!strcmp(k, "clone.filtersubmodules"))
761
+ config_filter_submodules = git_config_bool(k, v);
762
763
return git_default_config(k, v, cb);
764
}
@@ -871,6 +881,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
881
struct remote *remote;
882
int err = 0, complete_refs_before_fetch = 1;
883
int submodule_progress;
884
+ int filter_submodules = 0;
885
886
struct transport_ls_refs_options transport_ls_refs_options =
887
TRANSPORT_LS_REFS_OPTIONS_INIT;
@@ -1066,6 +1077,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
1077
if (option_reject_shallow != -1)
1078
reject_shallow = option_reject_shallow;
1079
1080
+ /*
1081
+ * If option_filter_submodules is specified from CLI option,
1082
+ * ignore config_filter_submodules from git_clone_config.
1083
+ */
1084
+ if (config_filter_submodules != -1)
1085
+ filter_submodules = config_filter_submodules;
1086
+ if (option_filter_submodules != -1)
1087
+ filter_submodules = option_filter_submodules;
1088
+
1089
+ /*
1090
+ * Exit if the user seems to be doing something silly with submodule
1091
+ * filter flags (but not with filter configs, as those should be
1092
+ * set-and-forget).
1093
+ */
1094
+ if (option_filter_submodules > 0 && !filter_options.choice)
1095
+ die(_("the option '%s' requires '%s'"),
1096
+ "--also-filter-submodules", "--filter");
1097
+ if (option_filter_submodules > 0 && !option_recurse_submodules.nr)
1098
+ die(_("the option '%s' requires '%s'"),
1099
+ "--also-filter-submodules", "--recurse-submodules");
1100
+
1101
/*
1102
* apply the remote name provided by --origin only after this second
1103
* call to git_config, to ensure it overrides all config-based values.
@@ -1299,7 +1331,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
1331
}
1332
1333
junk_mode = JUNK_LEAVE_REPO;
1302
- err = checkout(submodule_progress);
1334
+ err = checkout(submodule_progress, filter_submodules);
1335
1336
free(remote_name);
1337
strbuf_release(&reflog_msg);
builtin/submodule--helper.c
+27
-3
@@ -20,6 +20,7 @@
20
#include "diff.h"
21
#include "object-store.h"
22
#include "advice.h"
23
+#include "list-objects-filter-options.h"
24
25
#define OPT_QUIET (1 << 0)
26
#define OPT_CACHED (1 << 1)
@@ -1630,6 +1631,7 @@ struct module_clone_data {
1631
const char *name;
1632
const char *url;
1633
const char *depth;
1634
+ struct list_objects_filter_options *filter_options;
1635
struct string_list reference;
1636
unsigned int quiet: 1;
1637
unsigned int progress: 1;
@@ -1796,6 +1798,10 @@ static int clone_submodule(struct module_clone_data *clone_data)
1798
strvec_push(&cp.args, "--dissociate");
1799
if (sm_gitdir && *sm_gitdir)
1800
strvec_pushl(&cp.args, "--separate-git-dir", sm_gitdir, NULL);
1801
+ if (clone_data->filter_options && clone_data->filter_options->choice)
1802
+ strvec_pushf(&cp.args, "--filter=%s",
1803
+ expand_list_objects_filter_spec(
1804
+ clone_data->filter_options));
1805
if (clone_data->single_branch >= 0)
1806
strvec_push(&cp.args, clone_data->single_branch ?
1807
"--single-branch" :
@@ -1852,6 +1858,7 @@ static int module_clone(int argc, const char **argv, const char *prefix)
1858
{
1859
int dissociate = 0, quiet = 0, progress = 0, require_init = 0;
1860
struct module_clone_data clone_data = MODULE_CLONE_DATA_INIT;
1861
+ struct list_objects_filter_options filter_options;
1862
1863
struct option module_clone_options[] = {
1864
OPT_STRING(0, "prefix", &clone_data.prefix,
@@ -1881,17 +1888,19 @@ static int module_clone(int argc, const char **argv, const char *prefix)
1888
N_("disallow cloning into non-empty directory")),
1889
OPT_BOOL(0, "single-branch", &clone_data.single_branch,
1890
N_("clone only one branch, HEAD or --branch")),
1891
+ OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
1892
OPT_END()
1893
};
1894
1895
const char *const git_submodule_helper_usage[] = {
1896
N_("git submodule--helper clone [--prefix=<path>] [--quiet] "
1897
"[--reference <repository>] [--name <name>] [--depth <depth>] "
1890
- "[--single-branch] "
1898
+ "[--single-branch] [--filter <filter-spec>]"
1899
"--url <url> --path <path>"),
1900
NULL
1901
};
1902
1903
+ memset(&filter_options, 0, sizeof(filter_options));
1904
argc = parse_options(argc, argv, prefix, module_clone_options,
1905
git_submodule_helper_usage, 0);
1906
@@ -1899,12 +1908,14 @@ static int module_clone(int argc, const char **argv, const char *prefix)
1908
clone_data.quiet = !!quiet;
1909
clone_data.progress = !!progress;
1910
clone_data.require_init = !!require_init;
1911
+ clone_data.filter_options = &filter_options;
1912
1913
if (argc || !clone_data.url || !clone_data.path || !*(clone_data.path))
1914
usage_with_options(git_submodule_helper_usage,
1915
module_clone_options);
1916
1917
clone_submodule(&clone_data);
1918
+ list_objects_filter_release(&filter_options);
1919
return 0;
1920
}
1921
@@ -1994,6 +2005,7 @@ struct submodule_update_clone {
2005
const char *recursive_prefix;
2006
const char *prefix;
2007
int single_branch;
2008
+ struct list_objects_filter_options *filter_options;
2009
2010
/* to be consumed by git-submodule.sh */
2011
struct update_clone_data *update_clone;
@@ -2154,6 +2166,9 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
2166
strvec_pushl(&child->args, "--prefix", suc->prefix, NULL);
2167
if (suc->recommend_shallow && sub->recommend_shallow == 1)
2168
strvec_push(&child->args, "--depth=1");
2169
+ if (suc->filter_options && suc->filter_options->choice)
2170
+ strvec_pushf(&child->args, "--filter=%s",
2171
+ expand_list_objects_filter_spec(suc->filter_options));
2172
if (suc->require_init)
2173
strvec_push(&child->args, "--require-init");
2174
strvec_pushl(&child->args, "--path", sub->path, NULL);
@@ -2498,6 +2513,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)
2513
const char *update = NULL;
2514
struct pathspec pathspec;
2515
struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;
2516
+ struct list_objects_filter_options filter_options;
2517
+ int ret;
2518
2519
struct option module_update_clone_options[] = {
2520
OPT_STRING(0, "prefix", &prefix,
@@ -2528,6 +2545,7 @@ static int update_clone(int argc, const char **argv, const char *prefix)
2545
N_("disallow cloning into non-empty directory")),
2546
OPT_BOOL(0, "single-branch", &suc.single_branch,
2547
N_("clone only one branch, HEAD or --branch")),
2548
+ OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
2549
OPT_END()
2550
};
2551
@@ -2540,20 +2558,26 @@ static int update_clone(int argc, const char **argv, const char *prefix)
2558
update_clone_config_from_gitmodules(&suc.max_jobs);
2559
git_config(git_update_clone_config, &suc.max_jobs);
2560
2561
+ memset(&filter_options, 0, sizeof(filter_options));
2562
argc = parse_options(argc, argv, prefix, module_update_clone_options,
2563
git_submodule_helper_usage, 0);
2564
+ suc.filter_options = &filter_options;
2565
2566
if (update)
2567
if (parse_submodule_update_strategy(update, &suc.update) < 0)
2568
die(_("bad value for update parameter"));
2569
2550
- if (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0)
2570
+ if (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0) {
2571
+ list_objects_filter_release(&filter_options);
2572
return 1;
2573
+ }
2574
2575
if (pathspec.nr)
2576
suc.warn_if_uninitialized = 1;
2577
2556
- return update_submodules(&suc);
2578
+ ret = update_submodules(&suc);
2579
+ list_objects_filter_release(&filter_options);
2580
+ return ret;
2581
}
2582
2583
static int run_update_procedure(int argc, const char **argv, const char *prefix)
git-submodule.sh
+16
-1
@@ -10,7 +10,7 @@ USAGE="[--quiet] [--cached]
10
or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
11
or: $dashless [--quiet] init [--] [<path>...]
12
or: $dashless [--quiet] deinit [-f|--force] (--all| [--] <path>...)
13
- or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--[no-]single-branch] [--] [<path>...]
13
+ or: $dashless [--quiet] update [--init [--filter=<filter-spec>]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--[no-]single-branch] [--] [<path>...]
14
or: $dashless [--quiet] set-branch (--default|--branch <branch>) [--] <path>
15
or: $dashless [--quiet] set-url [--] <path> <newurl>
16
or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
@@ -49,6 +49,7 @@ dissociate=
49
single_branch=
50
jobs=
51
recommend_shallow=
52
+filter=
53
54
die_if_unmatched ()
55
{
@@ -347,6 +348,14 @@ cmd_update()
348
--no-single-branch)
349
single_branch="--no-single-branch"
350
;;
351
+ --filter)
352
+ case "$2" in '') usage ;; esac
353
+ filter="--filter=$2"
354
+ shift
355
+ ;;
356
+ --filter=*)
357
+ filter="$1"
358
+ ;;
359
--)
360
shift
361
break
@@ -361,6 +370,11 @@ cmd_update()
370
shift
371
done
372
373
+ if test -n "$filter" && test "$init" != "1"
374
+ then
375
+ usage
376
+ fi
377
+
378
if test -n "$init"
379
then
380
cmd_init "--" "$@" || return
@@ -379,6 +393,7 @@ cmd_update()
393
$single_branch \
394
$recommend_shallow \
395
$jobs \
396
+ $filter \
397
-- \
398
"$@" || echo "#unmatched" $?
399
} | {
t/t5617-clone-submodules-remote.sh
+41
@@ -28,6 +28,13 @@ test_expect_success 'setup' '
28
)
29
'
30
31
+# bare clone giving "srv.bare" for use as our server.
32
+test_expect_success 'setup bare clone for server' '
33
+ git clone --bare "file://$(pwd)/." srv.bare &&
34
+ git -C srv.bare config --local uploadpack.allowfilter 1 &&
35
+ git -C srv.bare config --local uploadpack.allowanysha1inwant 1
36
+'
37
+
38
test_expect_success 'clone with --no-remote-submodules' '
39
test_when_finished "rm -rf super_clone" &&
40
git clone --recurse-submodules --no-remote-submodules "file://$pwd/." super_clone &&
@@ -65,4 +72,38 @@ test_expect_success 'clone with --single-branch' '
72
)
73
'
74
75
+# do basic partial clone from "srv.bare"
76
+# confirm partial clone was registered in the local config for super and sub.
77
+test_expect_success 'clone with --filter' '
78
+ git clone --recurse-submodules \
79
+ --filter blob:none --also-filter-submodules \
80
+ "file://$pwd/srv.bare" super_clone &&
81
+ test_cmp_config -C super_clone true remote.origin.promisor &&
82
+ test_cmp_config -C super_clone blob:none remote.origin.partialclonefilter &&
83
+ test_cmp_config -C super_clone/sub true remote.origin.promisor &&
84
+ test_cmp_config -C super_clone/sub blob:none remote.origin.partialclonefilter
85
+'
86
+
87
+# check that clone.filterSubmodules works (--also-filter-submodules can be
88
+# omitted)
89
+test_expect_success 'filters applied with clone.filterSubmodules' '
90
+ test_config_global clone.filterSubmodules true &&
91
+ git clone --recurse-submodules --filter blob:none \
92
+ "file://$pwd/srv.bare" super_clone2 &&
93
+ test_cmp_config -C super_clone2 true remote.origin.promisor &&
94
+ test_cmp_config -C super_clone2 blob:none remote.origin.partialclonefilter &&
95
+ test_cmp_config -C super_clone2/sub true remote.origin.promisor &&
96
+ test_cmp_config -C super_clone2/sub blob:none remote.origin.partialclonefilter
97
+'
98
+
99
+test_expect_success '--no-also-filter-submodules overrides clone.filterSubmodules=true' '
100
+ test_config_global clone.filterSubmodules true &&
101
+ git clone --recurse-submodules --filter blob:none \
102
+ --no-also-filter-submodules \
103
+ "file://$pwd/srv.bare" super_clone3 &&
104
+ test_cmp_config -C super_clone3 true remote.origin.promisor &&
105
+ test_cmp_config -C super_clone3 blob:none remote.origin.partialclonefilter &&
106
+ test_cmp_config -C super_clone3/sub false --default false remote.origin.promisor
107
+'
108
+
109
test_done
t/t7814-grep-recurse-submodules.sh
+41
@@ -544,4 +544,45 @@ test_expect_failure 'grep saves textconv cache in the appropriate repository' '
544
test_path_is_file "$sub_textconv_cache"
545
'
546
547
+test_expect_success 'grep partially-cloned submodule' '
548
+ # Set up clean superproject and submodule for partial cloning.
549
+ git init super &&
550
+ git init super/sub &&
551
+ (
552
+ cd super &&
553
+ test_commit --no-tag "Add file in superproject" \
554
+ super-file "Some content for super-file" &&
555
+ test_commit -C sub --no-tag "Add file in submodule" \
556
+ sub-file "Some content for sub-file" &&
557
+ git submodule add ./sub &&
558
+ git commit -m "Add other as submodule sub" &&
559
+ test_tick &&
560
+ test_commit -C sub --no-tag --append "Update file in submodule" \
561
+ sub-file "Some more content for sub-file" &&
562
+ git add sub &&
563
+ git commit -m "Update submodule" &&
564
+ test_tick &&
565
+ git config --local uploadpack.allowfilter 1 &&
566
+ git config --local uploadpack.allowanysha1inwant 1 &&
567
+ git -C sub config --local uploadpack.allowfilter 1 &&
568
+ git -C sub config --local uploadpack.allowanysha1inwant 1
569
+ ) &&
570
+ # Clone the superproject & submodule, then make sure we can lazy-fetch submodule objects.
571
+ git clone --filter=blob:none --also-filter-submodules \
572
+ --recurse-submodules "file://$(pwd)/super" partial &&
573
+ (
574
+ cd partial &&
575
+ cat >expect <<-\EOF &&
576
+ HEAD^:sub/sub-file:Some content for sub-file
577
+ HEAD^:super-file:Some content for super-file
578
+ EOF
579
+
580
+ GIT_TRACE2_EVENT="$(pwd)/trace2.log" git grep -e content \
581
+ --recurse-submodules HEAD^ >actual &&
582
+ test_cmp expect actual &&
583
+ # Verify that we actually fetched data from the promisor remote:
584
+ grep \"category\":\"promisor\",\"key\":\"fetch_count\",\"value\":\"1\" trace2.log
585
+ )
586
+'
587
+
588
test_done