@matheusbsbs / matheus-bagliano / commits / bff4555767

backfill: add --sparse option

One way to significantly reduce the cost of a Git clone and later fetches is to use a blobless partial clone and combine that with a sparse-checkout that reduces the paths that need to be populated in the working directory. Not only does this reduce the cost of clones and fetches, the sparse-checkout reduces the number of objects needed to download from a promisor remote. However, history investigations can be expensive as computing blob diffs will trigger promisor remote requests for one object at a time. This can be avoided by downloading the blobs needed for the given sparse-checkout using 'git backfill' and its new '--sparse' mode, at a time that the user is willing to pay that extra cost. Note that this is distinctly different from the '--filter=sparse:<oid>' option, as this assumes that the partial clone has all reachable trees and we are using client-side logic to avoid downloading blobs outside of the sparse-checkout cone. This avoids the server-side cost of walking trees while also achieving a similar goal. It also downloads in batches based on similar path names, presenting a resumable download if things are interrupted. This augments the path-walk API to have a possibly-NULL 'pl' member that may point to a 'struct pattern_list'. This could be more general than the sparse-checkout definition at HEAD, but 'git backfill --sparse' is currently the only consumer. Be sure to test this in both cone mode and not cone mode. Cone mode has the benefit that the path-walk can skip certain paths once they would expand beyond the sparse-checkout. Non-cone mode can describe the included files using both positive and negative patterns, which changes the possible return values of path_matches_pattern_list(). Test both kinds of matches for increased coverage. To test this, we can create a blobless sparse clone, expand the sparse-checkout slightly, and then run 'git backfill --sparse' to see how much data is downloaded. The general steps are 1. git clone --filter=blob:none --sparse <url> 2. git sparse-checkout set <dir1> ... <dirN> 3. git backfill --sparse For the Git repository with the 'builtin' directory in the sparse-checkout, we get these results for various batch sizes: | Batch Size | Pack Count | Pack Size | Time | |-----------------|------------|-----------|-------| | (Initial clone) | 3 | 110 MB | | | 10K | 12 | 192 MB | 17.2s | | 15K | 9 | 192 MB | 15.5s | | 20K | 8 | 192 MB | 15.5s | | 25K | 7 | 192 MB | 14.7s | This case matters less because a full clone of the Git repository from GitHub is currently at 277 MB. Using a copy of the Linux repository with the 'kernel/' directory in the sparse-checkout, we get these results: | Batch Size | Pack Count | Pack Size | Time | |-----------------|------------|-----------|------| | (Initial clone) | 2 | 1,876 MB | | | 10K | 11 | 2,187 MB | 46s | | 25K | 7 | 2,188 MB | 43s | | 50K | 5 | 2,194 MB | 44s | | 100K | 4 | 2,194 MB | 48s | This case is more meaningful because a full clone of the Linux repository is currently over 6 GB, so this is a valuable way to download a fraction of the repository and no longer need network access for all reachable objects within the sparse-checkout. Choosing a batch size will depend on a lot of factors, including the user's network speed or reliability, the repository's file structure, and how many versions there are of the file within the sparse-checkout scope. There will not be a one-size-fits-all solution. Signed-off-by: Derrick Stolee <stolee@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Derrick Stolee committed Feb 3, 2025 at 17:11 UTC bff455576750bd013a3c87b15cc7086cb8c1eab0

10 files changed +208 -15

Documentation/git-backfill.txt

+5 -1

@@ -9,7 +9,7 @@ git-backfill - Download missing objects in a partial clone

9 SYNOPSIS

10 --------

11 [synopsis]

12 -git backfill [--min-batch-size=<n>]

12 +git backfill [--min-batch-size=<n>] [--[no-]sparse]

14 DESCRIPTION

15 -----------

@@ -57,6 +57,10 @@ OPTIONS

57 blobs seen at a given path. The default minimum batch size is

58 50,000.

60 +`--[no-]sparse`::

61 + Only download objects if they appear at a path that matches the

62 + current sparse-checkout.

63 +

64 SEE ALSO

65 --------

66 linkgit:git-clone[1].

Documentation/technical/api-path-walk.txt

@@ -56,6 +56,14 @@ better off using the revision walk API instead.

56 the revision walk so that the walk emits commits marked with the

57 `UNINTERESTING` flag.

59 +`pl`::

60 + This pattern list pointer allows focusing the path-walk search to

61 + a set of patterns, only emitting paths that match the given

62 + patterns. See linkgit:gitignore[5] or

63 + linkgit:git-sparse-checkout[1] for details about pattern lists.

64 + When the pattern list uses cone-mode patterns, then the path-walk

65 + API can prune the set of paths it walks to improve performance.

66 +

67 Examples

68 --------

builtin/backfill.c

+14 -1

@@ -4,6 +4,7 @@

4 #include "parse-options.h"

5 #include "repository.h"

6 #include "commit.h"

7 +#include "dir.h"

8 #include "hex.h"

9 #include "tree.h"

10 #include "tree-walk.h"

@@ -21,7 +22,7 @@

22 #include "path-walk.h"

24 static const char * const builtin_backfill_usage[] = {

24 - N_("git backfill [--min-batch-size=<n>]"),

25 + N_("git backfill [--min-batch-size=<n>] [--[no-]sparse]"),

26 NULL

27 };

@@ -29,6 +30,7 @@ struct backfill_context {

30 struct repository *repo;

31 struct oid_array current_batch;

32 size_t min_batch_size;

33 + int sparse;

34 };

36 static void backfill_context_clear(struct backfill_context *ctx)

@@ -78,6 +80,14 @@ static int do_backfill(struct backfill_context *ctx)

80 struct path_walk_info info = PATH_WALK_INFO_INIT;

81 int ret;

83 + if (ctx->sparse) {

84 + CALLOC_ARRAY(info.pl, 1);

85 + if (get_sparse_checkout_patterns(info.pl)) {

86 + path_walk_info_clear(&info);

87 + return error(_("problem loading sparse-checkout"));

88 + }

89 + }

90 +

91 repo_init_revisions(ctx->repo, &revs, "");

92 handle_revision_arg("HEAD", &revs, 0, 0);

@@ -106,10 +116,13 @@ int cmd_backfill(int argc, const char **argv, const char *prefix, struct reposit

116 .repo = repo,

117 .current_batch = OID_ARRAY_INIT,

118 .min_batch_size = 50000,

119 + .sparse = 0,

120 };

121 struct option options[] = {

122 OPT_INTEGER(0, "min-batch-size", &ctx.min_batch_size,

123 N_("Minimum number of objects to request at a time")),

124 + OPT_BOOL(0, "sparse", &ctx.sparse,

125 + N_("Restrict the missing objects to the current sparse-checkout")),

126 OPT_END(),

127 };

128

dir.c

+3 -7

@@ -1093,10 +1093,6 @@ static void invalidate_directory(struct untracked_cache *uc,

1093 dir->dirs[i]->recurse = 0;

1094 }

1095

1096 -static int add_patterns_from_buffer(char *buf, size_t size,

1097 - const char *base, int baselen,

1098 - struct pattern_list *pl);

1099 -

1096 /* Flags for add_patterns() */

1097 #define PATTERN_NOFOLLOW (1<<0)

1098

@@ -1186,9 +1182,9 @@ static int add_patterns(const char *fname, const char *base, int baselen,

1182 return 0;

1183 }

1184

1189 -static int add_patterns_from_buffer(char *buf, size_t size,

1190 - const char *base, int baselen,

1191 - struct pattern_list *pl)

1185 +int add_patterns_from_buffer(char *buf, size_t size,

1186 + const char *base, int baselen,

1187 + struct pattern_list *pl)

1188 {

1189 char *orig = buf;

1190 int i, lineno = 1;

dir.h

@@ -467,6 +467,9 @@ void add_patterns_from_file(struct dir_struct *, const char *fname);

467 int add_patterns_from_blob_to_list(struct object_id *oid,

468 const char *base, int baselen,

469 struct pattern_list *pl);

470 +int add_patterns_from_buffer(char *buf, size_t size,

471 + const char *base, int baselen,

472 + struct pattern_list *pl);

473 void parse_path_pattern(const char **string, int *patternlen, unsigned *flags, int *nowildcardlen);

474 void add_pattern(const char *string, const char *base,

475 int baselen, struct pattern_list *pl, int srcpos);

path-walk.c

+23 -5

@@ -12,6 +12,7 @@

12 #include "object.h"

13 #include "oid-array.h"

14 #include "prio-queue.h"

15 +#include "repository.h"

16 #include "revision.h"

17 #include "string-list.h"

18 #include "strmap.h"

@@ -172,6 +173,23 @@ static int add_tree_entries(struct path_walk_context *ctx,

173 if (type == OBJ_TREE)

174 strbuf_addch(&path, '/');

175

176 + if (ctx->info->pl) {

177 + int dtype;

178 + enum pattern_match_result match;

179 + match = path_matches_pattern_list(path.buf, path.len,

180 + path.buf + base_len, &dtype,

181 + ctx->info->pl,

182 + ctx->repo->index);

183 +

184 + if (ctx->info->pl->use_cone_patterns &&

185 + match == NOT_MATCHED)

186 + continue;

187 + else if (!ctx->info->pl->use_cone_patterns &&

188 + type == OBJ_BLOB &&

189 + match != MATCHED)

190 + continue;

191 + }

192 +

193 if (!(list = strmap_get(&ctx->paths_to_lists, path.buf))) {

194 CALLOC_ARRAY(list, 1);

195 list->type = type;

@@ -582,10 +600,10 @@ void path_walk_info_init(struct path_walk_info *info)

600 memcpy(info, &empty, sizeof(empty));

601 }

602

585 -void path_walk_info_clear(struct path_walk_info *info UNUSED)

603 +void path_walk_info_clear(struct path_walk_info *info)

604 {

587 - /*

588 - * This destructor is empty for now, as info->revs

589 - * is not owned by 'struct path_walk_info'.

590 - */

605 + if (info->pl) {

606 + clear_pattern_list(info->pl);

607 + free(info->pl);

608 + }

609 }

path-walk.h

+11

@@ -6,6 +6,7 @@

7 struct rev_info;

8 struct oid_array;

9 +struct pattern_list;

11 /**

12 * The type of a function pointer for the method that is called on a list of

@@ -48,6 +49,16 @@ struct path_walk_info {

49 * walk the children of such trees.

50 */

51 int prune_all_uninteresting;

52 +

53 + /**

54 + * Specify a sparse-checkout definition to match our paths to. Do not

55 + * walk outside of this sparse definition. If the patterns are in

56 + * cone mode, then the search may prune directories that are outside

57 + * of the cone. If not in cone mode, then all tree paths will be

58 + * explored but the path_fn will only be called when the path matches

59 + * the sparse-checkout patterns.

60 + */

61 + struct pattern_list *pl;

62 };

64 #define PATH_WALK_INFO_INIT { \

t/helper/test-path-walk.c

+21 -1

@@ -1,6 +1,7 @@

1 #define USE_THE_REPOSITORY_VARIABLE

3 #include "test-tool.h"

4 +#include "dir.h"

5 #include "environment.h"

6 #include "hex.h"

7 #include "object-name.h"

@@ -9,6 +10,7 @@

10 #include "revision.h"

11 #include "setup.h"

12 #include "parse-options.h"

13 +#include "strbuf.h"

14 #include "path-walk.h"

15 #include "oid-array.h"

@@ -65,7 +67,7 @@ static int emit_block(const char *path, struct oid_array *oids,

68 int cmd__path_walk(int argc, const char **argv)

69 {

68 - int res;

70 + int res, stdin_pl = 0;

71 struct rev_info revs = REV_INFO_INIT;

72 struct path_walk_info info = PATH_WALK_INFO_INIT;

73 struct path_walk_test_data data = { 0 };

@@ -80,6 +82,8 @@ int cmd__path_walk(int argc, const char **argv)

82 N_("toggle inclusion of tree objects")),

83 OPT_BOOL(0, "prune", &info.prune_all_uninteresting,

84 N_("toggle pruning of uninteresting paths")),

85 + OPT_BOOL(0, "stdin-pl", &stdin_pl,

86 + N_("read a pattern list over stdin")),

87 OPT_END(),

88 };

@@ -99,6 +103,17 @@ int cmd__path_walk(int argc, const char **argv)

103 info.path_fn = emit_block;

104 info.path_fn_data = &data;

105

106 + if (stdin_pl) {

107 + struct strbuf in = STRBUF_INIT;

108 + CALLOC_ARRAY(info.pl, 1);

109 +

110 + info.pl->use_cone_patterns = 1;

111 +

112 + strbuf_fread(&in, 2048, stdin);

113 + add_patterns_from_buffer(in.buf, in.len, "", 0, info.pl);

114 + strbuf_release(&in);

115 + }

116 +

117 res = walk_objects_by_path(&info);

118

119 printf("commits:%" PRIuMAX "\n"

@@ -107,6 +122,11 @@ int cmd__path_walk(int argc, const char **argv)

122 "tags:%" PRIuMAX "\n",

123 data.commit_nr, data.tree_nr, data.blob_nr, data.tag_nr);

124

125 + if (info.pl) {

126 + clear_pattern_list(info.pl);

127 + free(info.pl);

128 + }

129 +

130 release_revisions(&revs);

131 return res;

132 }

t/t5620-backfill.sh

+88

@@ -77,6 +77,94 @@ test_expect_success 'do partial clone 2, backfill min batch size' '

77 test_line_count = 0 revs2

78 '

80 +test_expect_success 'backfill --sparse' '

81 + git clone --sparse --filter=blob:none \

82 + --single-branch --branch=main \

83 + "file://$(pwd)/srv.bare" backfill3 &&

84 +

85 + # Initial checkout includes four files at root.

86 + git -C backfill3 rev-list --quiet --objects --missing=print HEAD >missing &&

87 + test_line_count = 44 missing &&

88 +

89 + # Initial sparse-checkout is just the files at root, so we get the

90 + # older versions of the four files at tip.

91 + GIT_TRACE2_EVENT="$(pwd)/sparse-trace1" git \

92 + -C backfill3 backfill --sparse &&

93 + test_trace2_data promisor fetch_count 4 <sparse-trace1 &&

94 + test_trace2_data path-walk paths 5 <sparse-trace1 &&

95 + git -C backfill3 rev-list --quiet --objects --missing=print HEAD >missing &&

96 + test_line_count = 40 missing &&

97 +

98 + # Expand the sparse-checkout to include 'd' recursively. This

99 + # engages the algorithm to skip the trees for 'a'. Note that

100 + # the "sparse-checkout set" command downloads the objects at tip

101 + # to satisfy the current checkout.

102 + git -C backfill3 sparse-checkout set d &&

103 + GIT_TRACE2_EVENT="$(pwd)/sparse-trace2" git \

104 + -C backfill3 backfill --sparse &&

105 + test_trace2_data promisor fetch_count 8 <sparse-trace2 &&

106 + test_trace2_data path-walk paths 15 <sparse-trace2 &&

107 + git -C backfill3 rev-list --quiet --objects --missing=print HEAD >missing &&

108 + test_line_count = 24 missing

109 +'

110 +

111 +test_expect_success 'backfill --sparse without cone mode (positive)' '

112 + git clone --no-checkout --filter=blob:none \

113 + --single-branch --branch=main \

114 + "file://$(pwd)/srv.bare" backfill4 &&

115 +

116 + # No blobs yet

117 + git -C backfill4 rev-list --quiet --objects --missing=print HEAD >missing &&

118 + test_line_count = 48 missing &&

119 +

120 + # Define sparse-checkout by filename regardless of parent directory.

121 + # This downloads 6 blobs to satisfy the checkout.

122 + git -C backfill4 sparse-checkout set --no-cone "**/file.1.txt" &&

123 + git -C backfill4 checkout main &&

124 +

125 + # Track new blob count

126 + git -C backfill4 rev-list --quiet --objects --missing=print HEAD >missing &&

127 + test_line_count = 42 missing &&

128 +

129 + GIT_TRACE2_EVENT="$(pwd)/no-cone-trace1" git \

130 + -C backfill4 backfill --sparse &&

131 + test_trace2_data promisor fetch_count 6 <no-cone-trace1 &&

132 +

133 + # This walk needed to visit all directories to search for these paths.

134 + test_trace2_data path-walk paths 12 <no-cone-trace1 &&

135 + git -C backfill4 rev-list --quiet --objects --missing=print HEAD >missing &&

136 + test_line_count = 36 missing

137 +'

138 +

139 +test_expect_success 'backfill --sparse without cone mode (negative)' '

140 + git clone --no-checkout --filter=blob:none \

141 + --single-branch --branch=main \

142 + "file://$(pwd)/srv.bare" backfill5 &&

143 +

144 + # No blobs yet

145 + git -C backfill5 rev-list --quiet --objects --missing=print HEAD >missing &&

146 + test_line_count = 48 missing &&

147 +

148 + # Define sparse-checkout by filename regardless of parent directory.

149 + # This downloads 18 blobs to satisfy the checkout

150 + git -C backfill5 sparse-checkout set --no-cone "**/file*" "!**/file.1.txt" &&

151 + git -C backfill5 checkout main &&

152 +

153 + # Track new blob count

154 + git -C backfill5 rev-list --quiet --objects --missing=print HEAD >missing &&

155 + test_line_count = 30 missing &&

156 +

157 + GIT_TRACE2_EVENT="$(pwd)/no-cone-trace2" git \

158 + -C backfill5 backfill --sparse &&

159 + test_trace2_data promisor fetch_count 18 <no-cone-trace2 &&

160 +

161 + # This walk needed to visit all directories to search for these paths, plus

162 + # 12 extra "file.?.txt" paths than the previous test.

163 + test_trace2_data path-walk paths 24 <no-cone-trace2 &&

164 + git -C backfill5 rev-list --quiet --objects --missing=print HEAD >missing &&

165 + test_line_count = 12 missing

166 +'

167 +

168 . "$TEST_DIRECTORY"/lib-httpd.sh

169 start_httpd

170

t/t6601-path-walk.sh

+32

@@ -176,6 +176,38 @@ test_expect_success 'branches and indexed objects mix well' '

176 test_cmp_sorted expect out

177 '

178

179 +test_expect_success 'base & topic, sparse' '

180 + cat >patterns <<-EOF &&

181 + /*

182 + !/*/

183 + /left/

184 + EOF

185 +

186 + test-tool path-walk --stdin-pl -- base topic <patterns >out &&

187 +

188 + cat >expect <<-EOF &&

189 + 0:commit::$(git rev-parse topic)

190 + 0:commit::$(git rev-parse base)

191 + 0:commit::$(git rev-parse base~1)

192 + 0:commit::$(git rev-parse base~2)

193 + 1:tree::$(git rev-parse topic^{tree})

194 + 1:tree::$(git rev-parse base^{tree})

195 + 1:tree::$(git rev-parse base~1^{tree})

196 + 1:tree::$(git rev-parse base~2^{tree})

197 + 2:blob:a:$(git rev-parse base~2:a)

198 + 3:tree:left/:$(git rev-parse base:left)

199 + 3:tree:left/:$(git rev-parse base~2:left)

200 + 4:blob:left/b:$(git rev-parse base~2:left/b)

201 + 4:blob:left/b:$(git rev-parse base:left/b)

202 + blobs:3

203 + commits:4

204 + tags:0

205 + trees:6

206 + EOF

207 +

208 + test_cmp_sorted expect out

209 +'

210 +

211 test_expect_success 'topic only' '

212 test-tool path-walk -- topic >out &&

213