path-walk: introduce an object walk by path
In anticipation of a few planned applications, introduce the most basic form of a path-walk API. It currently assumes that there are no UNINTERESTING objects, and does not include any complicated filters. It calls a function pointer on groups of tree and blob objects as grouped by path. This only includes objects the first time they are discovered, so an object that appears at multiple paths will not be included in two batches. These batches are collected in 'struct type_and_oid_list' objects, which store an object type and an oid_array of objects. The data structures are documented in 'struct path_walk_context', but in summary the most important are: * 'paths_to_lists' is a strmap that connects a path to a type_and_oid_list for that path. To avoid conflicts in path names, we make sure that tree paths end in "/" (except the root path with is an empty string) and blob paths do not end in "/". * 'path_stack' is a string list that is added to in an append-only way. This stores the stack of our depth-first search on the heap instead of using recursion. * 'path_stack_pushed' is a strmap that stores path names that were already added to 'path_stack', to avoid repeating paths in the stack. Mostly, this saves us from quadratic lookups from doing unsorted checks into the string_list. The coupling of 'path_stack' and 'path_stack_pushed' is protected by the push_to_stack() method. Call this instead of inserting into these structures directly. The walk_objects_by_path() method initializes these structures and starts walking commits from the given rev_info struct. The commits are used to find the list of root trees which populate the start of our depth-first search. The core of our depth-first search is in a while loop that continues while we have not indicated an early exit and our 'path_stack' still has entries in it. The loop body pops a path off of the stack and "visits" the path via the walk_path() method. The walk_path() method gets the list of OIDs from the 'path_to_lists' strmap and executes the callback method on that list with the given path and type. If the OIDs correspond to tree objects, then iterate over all trees in the list and run add_children() to add the child objects to their own lists, adding new entries to the stack if necessary. In testing, this depth-first search approach was the one that used the least memory while iterating over the object lists. There is still a chance that repositories with too-wide path patterns could cause memory pressure issues. Limiting the stack size could be done in the future by limiting how many objects are being considered in-progress, or by visiting blob paths earlier than trees. There are many future adaptations that could be made, but they are left for future updates when consumers are ready to take advantage of those features. Signed-off-by: Derrick Stolee <stolee@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Derrick Stolee committed
Dec 20, 2024 at 16:21 UTC
9d46bc791b77e059deb740fde3439a0417a91b5e
4 files changed
+372
Documentation/technical/api-path-walk.txt
new
+45
@@ -0,0 +1,45 @@
1
+Path-Walk API
2
+=============
3
+
4
+The path-walk API is used to walk reachable objects, but to visit objects
5
+in batches based on a common path they appear in, or by type.
6
+
7
+For example, all reachable commits are visited in a group. All tags are
8
+visited in a group. Then, all root trees are visited. At some point, all
9
+blobs reachable via a path `my/dir/to/A` are visited. When there are
10
+multiple paths possible to reach the same object, then only one of those
11
+paths is used to visit the object.
12
+
13
+Basics
14
+------
15
+
16
+To use the path-walk API, include `path-walk.h` and call
17
+`walk_objects_by_path()` with a customized `path_walk_info` struct. The
18
+struct is used to set all of the options for how the walk should proceed.
19
+Let's dig into the different options and their use.
20
+
21
+`path_fn` and `path_fn_data`::
22
+ The most important option is the `path_fn` option, which is a
23
+ function pointer to the callback that can execute logic on the
24
+ object IDs for objects grouped by type and path. This function
25
+ also receives a `data` value that corresponds to the
26
+ `path_fn_data` member, for providing custom data structures to
27
+ this callback function.
28
+
29
+`revs`::
30
+ To configure the exact details of the reachable set of objects,
31
+ use the `revs` member and initialize it using the revision
32
+ machinery in `revision.h`. Initialize `revs` using calls such as
33
+ `setup_revisions()` or `parse_revision_opt()`. Do not call
34
+ `prepare_revision_walk()`, as that will be called within
35
+ `walk_objects_by_path()`.
36
++
37
+It is also important that you do not specify the `--objects` flag for the
38
+`revs` struct. The revision walk should only be used to walk commits, and
39
+the objects will be walked in a separate way based on those starting
40
+commits.
41
+
42
+Examples
43
+--------
44
+
45
+See example usages in future changes.
Makefile
+1
@@ -1098,6 +1098,7 @@ LIB_OBJS += parse-options.o
1098
LIB_OBJS += patch-delta.o
1099
LIB_OBJS += patch-ids.o
1100
LIB_OBJS += path.o
1101
+LIB_OBJS += path-walk.o
1102
LIB_OBJS += pathspec.o
1103
LIB_OBJS += pkt-line.o
1104
LIB_OBJS += preload-index.o
path-walk.c
new
+279
@@ -0,0 +1,279 @@
1
+/*
2
+ * path-walk.c: implementation for path-based walks of the object graph.
3
+ */
4
+#include "git-compat-util.h"
5
+#include "path-walk.h"
6
+#include "blob.h"
7
+#include "commit.h"
8
+#include "dir.h"
9
+#include "hashmap.h"
10
+#include "hex.h"
11
+#include "object.h"
12
+#include "oid-array.h"
13
+#include "revision.h"
14
+#include "string-list.h"
15
+#include "strmap.h"
16
+#include "trace2.h"
17
+#include "tree.h"
18
+#include "tree-walk.h"
19
+
20
+struct type_and_oid_list {
21
+ enum object_type type;
22
+ struct oid_array oids;
23
+};
24
+
25
+#define TYPE_AND_OID_LIST_INIT { \
26
+ .type = OBJ_NONE, \
27
+ .oids = OID_ARRAY_INIT \
28
+}
29
+
30
+struct path_walk_context {
31
+ /**
32
+ * Repeats of data in 'struct path_walk_info' for
33
+ * access with fewer characters.
34
+ */
35
+ struct repository *repo;
36
+ struct rev_info *revs;
37
+ struct path_walk_info *info;
38
+
39
+ /**
40
+ * Map a path to a 'struct type_and_oid_list'
41
+ * containing the objects discovered at that
42
+ * path.
43
+ */
44
+ struct strmap paths_to_lists;
45
+
46
+ /**
47
+ * Store the current list of paths in a stack, to
48
+ * facilitate depth-first-search without recursion.
49
+ *
50
+ * Use path_stack_pushed to indicate whether a path
51
+ * was previously added to path_stack.
52
+ */
53
+ struct string_list path_stack;
54
+ struct strset path_stack_pushed;
55
+};
56
+
57
+static void push_to_stack(struct path_walk_context *ctx,
58
+ const char *path)
59
+{
60
+ if (strset_contains(&ctx->path_stack_pushed, path))
61
+ return;
62
+
63
+ strset_add(&ctx->path_stack_pushed, path);
64
+ string_list_append(&ctx->path_stack, path);
65
+}
66
+
67
+static int add_tree_entries(struct path_walk_context *ctx,
68
+ const char *base_path,
69
+ struct object_id *oid)
70
+{
71
+ struct tree_desc desc;
72
+ struct name_entry entry;
73
+ struct strbuf path = STRBUF_INIT;
74
+ size_t base_len;
75
+ struct tree *tree = lookup_tree(ctx->repo, oid);
76
+
77
+ if (!tree) {
78
+ error(_("failed to walk children of tree %s: not found"),
79
+ oid_to_hex(oid));
80
+ return -1;
81
+ } else if (parse_tree_gently(tree, 1)) {
82
+ error("bad tree object %s", oid_to_hex(oid));
83
+ return -1;
84
+ }
85
+
86
+ strbuf_addstr(&path, base_path);
87
+ base_len = path.len;
88
+
89
+ parse_tree(tree);
90
+ init_tree_desc(&desc, &tree->object.oid, tree->buffer, tree->size);
91
+ while (tree_entry(&desc, &entry)) {
92
+ struct type_and_oid_list *list;
93
+ struct object *o;
94
+ /* Not actually true, but we will ignore submodules later. */
95
+ enum object_type type = S_ISDIR(entry.mode) ? OBJ_TREE : OBJ_BLOB;
96
+
97
+ /* Skip submodules. */
98
+ if (S_ISGITLINK(entry.mode))
99
+ continue;
100
+
101
+ if (type == OBJ_TREE) {
102
+ struct tree *child = lookup_tree(ctx->repo, &entry.oid);
103
+ o = child ? &child->object : NULL;
104
+ } else if (type == OBJ_BLOB) {
105
+ struct blob *child = lookup_blob(ctx->repo, &entry.oid);
106
+ o = child ? &child->object : NULL;
107
+ } else {
108
+ BUG("invalid type for tree entry: %d", type);
109
+ }
110
+
111
+ if (!o) {
112
+ error(_("failed to find object %s"),
113
+ oid_to_hex(&o->oid));
114
+ return -1;
115
+ }
116
+
117
+ /* Skip this object if already seen. */
118
+ if (o->flags & SEEN)
119
+ continue;
120
+ o->flags |= SEEN;
121
+
122
+ strbuf_setlen(&path, base_len);
123
+ strbuf_add(&path, entry.path, entry.pathlen);
124
+
125
+ /*
126
+ * Trees will end with "/" for concatenation and distinction
127
+ * from blobs at the same path.
128
+ */
129
+ if (type == OBJ_TREE)
130
+ strbuf_addch(&path, '/');
131
+
132
+ if (!(list = strmap_get(&ctx->paths_to_lists, path.buf))) {
133
+ CALLOC_ARRAY(list, 1);
134
+ list->type = type;
135
+ strmap_put(&ctx->paths_to_lists, path.buf, list);
136
+ }
137
+ push_to_stack(ctx, path.buf);
138
+ oid_array_append(&list->oids, &entry.oid);
139
+ }
140
+
141
+ free_tree_buffer(tree);
142
+ strbuf_release(&path);
143
+ return 0;
144
+}
145
+
146
+/*
147
+ * For each path in paths_to_explore, walk the trees another level
148
+ * and add any found blobs to the batch (but only if they exist and
149
+ * haven't been added yet).
150
+ */
151
+static int walk_path(struct path_walk_context *ctx,
152
+ const char *path)
153
+{
154
+ struct type_and_oid_list *list;
155
+ int ret = 0;
156
+
157
+ list = strmap_get(&ctx->paths_to_lists, path);
158
+
159
+ if (!list->oids.nr)
160
+ return 0;
161
+
162
+ /* Evaluate function pointer on this data. */
163
+ ret = ctx->info->path_fn(path, &list->oids, list->type,
164
+ ctx->info->path_fn_data);
165
+
166
+ /* Expand data for children. */
167
+ if (list->type == OBJ_TREE) {
168
+ for (size_t i = 0; i < list->oids.nr; i++) {
169
+ ret |= add_tree_entries(ctx,
170
+ path,
171
+ &list->oids.oid[i]);
172
+ }
173
+ }
174
+
175
+ oid_array_clear(&list->oids);
176
+ strmap_remove(&ctx->paths_to_lists, path, 1);
177
+ return ret;
178
+}
179
+
180
+static void clear_paths_to_lists(struct strmap *map)
181
+{
182
+ struct hashmap_iter iter;
183
+ struct strmap_entry *e;
184
+
185
+ hashmap_for_each_entry(&map->map, &iter, e, ent) {
186
+ struct type_and_oid_list *list = e->value;
187
+ oid_array_clear(&list->oids);
188
+ }
189
+ strmap_clear(map, 1);
190
+ strmap_init(map);
191
+}
192
+
193
+/**
194
+ * Given the configuration of 'info', walk the commits based on 'info->revs' and
195
+ * call 'info->path_fn' on each discovered path.
196
+ *
197
+ * Returns nonzero on an error.
198
+ */
199
+int walk_objects_by_path(struct path_walk_info *info)
200
+{
201
+ const char *root_path = "";
202
+ int ret = 0;
203
+ size_t commits_nr = 0, paths_nr = 0;
204
+ struct commit *c;
205
+ struct type_and_oid_list *root_tree_list;
206
+ struct path_walk_context ctx = {
207
+ .repo = info->revs->repo,
208
+ .revs = info->revs,
209
+ .info = info,
210
+ .path_stack = STRING_LIST_INIT_DUP,
211
+ .path_stack_pushed = STRSET_INIT,
212
+ .paths_to_lists = STRMAP_INIT
213
+ };
214
+
215
+ trace2_region_enter("path-walk", "commit-walk", info->revs->repo);
216
+
217
+ /* Insert a single list for the root tree into the paths. */
218
+ CALLOC_ARRAY(root_tree_list, 1);
219
+ root_tree_list->type = OBJ_TREE;
220
+ strmap_put(&ctx.paths_to_lists, root_path, root_tree_list);
221
+ push_to_stack(&ctx, root_path);
222
+
223
+ if (prepare_revision_walk(info->revs))
224
+ die(_("failed to setup revision walk"));
225
+
226
+ while ((c = get_revision(info->revs))) {
227
+ struct object_id *oid = get_commit_tree_oid(c);
228
+ struct tree *t;
229
+ commits_nr++;
230
+
231
+ oid = get_commit_tree_oid(c);
232
+ t = lookup_tree(info->revs->repo, oid);
233
+
234
+ if (!t) {
235
+ error("could not find tree %s", oid_to_hex(oid));
236
+ return -1;
237
+ }
238
+
239
+ if (t->object.flags & SEEN)
240
+ continue;
241
+ t->object.flags |= SEEN;
242
+ oid_array_append(&root_tree_list->oids, oid);
243
+ }
244
+
245
+ trace2_data_intmax("path-walk", ctx.repo, "commits", commits_nr);
246
+ trace2_region_leave("path-walk", "commit-walk", info->revs->repo);
247
+
248
+ trace2_region_enter("path-walk", "path-walk", info->revs->repo);
249
+ while (!ret && ctx.path_stack.nr) {
250
+ char *path = ctx.path_stack.items[ctx.path_stack.nr - 1].string;
251
+ ctx.path_stack.nr--;
252
+ paths_nr++;
253
+
254
+ ret = walk_path(&ctx, path);
255
+
256
+ free(path);
257
+ }
258
+ trace2_data_intmax("path-walk", ctx.repo, "paths", paths_nr);
259
+ trace2_region_leave("path-walk", "path-walk", info->revs->repo);
260
+
261
+ clear_paths_to_lists(&ctx.paths_to_lists);
262
+ strset_clear(&ctx.path_stack_pushed);
263
+ string_list_clear(&ctx.path_stack, 0);
264
+ return ret;
265
+}
266
+
267
+void path_walk_info_init(struct path_walk_info *info)
268
+{
269
+ struct path_walk_info empty = PATH_WALK_INFO_INIT;
270
+ memcpy(info, &empty, sizeof(empty));
271
+}
272
+
273
+void path_walk_info_clear(struct path_walk_info *info UNUSED)
274
+{
275
+ /*
276
+ * This destructor is empty for now, as info->revs
277
+ * is not owned by 'struct path_walk_info'.
278
+ */
279
+}
path-walk.h
new
+47
@@ -0,0 +1,47 @@
1
+/*
2
+ * path-walk.h : Methods and structures for walking the object graph in batches
3
+ * by the paths that can reach those objects.
4
+ */
5
+#include "object.h" /* Required for 'enum object_type'. */
6
+
7
+struct rev_info;
8
+struct oid_array;
9
+
10
+/**
11
+ * The type of a function pointer for the method that is called on a list of
12
+ * objects reachable at a given path.
13
+ */
14
+typedef int (*path_fn)(const char *path,
15
+ struct oid_array *oids,
16
+ enum object_type type,
17
+ void *data);
18
+
19
+struct path_walk_info {
20
+ /**
21
+ * revs provides the definitions for the commit walk, including
22
+ * which commits are UNINTERESTING or not. This structure is
23
+ * expected to be owned by the caller.
24
+ */
25
+ struct rev_info *revs;
26
+
27
+ /**
28
+ * The caller wishes to execute custom logic on objects reachable at a
29
+ * given path. Every reachable object will be visited exactly once, and
30
+ * the first path to see an object wins. This may not be a stable choice.
31
+ */
32
+ path_fn path_fn;
33
+ void *path_fn_data;
34
+};
35
+
36
+#define PATH_WALK_INFO_INIT { 0 }
37
+
38
+void path_walk_info_init(struct path_walk_info *info);
39
+void path_walk_info_clear(struct path_walk_info *info);
40
+
41
+/**
42
+ * Given the configuration of 'info', walk the commits based on 'info->revs' and
43
+ * call 'info->path_fn' on each discovered path.
44
+ *
45
+ * Returns nonzero on an error.
46
+ */
47
+int walk_objects_by_path(struct path_walk_info *info);