@matheusbsbs / matheus-bagliano / commits / faf558b23e

pseudo-merge: implement support for selecting pseudo-merge commits

Teach the new pseudo-merge machinery how to select non-bitmapped commits for inclusion in different pseudo-merge group(s) based on a handful of criteria. Note that the selected pseudo-merge commits aren't actually used or written anywhere yet. This will be done in the following commit. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Taylor Blau committed May 23, 2024 at 17:26 UTC faf558b23ef55b18f395d1f7a7c2714ccc1320e2
7 files changed +747
Documentation/config.txt
+2
@@ -383,6 +383,8 @@ include::config/apply.txt[]
383
384 include::config/attr.txt[]
385
386 +include::config/bitmap-pseudo-merge.txt[]
387 +
388 include::config/blame.txt[]
389
390 include::config/branch.txt[]
Documentation/config/bitmap-pseudo-merge.txt new
+91
@@ -0,0 +1,91 @@
1 +NOTE: The configuration options in `bitmapPseudoMerge.*` are considered
2 +EXPERIMENTAL and may be subject to change or be removed entirely in the
3 +future. For more information about the pseudo-merge bitmap feature, see
4 +the "Pseudo-merge bitmaps" section of linkgit:gitpacking[7].
5 +
6 +bitmapPseudoMerge.<name>.pattern::
7 + Regular expression used to match reference names. Commits
8 + pointed to by references matching this pattern (and meeting
9 + the below criteria, like `bitmapPseudoMerge.<name>.sampleRate`
10 + and `bitmapPseudoMerge.<name>.threshold`) will be considered
11 + for inclusion in a pseudo-merge bitmap.
12 ++
13 +Commits are grouped into pseudo-merge groups based on whether or not
14 +any reference(s) that point at a given commit match the pattern, which
15 +is an extended regular expression.
16 ++
17 +Within a pseudo-merge group, commits may be further grouped into
18 +sub-groups based on the capture groups in the pattern. These
19 +sub-groupings are formed from the regular expressions by concatenating
20 +any capture groups from the regular expression, with a '-' dash in
21 +between.
22 ++
23 +For example, if the pattern is `refs/tags/`, then all tags (provided
24 +they meet the below criteria) will be considered candidates for the
25 +same pseudo-merge group. However, if the pattern is instead
26 +`refs/remotes/([0-9])+/tags/`, then tags from different remotes will
27 +be grouped into separate pseudo-merge groups, based on the remote
28 +number.
29 +
30 +bitmapPseudoMerge.<name>.decay::
31 + Determines the rate at which consecutive pseudo-merge bitmap
32 + groups decrease in size. Must be non-negative. This parameter
33 + can be thought of as `k` in the function `f(n) = C * n^-k`,
34 + where `f(n)` is the size of the `n`th group.
35 ++
36 +Setting the decay rate equal to `0` will cause all groups to be the
37 +same size. Setting the decay rate equal to `1` will cause the `n`th
38 +group to be `1/n` the size of the initial group. Higher values of the
39 +decay rate cause consecutive groups to shrink at an increasing rate.
40 +The default is `1`.
41 ++
42 +If all groups are the same size, it is possible that groups containing
43 +newer commits will be able to be used less often than earlier groups,
44 +since it is more likely that the references pointing at newer commits
45 +will be updated more often than a reference pointing at an old commit.
46 +
47 +bitmapPseudoMerge.<name>.sampleRate::
48 + Determines the proportion of non-bitmapped commits (among
49 + reference tips) which are selected for inclusion in an
50 + unstable pseudo-merge bitmap. Must be between `0` and `1`
51 + (inclusive). The default is `1`.
52 +
53 +bitmapPseudoMerge.<name>.threshold::
54 + Determines the minimum age of non-bitmapped commits (among
55 + reference tips, as above) which are candidates for inclusion
56 + in an unstable pseudo-merge bitmap. The default is
57 + `1.week.ago`.
58 +
59 +bitmapPseudoMerge.<name>.maxMerges::
60 + Determines the maximum number of pseudo-merge commits among
61 + which commits may be distributed.
62 ++
63 +For pseudo-merge groups whose pattern does not contain any capture
64 +groups, this setting is applied for all commits matching the regular
65 +expression. For patterns that have one or more capture groups, this
66 +setting is applied for each distinct capture group.
67 ++
68 +For example, if your capture group is `refs/tags/`, then this setting
69 +will distribute all tags into a maximum of `maxMerges` pseudo-merge
70 +commits. However, if your capture group is, say,
71 +`refs/remotes/([0-9]+)/tags/`, then this setting will be applied to
72 +each remote's set of tags individually.
73 ++
74 +Must be non-negative. The default value is 64.
75 +
76 +bitmapPseudoMerge.<name>.stableThreshold::
77 + Determines the minimum age of commits (among reference tips,
78 + as above, however stable commits are still considered
79 + candidates even when they have been covered by a bitmap) which
80 + are candidates for a stable a pseudo-merge bitmap. The default
81 + is `1.month.ago`.
82 ++
83 +Setting this threshold to a smaller value (e.g., 1.week.ago) will cause
84 +more stable groups to be generated (which impose a one-time generation
85 +cost) but those groups will likely become stale over time. Using a
86 +larger value incurs the opposite penalty (fewer stable groups which are
87 +more useful).
88 +
89 +bitmapPseudoMerge.<name>.stableSize::
90 + Determines the size (in number of commits) of a stable
91 + psuedo-merge bitmap. The default is `512`.
Documentation/gitpacking.txt
+83
@@ -96,6 +96,89 @@ can take advantage of the fact that we only care about the union of
96 objects reachable from all of those tags, and answer the query much
97 faster.
98
99 +=== Configuration
100 +
101 +Reference tips are grouped into different pseudo-merge groups according
102 +to two criteria. A reference name matches one or more of the defined
103 +pseudo-merge patterns, and optionally one or more capture groups within
104 +that pattern which further partition the group.
105 +
106 +Within a group, commits may be considered "stable", or "unstable"
107 +depending on their age. These are adjusted by setting the
108 +`bitmapPseudoMerge.<name>.stableThreshold` and
109 +`bitmapPseudoMerge.<name>.threshold` configuration values, respectively.
110 +
111 +All stable commits are grouped into pseudo-merges of equal size
112 +(`bitmapPseudoMerge.<name>.stableSize`). If the `stableSize`
113 +configuration is set to, say, 100, then the first 100 commits (ordered
114 +by committer date) which are older than the `stableThreshold` value will
115 +form one group, the next 100 commits will form another group, and so on.
116 +
117 +Among unstable commits, the pseudo-merge machinery will attempt to
118 +combine older commits into large groups as opposed to newer commits
119 +which will appear in smaller groups. This is based on the heuristic that
120 +references whose tip commit is older are less likely to be modified to
121 +point at a different commit than a reference whose tip commit is newer.
122 +
123 +The size of groups is determined by a power-law decay function, and the
124 +decay parameter roughly corresponds to "k" in `f(n) = C*n^(-k/100)`,
125 +where `f(n)` describes the size of the `n`-th pseudo-merge group. The
126 +sample rate controls what percentage of eligible commits are considered
127 +as candidates. The threshold parameter indicates the minimum age (so as
128 +to avoid including too-recent commits in a pseudo-merge group, making it
129 +less likely to be valid). The "maxMerges" parameter sets an upper-bound
130 +on the number of pseudo-merge commits an individual group
131 +
132 +The "stable"-related parameters control "stable" pseudo-merge groups,
133 +comprised of a fixed number of commits which are older than the
134 +configured "stable threshold" value and may be grouped together in
135 +chunks of "stableSize" in order of age.
136 +
137 +The exact configuration for pseudo-merges is as follows:
138 +
139 +include::config/bitmap-pseudo-merge.txt[]
140 +
141 +=== Examples
142 +
143 +Suppose that you have a repository with a large number of references,
144 +and you want a bare-bones configuration of pseudo-merge bitmaps that
145 +will enhance bitmap coverage of the `refs/` namespace. You may start
146 +wiht a configuration like so:
147 +
148 + [bitmapPseudoMerge "all"]
149 + pattern = "refs/"
150 + threshold = now
151 + stableThreshold = never
152 + sampleRate = 100
153 + maxMerges = 64
154 +
155 +This will create pseudo-merge bitmaps for all references, regardless of
156 +their age, and group them into 64 pseudo-merge commits.
157 +
158 +If you wanted to separate tags from branches when generating
159 +pseudo-merge commits, you would instead define the pattern with a
160 +capture group, like so:
161 +
162 + [bitmapPseudoMerge "all"]
163 + pattern = "refs/(heads/tags)/"
164 +
165 +Suppose instead that you are working in a fork-network repository, with
166 +each fork specified by some numeric ID, and whose refs reside in
167 +`refs/virtual/NNN/` (where `NNN` is the numeric ID corresponding to some
168 +fork) in the network. In this instance, you may instead write something
169 +like:
170 +
171 + [bitmapPseudoMerge "all"]
172 + pattern = "refs/virtual/([0-9]+)/(heads|tags)/"
173 + threshold = now
174 + stableThreshold = never
175 + sampleRate = 100
176 + maxMerges = 64
177 +
178 +Which would generate pseudo-merge group identifiers like "1234-heads",
179 +and "5678-tags" (for branches in fork "1234", and tags in remote "5678",
180 +respectively).
181 +
182 SEE ALSO
183 --------
184 linkgit:git-pack-objects[1]
pack-bitmap-write.c
+21
@@ -17,6 +17,7 @@
17 #include "trace2.h"
18 #include "tree.h"
19 #include "tree-walk.h"
20 +#include "pseudo-merge.h"
21
22 struct bitmapped_commit {
23 struct commit *commit;
@@ -39,11 +40,25 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)
40 if (writer->bitmaps)
41 BUG("bitmap writer already initialized");
42 writer->bitmaps = kh_init_oid_map();
43 + writer->pseudo_merge_commits = kh_init_oid_map();
44 +
45 + string_list_init_dup(&writer->pseudo_merge_groups);
46 +
47 + load_pseudo_merges_from_config(&writer->pseudo_merge_groups);
48 +}
49 +
50 +static void free_pseudo_merge_commit_idx(struct pseudo_merge_commit_idx *idx)
51 +{
52 + if (!idx)
53 + return;
54 + free(idx->pseudo_merge);
55 + free(idx);
56 }
57
58 void bitmap_writer_free(struct bitmap_writer *writer)
59 {
60 uint32_t i;
61 + struct pseudo_merge_commit_idx *idx;
62
63 if (!writer)
64 return;
@@ -55,6 +70,10 @@ void bitmap_writer_free(struct bitmap_writer *writer)
70
71 kh_destroy_oid_map(writer->bitmaps);
72
73 + kh_foreach_value(writer->pseudo_merge_commits, idx,
74 + free_pseudo_merge_commit_idx(idx));
75 + kh_destroy_oid_map(writer->pseudo_merge_commits);
76 +
77 for (i = 0; i < writer->selected_nr; i++) {
78 struct bitmapped_commit *bc = &writer->selected[i];
79 if (bc->write_as != bc->bitmap)
@@ -703,6 +722,8 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,
722 }
723
724 stop_progress(&writer->progress);
725 +
726 + select_pseudo_merges(writer, indexed_commits, indexed_commits_nr);
727 }
728
729
pack-bitmap.h
+2
@@ -110,6 +110,8 @@ struct bitmap_writer {
110 struct bitmapped_commit *selected;
111 unsigned int selected_nr, selected_alloc;
112
113 + struct string_list pseudo_merge_groups;
114 + kh_oid_map_t *pseudo_merge_commits; /* oid -> pseudo merge(s) */
115 uint32_t pseudo_merges_nr;
116
117 struct progress *progress;
pseudo-merge.c
+454
@@ -1,2 +1,456 @@
1 #include "git-compat-util.h"
2 #include "pseudo-merge.h"
3 +#include "date.h"
4 +#include "oid-array.h"
5 +#include "strbuf.h"
6 +#include "config.h"
7 +#include "string-list.h"
8 +#include "refs.h"
9 +#include "pack-bitmap.h"
10 +#include "commit.h"
11 +#include "alloc.h"
12 +#include "progress.h"
13 +
14 +#define DEFAULT_PSEUDO_MERGE_DECAY 1.0
15 +#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64
16 +#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 1
17 +#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate("1.week.ago")
18 +#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate("1.month.ago")
19 +#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512
20 +
21 +static double gitexp(double base, int exp)
22 +{
23 + double result = 1;
24 + while (1) {
25 + if (exp % 2)
26 + result *= base;
27 + exp >>= 1;
28 + if (!exp)
29 + break;
30 + base *= base;
31 + }
32 + return result;
33 +}
34 +
35 +static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group,
36 + const struct pseudo_merge_matches *matches,
37 + uint32_t i)
38 +{
39 + double C = 0.0f;
40 + uint32_t n;
41 +
42 + /*
43 + * The size of pseudo-merge groups decays according to a power series,
44 + * which looks like:
45 + *
46 + * f(n) = C * n^-k
47 + *
48 + * , where 'n' is the n-th pseudo-merge group, 'f(n)' is its size, 'k'
49 + * is the decay rate, and 'C' is a scaling value.
50 + *
51 + * The value of C depends on the number of groups, decay rate, and total
52 + * number of commits. It is computed such that if there are M and N
53 + * total groups and commits, respectively, that:
54 + *
55 + * N = f(0) + f(1) + ... f(M-1)
56 + *
57 + * Rearranging to isolate C, we get:
58 + *
59 + * N = \sum_{n=1}^M C / n^k
60 + *
61 + * N / C = \sum_{n=1}^M n^-k
62 + *
63 + * C = N / \sum_{n=1}^M n^-k
64 + *
65 + * For example, if we have a decay rate of 'k' being equal to 1.5, 'N'
66 + * total commits equal to 10,000, and 'M' being equal to 6 groups, then
67 + * the (rounded) group sizes are:
68 + *
69 + * { 5469, 1934, 1053, 684, 489, 372 }
70 + *
71 + * increasing the number of total groups, say to 10, scales the group
72 + * sizes appropriately:
73 + *
74 + * { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 }
75 + */
76 + for (n = 0; n < group->max_merges; n++)
77 + C += 1.0 / gitexp(n + 1, group->decay);
78 + C = matches->unstable_nr / C;
79 +
80 + return (uint32_t)((C / gitexp(i + 1, group->decay)) + 0.5);
81 +}
82 +
83 +static void pseudo_merge_group_init(struct pseudo_merge_group *group)
84 +{
85 + memset(group, 0, sizeof(struct pseudo_merge_group));
86 +
87 + strmap_init_with_options(&group->matches, NULL, 0);
88 +
89 + group->decay = DEFAULT_PSEUDO_MERGE_DECAY;
90 + group->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;
91 + group->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;
92 + group->threshold = DEFAULT_PSEUDO_MERGE_THRESHOLD;
93 + group->stable_threshold = DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD;
94 + group->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;
95 +}
96 +
97 +static int pseudo_merge_config(const char *var, const char *value,
98 + const struct config_context *ctx,
99 + void *cb_data)
100 +{
101 + struct string_list *list = cb_data;
102 + struct string_list_item *item;
103 + struct pseudo_merge_group *group;
104 + struct strbuf buf = STRBUF_INIT;
105 + const char *sub, *key;
106 + size_t sub_len;
107 + int ret = 0;
108 +
109 + if (parse_config_key(var, "bitmappseudomerge", &sub, &sub_len, &key))
110 + goto done;
111 +
112 + if (!sub_len)
113 + goto done;
114 +
115 + strbuf_add(&buf, sub, sub_len);
116 +
117 + item = string_list_lookup(list, buf.buf);
118 + if (!item) {
119 + item = string_list_insert(list, buf.buf);
120 +
121 + item->util = xmalloc(sizeof(struct pseudo_merge_group));
122 + pseudo_merge_group_init(item->util);
123 + }
124 +
125 + group = item->util;
126 +
127 + if (!strcmp(key, "pattern")) {
128 + struct strbuf re = STRBUF_INIT;
129 +
130 + free(group->pattern);
131 + if (*value != '^')
132 + strbuf_addch(&re, '^');
133 + strbuf_addstr(&re, value);
134 +
135 + group->pattern = xcalloc(1, sizeof(regex_t));
136 + if (regcomp(group->pattern, re.buf, REG_EXTENDED))
137 + die(_("failed to load pseudo-merge regex for %s: '%s'"),
138 + sub, re.buf);
139 +
140 + strbuf_release(&re);
141 + } else if (!strcmp(key, "decay")) {
142 + group->decay = git_config_double(var, value, ctx->kvi);
143 + if (group->decay < 0) {
144 + warning(_("%s must be non-negative, using default"), var);
145 + group->decay = DEFAULT_PSEUDO_MERGE_DECAY;
146 + }
147 + } else if (!strcmp(key, "samplerate")) {
148 + group->sample_rate = git_config_double(var, value, ctx->kvi);
149 + if (!(0 <= group->sample_rate && group->sample_rate <= 1)) {
150 + warning(_("%s must be between 0 and 1, using default"), var);
151 + group->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;
152 + }
153 + } else if (!strcmp(key, "threshold")) {
154 + if (git_config_expiry_date(&group->threshold, var, value)) {
155 + ret = -1;
156 + goto done;
157 + }
158 + } else if (!strcmp(key, "maxmerges")) {
159 + group->max_merges = git_config_int(var, value, ctx->kvi);
160 + if (group->max_merges < 0) {
161 + warning(_("%s must be non-negative, using default"), var);
162 + group->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;
163 + }
164 + } else if (!strcmp(key, "stablethreshold")) {
165 + if (git_config_expiry_date(&group->stable_threshold, var, value)) {
166 + ret = -1;
167 + goto done;
168 + }
169 + } else if (!strcmp(key, "stablesize")) {
170 + group->stable_size = git_config_int(var, value, ctx->kvi);
171 + if (group->stable_size <= 0) {
172 + warning(_("%s must be positive, using default"), var);
173 + group->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;
174 + }
175 + }
176 +
177 +done:
178 + strbuf_release(&buf);
179 +
180 + return ret;
181 +}
182 +
183 +void load_pseudo_merges_from_config(struct string_list *list)
184 +{
185 + struct string_list_item *item;
186 +
187 + git_config(pseudo_merge_config, list);
188 +
189 + for_each_string_list_item(item, list) {
190 + struct pseudo_merge_group *group = item->util;
191 + if (!group->pattern)
192 + die(_("pseudo-merge group '%s' missing required pattern"),
193 + item->string);
194 + if (group->threshold < group->stable_threshold)
195 + die(_("pseudo-merge group '%s' has unstable threshold "
196 + "before stable one"), item->string);
197 + }
198 +}
199 +
200 +static int find_pseudo_merge_group_for_ref(const char *refname,
201 + const struct object_id *oid,
202 + int flags UNUSED,
203 + void *_data)
204 +{
205 + struct bitmap_writer *writer = _data;
206 + struct object_id peeled;
207 + struct commit *c;
208 + uint32_t i;
209 + int has_bitmap;
210 +
211 + if (!peel_iterated_oid(oid, &peeled))
212 + oid = &peeled;
213 +
214 + c = lookup_commit(the_repository, oid);
215 + if (!c)
216 + return 0;
217 +
218 + has_bitmap = bitmap_writer_has_bitmapped_object_id(writer, oid);
219 +
220 + for (i = 0; i < writer->pseudo_merge_groups.nr; i++) {
221 + struct pseudo_merge_group *group;
222 + struct pseudo_merge_matches *matches;
223 + struct strbuf group_name = STRBUF_INIT;
224 + regmatch_t captures[16];
225 + size_t j;
226 +
227 + group = writer->pseudo_merge_groups.items[i].util;
228 + if (regexec(group->pattern, refname, ARRAY_SIZE(captures),
229 + captures, 0))
230 + continue;
231 +
232 + if (captures[ARRAY_SIZE(captures) - 1].rm_so != -1)
233 + warning(_("pseudo-merge regex from config has too many capture "
234 + "groups (max=%"PRIuMAX")"),
235 + (uintmax_t)ARRAY_SIZE(captures) - 2);
236 +
237 + for (j = !!group->pattern->re_nsub; j < ARRAY_SIZE(captures); j++) {
238 + regmatch_t *match = &captures[j];
239 + if (match->rm_so == -1)
240 + continue;
241 +
242 + if (group_name.len)
243 + strbuf_addch(&group_name, '-');
244 +
245 + strbuf_add(&group_name, refname + match->rm_so,
246 + match->rm_eo - match->rm_so);
247 + }
248 +
249 + matches = strmap_get(&group->matches, group_name.buf);
250 + if (!matches) {
251 + matches = xcalloc(1, sizeof(*matches));
252 + strmap_put(&group->matches, strbuf_detach(&group_name, NULL),
253 + matches);
254 + }
255 +
256 + if (c->date <= group->stable_threshold) {
257 + ALLOC_GROW(matches->stable, matches->stable_nr + 1,
258 + matches->stable_alloc);
259 + matches->stable[matches->stable_nr++] = c;
260 + } else if (c->date <= group->threshold && !has_bitmap) {
261 + ALLOC_GROW(matches->unstable, matches->unstable_nr + 1,
262 + matches->unstable_alloc);
263 + matches->unstable[matches->unstable_nr++] = c;
264 + }
265 +
266 + strbuf_release(&group_name);
267 + }
268 +
269 + return 0;
270 +}
271 +
272 +static struct commit *push_pseudo_merge(struct pseudo_merge_group *group)
273 +{
274 + struct commit *merge;
275 +
276 + ALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc);
277 +
278 + merge = alloc_commit_node(the_repository);
279 + merge->object.parsed = 1;
280 + merge->object.flags |= BITMAP_PSEUDO_MERGE;
281 +
282 + group->merges[group->merges_nr++] = merge;
283 +
284 + return merge;
285 +}
286 +
287 +static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,
288 + const struct object_id *oid)
289 +
290 +{
291 + struct pseudo_merge_commit_idx *pmc;
292 + int hash_ret;
293 + khiter_t hash_pos = kh_put_oid_map(pseudo_merge_commits, *oid,
294 + &hash_ret);
295 +
296 + if (hash_ret) {
297 + CALLOC_ARRAY(pmc, 1);
298 + kh_value(pseudo_merge_commits, hash_pos) = pmc;
299 + } else {
300 + pmc = kh_value(pseudo_merge_commits, hash_pos);
301 + }
302 +
303 + return pmc;
304 +}
305 +
306 +#define MIN_PSEUDO_MERGE_SIZE 8
307 +
308 +static void select_pseudo_merges_1(struct bitmap_writer *writer,
309 + struct pseudo_merge_group *group,
310 + struct pseudo_merge_matches *matches)
311 +{
312 + uint32_t i, j;
313 + uint32_t stable_merges_nr;
314 +
315 + if (!matches->stable_nr && !matches->unstable_nr)
316 + return; /* all tips in this group already have bitmaps */
317 +
318 + stable_merges_nr = matches->stable_nr / group->stable_size;
319 + if (matches->stable_nr % group->stable_size)
320 + stable_merges_nr++;
321 +
322 + /* make stable_merges_nr pseudo merges for stable commits */
323 + for (i = 0, j = 0; i < stable_merges_nr; i++) {
324 + struct commit *merge;
325 + struct commit_list **p;
326 +
327 + merge = push_pseudo_merge(group);
328 + p = &merge->parents;
329 +
330 + /*
331 + * For each pseudo-merge created above, add parents to the
332 + * allocated commit node from the stable set of commits
333 + * (un-bitmapped, newer than the stable threshold).
334 + */
335 + do {
336 + struct commit *c;
337 + struct pseudo_merge_commit_idx *pmc;
338 +
339 + if (j >= matches->stable_nr)
340 + break;
341 +
342 + c = matches->stable[j++];
343 + /*
344 + * Here and below, make sure that we keep our mapping of
345 + * commits -> pseudo-merge(s) which include the key'd
346 + * commit up-to-date.
347 + */
348 + pmc = pseudo_merge_idx(writer->pseudo_merge_commits,
349 + &c->object.oid);
350 +
351 + ALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);
352 +
353 + pmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr;
354 + p = commit_list_append(c, p);
355 + } while (j % group->stable_size);
356 +
357 + bitmap_writer_push_commit(writer, merge, 1);
358 + writer->pseudo_merges_nr++;
359 + }
360 +
361 + /* make up to group->max_merges pseudo merges for unstable commits */
362 + for (i = 0, j = 0; i < group->max_merges; i++) {
363 + struct commit *merge;
364 + struct commit_list **p;
365 + uint32_t size, end;
366 +
367 + merge = push_pseudo_merge(group);
368 + p = &merge->parents;
369 +
370 + size = pseudo_merge_group_size(group, matches, i);
371 + end = size < MIN_PSEUDO_MERGE_SIZE ? matches->unstable_nr : j + size;
372 +
373 + /*
374 + * For each pseudo-merge commit created above, add parents to
375 + * the allocated commit node from the unstable set of commits
376 + * (newer than the stable threshold).
377 + *
378 + * Account for the sample rate, since not every candidate from
379 + * the set of stable commits will be included as a pseudo-merge
380 + * parent.
381 + */
382 + for (; j < end && j < matches->unstable_nr; j++) {
383 + struct commit *c = matches->unstable[j];
384 + struct pseudo_merge_commit_idx *pmc;
385 +
386 + if (j % (uint32_t)(1.0 / group->sample_rate))
387 + continue;
388 +
389 + pmc = pseudo_merge_idx(writer->pseudo_merge_commits,
390 + &c->object.oid);
391 +
392 + ALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);
393 +
394 + pmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr;
395 + p = commit_list_append(c, p);
396 + }
397 +
398 + bitmap_writer_push_commit(writer, merge, 1);
399 + writer->pseudo_merges_nr++;
400 + if (end >= matches->unstable_nr)
401 + break;
402 + }
403 +}
404 +
405 +static int commit_date_cmp(const void *va, const void *vb)
406 +{
407 + timestamp_t a = (*(const struct commit **)va)->date;
408 + timestamp_t b = (*(const struct commit **)vb)->date;
409 +
410 + if (a < b)
411 + return -1;
412 + else if (a > b)
413 + return 1;
414 + return 0;
415 +}
416 +
417 +static void sort_pseudo_merge_matches(struct pseudo_merge_matches *matches)
418 +{
419 + QSORT(matches->stable, matches->stable_nr, commit_date_cmp);
420 + QSORT(matches->unstable, matches->unstable_nr, commit_date_cmp);
421 +}
422 +
423 +void select_pseudo_merges(struct bitmap_writer *writer,
424 + struct commit **commits, size_t commits_nr)
425 +{
426 + struct progress *progress = NULL;
427 + uint32_t i;
428 +
429 + if (!writer->pseudo_merge_groups.nr)
430 + return;
431 +
432 + if (writer->show_progress)
433 + progress = start_progress("Selecting pseudo-merge commits",
434 + writer->pseudo_merge_groups.nr);
435 +
436 + for_each_ref(find_pseudo_merge_group_for_ref, writer);
437 +
438 + for (i = 0; i < writer->pseudo_merge_groups.nr; i++) {
439 + struct pseudo_merge_group *group;
440 + struct hashmap_iter iter;
441 + struct strmap_entry *e;
442 +
443 + group = writer->pseudo_merge_groups.items[i].util;
444 + strmap_for_each_entry(&group->matches, &iter, e) {
445 + struct pseudo_merge_matches *matches = e->value;
446 +
447 + sort_pseudo_merge_matches(matches);
448 +
449 + select_pseudo_merges_1(writer, group, matches);
450 + }
451 +
452 + display_progress(progress, i + 1);
453 + }
454 +
455 + stop_progress(&progress);
456 +}
pseudo-merge.h
+94
@@ -2,5 +2,99 @@
2 #define PSEUDO_MERGE_H
3
4 #include "git-compat-util.h"
5 +#include "strmap.h"
6 +#include "khash.h"
7 +#include "ewah/ewok.h"
8 +
9 +struct commit;
10 +struct string_list;
11 +struct bitmap_index;
12 +struct bitmap_writer;
13 +
14 +/*
15 + * A pseudo-merge group tracks the set of non-bitmapped reference tips
16 + * that match the given pattern.
17 + *
18 + * Within those matches, they are further segmented by separating
19 + * consecutive capture groups with '-' dash character capture groups
20 + * with '-' dash characters.
21 + *
22 + * Those groups are then ordered by committer date and partitioned
23 + * into individual pseudo-merge(s) according to the decay, max_merges,
24 + * sample_rate, and threshold parameters.
25 + */
26 +struct pseudo_merge_group {
27 + regex_t *pattern;
28 +
29 + /* capture group(s) -> struct pseudo_merge_matches */
30 + struct strmap matches;
31 +
32 + /*
33 + * The individual pseudo-merge(s) that are generated from the
34 + * above array of matches, partitioned according to the below
35 + * parameters.
36 + */
37 + struct commit **merges;
38 + size_t merges_nr;
39 + size_t merges_alloc;
40 +
41 + /*
42 + * Pseudo-merge grouping parameters. See git-config(1) for
43 + * more information.
44 + */
45 + double decay;
46 + int max_merges;
47 + double sample_rate;
48 + int stable_size;
49 + timestamp_t threshold;
50 + timestamp_t stable_threshold;
51 +};
52 +
53 +struct pseudo_merge_matches {
54 + struct commit **stable;
55 + struct commit **unstable;
56 + size_t stable_nr, stable_alloc;
57 + size_t unstable_nr, unstable_alloc;
58 +};
59 +
60 +/*
61 + * Read the repository's configuration:
62 + *
63 + * - bitmapPseudoMerge.<name>.pattern
64 + * - bitmapPseudoMerge.<name>.decay
65 + * - bitmapPseudoMerge.<name>.sampleRate
66 + * - bitmapPseudoMerge.<name>.threshold
67 + * - bitmapPseudoMerge.<name>.maxMerges
68 + * - bitmapPseudoMerge.<name>.stableThreshold
69 + * - bitmapPseudoMerge.<name>.stableSize
70 + *
71 + * and populates the given `list` with pseudo-merge groups. String
72 + * entry keys are the pseudo-merge group names, and the values are
73 + * pointers to the pseudo_merge_group structure itself.
74 + */
75 +void load_pseudo_merges_from_config(struct string_list *list);
76 +
77 +/*
78 + * A pseudo-merge commit index (pseudo_merge_commit_idx) maps a
79 + * particular (non-pseudo-merge) commit to the list of pseudo-merge(s)
80 + * it appears in.
81 + */
82 +struct pseudo_merge_commit_idx {
83 + uint32_t *pseudo_merge;
84 + size_t nr, alloc;
85 +};
86 +
87 +/*
88 + * Selects pseudo-merges from a list of commits, populating the given
89 + * string_list of pseudo-merge groups.
90 + *
91 + * Populates the pseudo_merge_commits map with a commit_idx
92 + * corresponding to each commit in the list. Counts the total number
93 + * of pseudo-merges generated.
94 + *
95 + * Optionally shows a progress meter.
96 + */
97 +void select_pseudo_merges(struct bitmap_writer *writer,
98 + struct commit **commits, size_t commits_nr);
99
100 #endif