for-each-ref: add ahead-behind format atom
The previous change implemented the ahead_behind() method, including an algorithm to compute the ahead/behind values for a number of commit tips relative to a number of commit bases. Now, integrate that algorithm as part of 'git for-each-ref' hidden behind a new format atom, ahead-behind. This naturally extends to 'git branch' and 'git tag' builtins, as well. This format allows specifying multiple bases, if so desired, and all matching references are compared against all of those bases. For this reason, failing to read a reference provided from these atoms results in an error. In order to translate the ahead_behind() method information to the format output code in ref-filter.c, we must populate arrays of ahead_behind_count structs. In struct ref_array, we store the full array that will be passed to ahead_behind(). In struct ref_array_item, we store an array of pointers that point to the relvant items within the full array. In this way, we can pull all relevant ahead/behind values directly when formatting output for a specific item. It also ensures the lifetime of the ahead_behind_count structs matches the time that the array is being used. Add specific tests of the ahead/behind counts in t6600-test-reach.sh, as it has an interesting repository shape. In particular, its merging strategy and its use of different commit-graphs would demonstrate over- counting if the ahead_behind() method did not already account for that possibility. Also add tests for the specific for-each-ref, branch, and tag builtins. In the case of 'git tag', there are intersting cases that happen when some of the selected tips are not commits. This requires careful logic around commits_nr in the second loop of filter_ahead_behind(). Also, the test in t7004 is carefully located to avoid being dependent on the GPG prereq. It also avoids using the test_commit helper, as that will add ticks to the time and disrupt the expected timestamps in later tag tests. Also add performance tests in a new p1300-graph-walks.sh script. This will be useful for more uses in the future, but for now compare the ahead-behind counting algorithm in 'git for-each-ref' to the naive implementation by running 'git rev-list --count' processes for each input. For the Git source code repository, the improvement is already obvious: Test this tree --------------------------------------------------------------- 1500.2: ahead-behind counts: git for-each-ref 0.07(0.07+0.00) 1500.3: ahead-behind counts: git branch 0.07(0.06+0.00) 1500.4: ahead-behind counts: git tag 0.07(0.06+0.00) 1500.5: ahead-behind counts: git rev-list 1.32(1.04+0.27) But the standard performance benchmark is the Linux kernel repository, which demosntrates a significant improvement: Test this tree --------------------------------------------------------------- 1500.2: ahead-behind counts: git for-each-ref 0.27(0.24+0.02) 1500.3: ahead-behind counts: git branch 0.27(0.24+0.03) 1500.4: ahead-behind counts: git tag 0.28(0.27+0.01) 1500.5: ahead-behind counts: git rev-list 4.57(4.03+0.54) The 'git rev-list' test exists in this change as a demonstration, but it will be removed in the next change to avoid wasting time on this comparison. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Derrick Stolee committed
Mar 20, 2023 at 11:26 UTC
49abcd21da65f12b4ae873433e5f6220bfaae1da
11 files changed
+295
-1
Documentation/git-for-each-ref.txt
+5
@@ -222,6 +222,11 @@ worktreepath::
222
out, if it is checked out in any linked worktree. Empty string
223
otherwise.
224
225
+ahead-behind:<committish>::
226
+ Two integers, separated by a space, demonstrating the number of
227
+ commits ahead and behind, respectively, when comparing the output
228
+ ref to the `<committish>` specified in the format.
229
+
230
In addition to the above, for commit and tag objects, the header
231
field names (`tree`, `parent`, `object`, `type`, and `tag`) can
232
be used to specify the value in the header field.
builtin/branch.c
+1
@@ -448,6 +448,7 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin
448
if (verify_ref_format(format))
449
die(_("unable to parse format string"));
450
451
+ filter_ahead_behind(the_repository, format, &array);
452
ref_array_sort(sorting, &array);
453
454
for (i = 0; i < array.nr; i++) {
builtin/for-each-ref.c
+3
@@ -6,6 +6,7 @@
6
#include "parse-options.h"
7
#include "ref-filter.h"
8
#include "strvec.h"
9
+#include "commit-reach.h"
10
11
static char const * const for_each_ref_usage[] = {
12
N_("git for-each-ref [<options>] [<pattern>]"),
@@ -98,6 +99,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
99
100
filter.match_as_path = 1;
101
filter_refs(&array, &filter, FILTER_REFS_ALL);
102
+ filter_ahead_behind(the_repository, &format, &array);
103
+
104
ref_array_sort(sorting, &array);
105
106
if (!maxcount || array.nr < maxcount)
builtin/tag.c
+1
@@ -66,6 +66,7 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,
66
die(_("unable to parse format string"));
67
filter->with_commit_tag_algo = 1;
68
filter_refs(&array, filter, FILTER_REFS_TAGS);
69
+ filter_ahead_behind(the_repository, format, &array);
70
ref_array_sort(sorting, &array);
71
72
for (i = 0; i < array.nr; i++) {
ref-filter.c
+73
@@ -158,6 +158,7 @@ enum atom_type {
158
ATOM_THEN,
159
ATOM_ELSE,
160
ATOM_REST,
161
+ ATOM_AHEADBEHIND,
162
};
163
164
/*
@@ -586,6 +587,22 @@ static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,
587
return 0;
588
}
589
590
+static int ahead_behind_atom_parser(struct ref_format *format, struct used_atom *atom,
591
+ const char *arg, struct strbuf *err)
592
+{
593
+ struct string_list_item *item;
594
+
595
+ if (!arg)
596
+ return strbuf_addf_ret(err, -1, _("expected format: %%(ahead-behind:<committish>)"));
597
+
598
+ item = string_list_append(&format->bases, arg);
599
+ item->util = lookup_commit_reference_by_name(arg);
600
+ if (!item->util)
601
+ die("failed to find '%s'", arg);
602
+
603
+ return 0;
604
+}
605
+
606
static int head_atom_parser(struct ref_format *format, struct used_atom *atom,
607
const char *arg, struct strbuf *err)
608
{
@@ -645,6 +662,7 @@ static struct {
662
[ATOM_THEN] = { "then", SOURCE_NONE },
663
[ATOM_ELSE] = { "else", SOURCE_NONE },
664
[ATOM_REST] = { "rest", SOURCE_NONE, FIELD_STR, rest_atom_parser },
665
+ [ATOM_AHEADBEHIND] = { "ahead-behind", SOURCE_OTHER, FIELD_STR, ahead_behind_atom_parser },
666
/*
667
* Please update $__git_ref_fieldlist in git-completion.bash
668
* when you add new atoms
@@ -1848,6 +1866,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
1866
struct object *obj;
1867
int i;
1868
struct object_info empty = OBJECT_INFO_INIT;
1869
+ int ahead_behind_atoms = 0;
1870
1871
CALLOC_ARRAY(ref->value, used_atom_cnt);
1872
@@ -1978,6 +1997,16 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
1997
else
1998
v->s = xstrdup("");
1999
continue;
2000
+ } else if (atom_type == ATOM_AHEADBEHIND) {
2001
+ if (ref->counts) {
2002
+ const struct ahead_behind_count *count;
2003
+ count = ref->counts[ahead_behind_atoms++];
2004
+ v->s = xstrfmt("%d %d", count->ahead, count->behind);
2005
+ } else {
2006
+ /* Not a commit. */
2007
+ v->s = xstrdup("");
2008
+ }
2009
+ continue;
2010
} else
2011
continue;
2012
@@ -2328,6 +2357,7 @@ static void free_array_item(struct ref_array_item *item)
2357
free((char *)item->value[i].s);
2358
free(item->value);
2359
}
2360
+ free(item->counts);
2361
free(item);
2362
}
2363
@@ -2356,6 +2386,8 @@ void ref_array_clear(struct ref_array *array)
2386
free_worktrees(ref_to_worktree_map.worktrees);
2387
ref_to_worktree_map.worktrees = NULL;
2388
}
2389
+
2390
+ FREE_AND_NULL(array->counts);
2391
}
2392
2393
#define EXCLUDE_REACHED 0
@@ -2418,6 +2450,47 @@ static void reach_filter(struct ref_array *array,
2450
free(to_clear);
2451
}
2452
2453
+void filter_ahead_behind(struct repository *r,
2454
+ struct ref_format *format,
2455
+ struct ref_array *array)
2456
+{
2457
+ struct commit **commits;
2458
+ size_t commits_nr = format->bases.nr + array->nr;
2459
+
2460
+ if (!format->bases.nr || !array->nr)
2461
+ return;
2462
+
2463
+ ALLOC_ARRAY(commits, commits_nr);
2464
+ for (size_t i = 0; i < format->bases.nr; i++)
2465
+ commits[i] = format->bases.items[i].util;
2466
+
2467
+ ALLOC_ARRAY(array->counts, st_mult(format->bases.nr, array->nr));
2468
+
2469
+ commits_nr = format->bases.nr;
2470
+ array->counts_nr = 0;
2471
+ for (size_t i = 0; i < array->nr; i++) {
2472
+ const char *name = array->items[i]->refname;
2473
+ commits[commits_nr] = lookup_commit_reference_by_name(name);
2474
+
2475
+ if (!commits[commits_nr])
2476
+ continue;
2477
+
2478
+ CALLOC_ARRAY(array->items[i]->counts, format->bases.nr);
2479
+ for (size_t j = 0; j < format->bases.nr; j++) {
2480
+ struct ahead_behind_count *count;
2481
+ count = &array->counts[array->counts_nr++];
2482
+ count->tip_index = commits_nr;
2483
+ count->base_index = j;
2484
+
2485
+ array->items[i]->counts[j] = count;
2486
+ }
2487
+ commits_nr++;
2488
+ }
2489
+
2490
+ ahead_behind(r, commits, commits_nr, array->counts, array->counts_nr);
2491
+ free(commits);
2492
+}
2493
+
2494
/*
2495
* API for filtering a set of refs. Based on the type of refs the user
2496
* has requested, we iterate through those refs and apply filters
ref-filter.h
+25
-1
@@ -5,6 +5,7 @@
5
#include "refs.h"
6
#include "commit.h"
7
#include "parse-options.h"
8
+#include "string-list.h"
9
10
/* Quoting styles */
11
#define QUOTE_NONE 0
@@ -24,6 +25,7 @@
25
26
struct atom_value;
27
struct ref_sorting;
28
+struct ahead_behind_count;
29
30
enum ref_sorting_order {
31
REF_SORTING_REVERSE = 1<<0,
@@ -40,6 +42,8 @@ struct ref_array_item {
42
const char *symref;
43
struct commit *commit;
44
struct atom_value *value;
45
+ struct ahead_behind_count **counts;
46
+
47
char refname[FLEX_ARRAY];
48
};
49
@@ -47,6 +51,9 @@ struct ref_array {
51
int nr, alloc;
52
struct ref_array_item **items;
53
struct rev_info *revs;
54
+
55
+ struct ahead_behind_count *counts;
56
+ size_t counts_nr;
57
};
58
59
struct ref_filter {
@@ -80,9 +87,15 @@ struct ref_format {
87
88
/* Internal state to ref-filter */
89
int need_color_reset_at_eol;
90
+
91
+ /* List of bases for ahead-behind counts. */
92
+ struct string_list bases;
93
};
94
85
-#define REF_FORMAT_INIT { .use_color = -1 }
95
+#define REF_FORMAT_INIT { \
96
+ .use_color = -1, \
97
+ .bases = STRING_LIST_INIT_DUP, \
98
+}
99
100
/* Macros for checking --merged and --no-merged options */
101
#define _OPT_MERGED_NO_MERGED(option, filter, h) \
@@ -143,4 +156,15 @@ struct ref_array_item *ref_array_push(struct ref_array *array,
156
const char *refname,
157
const struct object_id *oid);
158
159
+/*
160
+ * If the provided format includes ahead-behind atoms, then compute the
161
+ * ahead-behind values for the array of filtered references. Must be
162
+ * called after filter_refs() but before outputting the formatted refs.
163
+ *
164
+ * If this is not called, then any ahead-behind atoms will be blank.
165
+ */
166
+void filter_ahead_behind(struct repository *r,
167
+ struct ref_format *format,
168
+ struct ref_array *array);
169
+
170
#endif /* REF_FILTER_H */
t/perf/p1500-graph-walks.sh
new
+45
@@ -0,0 +1,45 @@
1
+#!/bin/sh
2
+
3
+test_description='Commit walk performance tests'
4
+. ./perf-lib.sh
5
+
6
+test_perf_large_repo
7
+
8
+test_expect_success 'setup' '
9
+ git for-each-ref --format="%(refname)" "refs/heads/*" "refs/tags/*" >allrefs &&
10
+ sort -r allrefs | head -n 50 >refs &&
11
+ for ref in $(cat refs)
12
+ do
13
+ git branch -f ref-$ref $ref &&
14
+ echo ref-$ref ||
15
+ return 1
16
+ done >branches &&
17
+ for ref in $(cat refs)
18
+ do
19
+ git tag -f tag-$ref $ref &&
20
+ echo tag-$ref ||
21
+ return 1
22
+ done >tags &&
23
+ git commit-graph write --reachable
24
+'
25
+
26
+test_perf 'ahead-behind counts: git for-each-ref' '
27
+ git for-each-ref --format="%(ahead-behind:HEAD)" --stdin <refs
28
+'
29
+
30
+test_perf 'ahead-behind counts: git branch' '
31
+ xargs git branch -l --format="%(ahead-behind:HEAD)" <branches
32
+'
33
+
34
+test_perf 'ahead-behind counts: git tag' '
35
+ xargs git tag -l --format="%(ahead-behind:HEAD)" <tags
36
+'
37
+
38
+test_perf 'ahead-behind counts: git rev-list' '
39
+ for r in $(cat refs)
40
+ do
41
+ git rev-list --count "HEAD..$r" || return 1
42
+ done
43
+'
44
+
45
+test_done
t/t3203-branch-output.sh
+14
@@ -337,6 +337,20 @@ test_expect_success 'git branch --format option' '
337
test_cmp expect actual
338
'
339
340
+test_expect_success 'git branch --format with ahead-behind' '
341
+ cat >expect <<-\EOF &&
342
+ (HEAD detached from fromtag) 0 0
343
+ refs/heads/ambiguous 0 0
344
+ refs/heads/branch-one 1 0
345
+ refs/heads/branch-two 0 0
346
+ refs/heads/main 1 0
347
+ refs/heads/ref-to-branch 1 0
348
+ refs/heads/ref-to-remote 1 0
349
+ EOF
350
+ git branch --format="%(refname) %(ahead-behind:HEAD)" >actual &&
351
+ test_cmp expect actual
352
+'
353
+
354
test_expect_success 'git branch with --format=%(rest) must fail' '
355
test_must_fail git branch --format="%(rest)" >actual
356
'
t/t6301-for-each-ref-errors.sh
+14
@@ -54,4 +54,18 @@ test_expect_success 'Missing objects are reported correctly' '
54
test_must_be_empty brief-err
55
'
56
57
+test_expect_success 'ahead-behind requires an argument' '
58
+ test_must_fail git for-each-ref \
59
+ --format="%(ahead-behind)" 2>err &&
60
+ echo "fatal: expected format: %(ahead-behind:<committish>)" >expect &&
61
+ test_cmp expect err
62
+'
63
+
64
+test_expect_success 'missing ahead-behind base' '
65
+ test_must_fail git for-each-ref \
66
+ --format="%(ahead-behind:refs/heads/missing)" 2>err &&
67
+ echo "fatal: failed to find '\''refs/heads/missing'\''" >expect &&
68
+ test_cmp expect err
69
+'
70
+
71
test_done
t/t6600-test-reach.sh
+86
@@ -443,4 +443,90 @@ test_expect_success 'get_reachable_subset:none' '
443
test_all_modes get_reachable_subset
444
'
445
446
+test_expect_success 'for-each-ref ahead-behind:linear' '
447
+ cat >input <<-\EOF &&
448
+ refs/heads/commit-1-1
449
+ refs/heads/commit-1-3
450
+ refs/heads/commit-1-5
451
+ refs/heads/commit-1-8
452
+ EOF
453
+ cat >expect <<-\EOF &&
454
+ refs/heads/commit-1-1 0 8
455
+ refs/heads/commit-1-3 0 6
456
+ refs/heads/commit-1-5 0 4
457
+ refs/heads/commit-1-8 0 1
458
+ EOF
459
+ run_all_modes git for-each-ref \
460
+ --format="%(refname) %(ahead-behind:commit-1-9)" --stdin
461
+'
462
+
463
+test_expect_success 'for-each-ref ahead-behind:all' '
464
+ cat >input <<-\EOF &&
465
+ refs/heads/commit-1-1
466
+ refs/heads/commit-2-4
467
+ refs/heads/commit-4-2
468
+ refs/heads/commit-4-4
469
+ EOF
470
+ cat >expect <<-\EOF &&
471
+ refs/heads/commit-1-1 0 24
472
+ refs/heads/commit-2-4 0 17
473
+ refs/heads/commit-4-2 0 17
474
+ refs/heads/commit-4-4 0 9
475
+ EOF
476
+ run_all_modes git for-each-ref \
477
+ --format="%(refname) %(ahead-behind:commit-5-5)" --stdin
478
+'
479
+
480
+test_expect_success 'for-each-ref ahead-behind:some' '
481
+ cat >input <<-\EOF &&
482
+ refs/heads/commit-1-1
483
+ refs/heads/commit-5-3
484
+ refs/heads/commit-4-8
485
+ refs/heads/commit-9-9
486
+ EOF
487
+ cat >expect <<-\EOF &&
488
+ refs/heads/commit-1-1 0 53
489
+ refs/heads/commit-4-8 8 30
490
+ refs/heads/commit-5-3 0 39
491
+ refs/heads/commit-9-9 27 0
492
+ EOF
493
+ run_all_modes git for-each-ref \
494
+ --format="%(refname) %(ahead-behind:commit-9-6)" --stdin
495
+'
496
+
497
+test_expect_success 'for-each-ref ahead-behind:some, multibase' '
498
+ cat >input <<-\EOF &&
499
+ refs/heads/commit-1-1
500
+ refs/heads/commit-5-3
501
+ refs/heads/commit-7-8
502
+ refs/heads/commit-4-8
503
+ refs/heads/commit-9-9
504
+ EOF
505
+ cat >expect <<-\EOF &&
506
+ refs/heads/commit-1-1 0 53 0 53
507
+ refs/heads/commit-4-8 8 30 0 22
508
+ refs/heads/commit-5-3 0 39 0 39
509
+ refs/heads/commit-7-8 14 12 8 6
510
+ refs/heads/commit-9-9 27 0 27 0
511
+ EOF
512
+ run_all_modes git for-each-ref \
513
+ --format="%(refname) %(ahead-behind:commit-9-6) %(ahead-behind:commit-6-9)" \
514
+ --stdin
515
+'
516
+
517
+test_expect_success 'for-each-ref ahead-behind:none' '
518
+ cat >input <<-\EOF &&
519
+ refs/heads/commit-7-5
520
+ refs/heads/commit-4-8
521
+ refs/heads/commit-9-9
522
+ EOF
523
+ cat >expect <<-\EOF &&
524
+ refs/heads/commit-4-8 16 16
525
+ refs/heads/commit-7-5 7 4
526
+ refs/heads/commit-9-9 49 0
527
+ EOF
528
+ run_all_modes git for-each-ref \
529
+ --format="%(refname) %(ahead-behind:commit-8-4)" --stdin
530
+'
531
+
532
test_done
t/t7004-tag.sh
+28
@@ -792,6 +792,34 @@ test_expect_success 'annotations for blobs are empty' '
792
test_cmp expect actual
793
'
794
795
+# Run this before doing any signing, so the test has the same results
796
+# regardless of the GPG prereq.
797
+test_expect_success 'git tag --format with ahead-behind' '
798
+ test_when_finished git reset --hard tag-one-line &&
799
+ git commit --allow-empty -m "left" &&
800
+ git tag -a -m left tag-left &&
801
+ git reset --hard HEAD~1 &&
802
+ git commit --allow-empty -m "right" &&
803
+ git tag -a -m left tag-right &&
804
+
805
+ # Use " !" at the end to demonstrate whitespace
806
+ # around empty ahead-behind token for tag-blob.
807
+ cat >expect <<-EOF &&
808
+ refs/tags/tag-blob !
809
+ refs/tags/tag-left 1 1 !
810
+ refs/tags/tag-lines 0 1 !
811
+ refs/tags/tag-one-line 0 1 !
812
+ refs/tags/tag-right 0 0 !
813
+ refs/tags/tag-zero-lines 0 1 !
814
+ EOF
815
+ git tag -l --format="%(refname) %(ahead-behind:HEAD) !" >actual 2>err &&
816
+ grep "refs/tags/tag" actual >actual.focus &&
817
+ test_cmp expect actual.focus &&
818
+
819
+ # Error reported for tags that point to non-commits.
820
+ grep "error: object [0-9a-f]* is a blob, not a commit" err
821
+'
822
+
823
# trying to verify annotated non-signed tags:
824
825
test_expect_success GPG \