@matheusbsbs / matheus-bagliano / commits / 59c35fac54

refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

When iterating through the `packed-refs` file in order to answer a query like: $ git for-each-ref --exclude=refs/__hidden__ it would be useful to avoid walking over all of the entries in `refs/__hidden__/*` when possible, since we know that the ref-filter code is going to throw them away anyways. In certain circumstances, doing so is possible. The algorithm for doing so is as follows: - For each excluded pattern, find the first record that matches it, and the first record that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern). - Sort the set of excluded regions from the previous step in ascending order of the first location within the `packed-refs` file that matches. - Clean up the results from the previous step: discard empty regions, and combine adjacent regions. The set of regions which remains is referred to as the "jump list", and never contains any references which should be included in the result set. Then when iterating through the `packed-refs` file, if `iter->pos` is ever contained in one of the regions from the previous steps, advance `iter->pos` past the end of that region, and continue enumeration. Note that we only perform this optimization when none of the excluded pattern(s) have special meta-characters in them. For a pattern like "refs/foo[ac]", the excluded regions ("refs/fooa", "refs/fooc", and everything underneath them) are not connected. A future implementation that handles this case may split the character class (pretending as if two patterns were excluded: "refs/fooa", and "refs/fooc"). There are a few other gotchas worth considering. First, note that the jump list is sorted, so once we jump past a region, we can avoid considering it (or any regions preceding it) again. The member `jump_pos` is used to track the first next-possible region to jump through. Second, note that the jump list is best-effort, since we do not handle loose references, and because of the meta-character issue above. The jump list may not skip past all references which won't appear in the results, but will never skip over a reference which does appear in the result set. In repositories with a large number of hidden references, the speed-up can be significant. Tests here are done with a copy of linux.git with a reference "refs/pull/N" pointing at every commit, as in: $ git rev-list HEAD | awk '{ print "create refs/pull/" NR " " $0 }' | git update-ref --stdin $ git pack-refs --all , it is significantly faster to have `for-each-ref` jump over the excluded references, as opposed to filtering them out after the fact: $ hyperfine \ 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \ 'git.prev for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' \ 'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/" Time (mean ± σ): 798.1 ms ± 3.3 ms [User: 687.6 ms, System: 146.4 ms] Range (min … max): 794.5 ms … 805.5 ms 10 runs Benchmark 2: git.prev for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull" Time (mean ± σ): 98.9 ms ± 1.4 ms [User: 93.1 ms, System: 5.7 ms] Range (min … max): 97.0 ms … 104.0 ms 29 runs Benchmark 3: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull" Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 0.7 ms, System: 3.8 ms] Range (min … max): 4.1 ms … 5.8 ms 524 runs Summary 'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran 21.87 ± 1.05 times faster than 'git.prev for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' 176.52 ± 8.19 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' (Comparing stock git and this patch isn't quite fair, since an earlier commit in this series adds a naive implementation of the `--exclude` option. `git.prev` is built from the previous commit and includes this naive implementation). Using the jump list is fairly straightforward (see the changes to `refs/packed-backend.c::next_record()`), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.). Co-authored-by: Jeff King <peff@peff.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Taylor Blau committed Jul 10, 2023 at 17:12 UTC 59c35fac54054b3f781b0275eac7d7c54468f0d5

5 files changed +276 -8

ref-filter.c

+4 -2

@@ -2210,12 +2210,14 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,

2210

2211 if (!filter->name_patterns[0]) {

2212 /* no patterns; we have to look at everything */

2213 - return for_each_fullref_in("", cb, cb_data);

2213 + return refs_for_each_fullref_in(get_main_ref_store(the_repository),

2214 + "", filter->exclude.v, cb, cb_data);

2215 }

2216

2217 return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),

2218 NULL, filter->name_patterns,

2218 - NULL, cb, cb_data);

2219 + filter->exclude.v,

2220 + cb, cb_data);

2221 }

2222

2223 /*

refs.h

@@ -337,6 +337,10 @@ int for_each_ref(each_ref_fn fn, void *cb_data);

337 */

338 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);

339

340 +/*

341 + * references matching any pattern in "exclude_patterns" are omitted from the

342 + * result set on a best-effort basis.

343 + */

344 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,

345 const char **exclude_patterns,

346 each_ref_fn fn, void *cb_data);

refs/packed-backend.c

+157 -6

@@ -303,7 +303,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)

303 * Compare a snapshot record at `rec` to the specified NUL-terminated

304 * refname.

305 */

306 -static int cmp_record_to_refname(const char *rec, const char *refname)

306 +static int cmp_record_to_refname(const char *rec, const char *refname,

307 + int start)

308 {

309 const char *r1 = rec + the_hash_algo->hexsz + 1;

310 const char *r2 = refname;

@@ -312,7 +313,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)

313 if (*r1 == '\n')

314 return *r2 ? -1 : 0;

315 if (!*r2)

315 - return 1;

316 + return start ? 1 : -1;

317 if (*r1 != *r2)

318 return (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;

319 r1++;

@@ -528,7 +529,8 @@ static int load_contents(struct snapshot *snapshot)

529 }

530

531 static const char *find_reference_location_1(struct snapshot *snapshot,

531 - const char *refname, int mustexist)

532 + const char *refname, int mustexist,

533 + int start)

534 {

535 /*

536 * This is not *quite* a garden-variety binary search, because

@@ -558,7 +560,7 @@ static const char *find_reference_location_1(struct snapshot *snapshot,

560

561 mid = lo + (hi - lo) / 2;

562 rec = find_start_of_record(lo, mid);

561 - cmp = cmp_record_to_refname(rec, refname);

563 + cmp = cmp_record_to_refname(rec, refname, start);

564 if (cmp < 0) {

565 lo = find_end_of_record(mid, hi);

566 } else if (cmp > 0) {

@@ -591,7 +593,22 @@ static const char *find_reference_location_1(struct snapshot *snapshot,

593 static const char *find_reference_location(struct snapshot *snapshot,

594 const char *refname, int mustexist)

595 {

594 - return find_reference_location_1(snapshot, refname, mustexist);

596 + return find_reference_location_1(snapshot, refname, mustexist, 1);

597 +}

598 +

599 +/*

600 + * Find the place in `snapshot->buf` after the end of the record for

601 + * `refname`. In other words, find the location of first thing *after*

602 + * `refname`.

603 + *

604 + * Other semantics are identical to the ones in

605 + * `find_reference_location()`.

606 + */

607 +static const char *find_reference_location_end(struct snapshot *snapshot,

608 + const char *refname,

609 + int mustexist)

610 +{

611 + return find_reference_location_1(snapshot, refname, mustexist, 0);

612 }

613

614 /*

@@ -785,6 +802,13 @@ struct packed_ref_iterator {

802 /* The end of the part of the buffer that will be iterated over: */

803 const char *eof;

804

805 + struct jump_list_entry {

806 + const char *start;

807 + const char *end;

808 + } *jump;

809 + size_t jump_nr, jump_alloc;

810 + size_t jump_cur;

811 +

812 /* Scratch space for current values: */

813 struct object_id oid, peeled;

814 struct strbuf refname_buf;

@@ -802,14 +826,35 @@ struct packed_ref_iterator {

826 */

827 static int next_record(struct packed_ref_iterator *iter)

828 {

805 - const char *p = iter->pos, *eol;

829 + const char *p, *eol;

830

831 strbuf_reset(&iter->refname_buf);

832

833 + /*

834 + * If iter->pos is contained within a skipped region, jump past

835 + * it.

836 + *

837 + * Note that each skipped region is considered at most once,

838 + * since they are ordered based on their starting position.

839 + */

840 + while (iter->jump_cur < iter->jump_nr) {

841 + struct jump_list_entry *curr = &iter->jump[iter->jump_cur];

842 + if (iter->pos < curr->start)

843 + break; /* not to the next jump yet */

844 +

845 + iter->jump_cur++;

846 + if (iter->pos < curr->end) {

847 + iter->pos = curr->end;

848 + /* jumps are coalesced, so only one jump is necessary */

849 + break;

850 + }

851 + }

852 +

853 if (iter->pos == iter->eof)

854 return ITER_DONE;

855

856 iter->base.flags = REF_ISPACKED;

857 + p = iter->pos;

858

859 if (iter->eof - p < the_hash_algo->hexsz + 2 ||

860 parse_oid_hex(p, &iter->oid, &p) ||

@@ -917,6 +962,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)

962 int ok = ITER_DONE;

963

964 strbuf_release(&iter->refname_buf);

965 + free(iter->jump);

966 release_snapshot(iter->snapshot);

967 base_ref_iterator_free(ref_iterator);

968 return ok;

@@ -928,6 +974,108 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {

974 .abort = packed_ref_iterator_abort

975 };

976

977 +static int jump_list_entry_cmp(const void *va, const void *vb)

978 +{

979 + const struct jump_list_entry *a = va;

980 + const struct jump_list_entry *b = vb;

981 +

982 + if (a->start < b->start)

983 + return -1;

984 + if (a->start > b->start)

985 + return 1;

986 + return 0;

987 +}

988 +

989 +static int has_glob_special(const char *str)

990 +{

991 + const char *p;

992 + for (p = str; *p; p++) {

993 + if (is_glob_special(*p))

994 + return 1;

995 + }

996 + return 0;

997 +}

998 +

999 +static void populate_excluded_jump_list(struct packed_ref_iterator *iter,

1000 + struct snapshot *snapshot,

1001 + const char **excluded_patterns)

1002 +{

1003 + size_t i, j;

1004 + const char **pattern;

1005 + struct jump_list_entry *last_disjoint;

1006 +

1007 + if (!excluded_patterns)

1008 + return;

1009 +

1010 + for (pattern = excluded_patterns; *pattern; pattern++) {

1011 + struct jump_list_entry *e;

1012 + const char *start, *end;

1013 +

1014 + /*

1015 + * We can't feed any excludes with globs in them to the

1016 + * refs machinery. It only understands prefix matching.

1017 + * We likewise can't even feed the string leading up to

1018 + * the first meta-character, as something like "foo[a]"

1019 + * should not exclude "foobar" (but the prefix "foo"

1020 + * would match that and mark it for exclusion).

1021 + */

1022 + if (has_glob_special(*pattern))

1023 + continue;

1024 +

1025 + start = find_reference_location(snapshot, *pattern, 0);

1026 + end = find_reference_location_end(snapshot, *pattern, 0);

1027 +

1028 + if (start == end)

1029 + continue; /* nothing to jump over */

1030 +

1031 + ALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);

1032 +

1033 + e = &iter->jump[iter->jump_nr++];

1034 + e->start = start;

1035 + e->end = end;

1036 + }

1037 +

1038 + if (!iter->jump_nr) {

1039 + /*

1040 + * Every entry in exclude_patterns has a meta-character,

1041 + * nothing to do here.

1042 + */

1043 + return;

1044 + }

1045 +

1046 + QSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);

1047 +

1048 + /*

1049 + * As an optimization, merge adjacent entries in the jump list

1050 + * to jump forwards as far as possible when entering a skipped

1051 + * region.

1052 + *

1053 + * For example, if we have two skipped regions:

1054 + *

1055 + * [[A, B], [B, C]]

1056 + *

1057 + * we want to combine that into a single entry jumping from A to

1058 + * C.

1059 + */

1060 + last_disjoint = iter->jump;

1061 +

1062 + for (i = 1, j = 1; i < iter->jump_nr; i++) {

1063 + struct jump_list_entry *ours = &iter->jump[i];

1064 + if (ours->start <= last_disjoint->end) {

1065 + /* overlapping regions extend the previous one */

1066 + last_disjoint->end = last_disjoint->end > ours->end

1067 + ? last_disjoint->end : ours->end;

1068 + } else {

1069 + /* otherwise, insert a new region */

1070 + iter->jump[j++] = *ours;

1071 + last_disjoint = ours;

1072 + }

1073 + }

1074 +

1075 + iter->jump_nr = j;

1076 + iter->jump_cur = 0;

1077 +}

1078 +

1079 static struct ref_iterator *packed_ref_iterator_begin(

1080 struct ref_store *ref_store,

1081 const char *prefix, const char **exclude_patterns,

@@ -963,6 +1111,9 @@ static struct ref_iterator *packed_ref_iterator_begin(

1111 ref_iterator = &iter->base;

1112 base_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);

1113

1114 + if (exclude_patterns)

1115 + populate_excluded_jump_list(iter, snapshot, exclude_patterns);

1116 +

1117 iter->snapshot = snapshot;

1118 acquire_snapshot(snapshot);

1119

t/helper/test-ref-store.c

+10

@@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)

175 return refs_for_each_ref_in(refs, prefix, each_ref, NULL);

176 }

177

178 +static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)

179 +{

180 + const char *prefix = notnull(*argv++, "prefix");

181 + const char **exclude_patterns = argv;

182 +

183 + return refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,

184 + NULL);

185 +}

186 +

187 static int cmd_resolve_ref(struct ref_store *refs, const char **argv)

188 {

189 struct object_id oid = *null_oid();

@@ -307,6 +316,7 @@ static struct command commands[] = {

316 { "delete-refs", cmd_delete_refs },

317 { "rename-ref", cmd_rename_ref },

318 { "for-each-ref", cmd_for_each_ref },

319 + { "for-each-ref--exclude", cmd_for_each_ref__exclude },

320 { "resolve-ref", cmd_resolve_ref },

321 { "verify-ref", cmd_verify_ref },

322 { "for-each-reflog", cmd_for_each_reflog },

t/t1419-exclude-refs.sh new

+101

@@ -0,0 +1,101 @@

1 +#!/bin/sh

2 +

3 +test_description='test exclude_patterns functionality in main ref store'

4 +

5 +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main

6 +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME

7 +

8 +TEST_PASSES_SANITIZE_LEAK=true

9 +. ./test-lib.sh

10 +

11 +for_each_ref__exclude () {

12 + test-tool ref-store main for-each-ref--exclude "$@" >actual.raw

13 + cut -d ' ' -f 2 actual.raw

14 +}

15 +

16 +for_each_ref () {

17 + git for-each-ref --format='%(refname)' "$@"

18 +}

19 +

20 +test_expect_success 'setup' '

21 + test_commit --no-tag base &&

22 + base="$(git rev-parse HEAD)" &&

23 +

24 + for name in foo bar baz quux

25 + do

26 + for i in 1 2 3

27 + do

28 + echo "create refs/heads/$name/$i $base" || return 1

29 + done || return 1

30 + done >in &&

31 + echo "delete refs/heads/main" >>in &&

32 +

33 + git update-ref --stdin <in &&

34 + git pack-refs --all

35 +'

36 +

37 +test_expect_success 'excluded region in middle' '

38 + for_each_ref__exclude refs/heads refs/heads/foo >actual &&

39 + for_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&

40 +

41 + test_cmp expect actual

42 +'

43 +

44 +test_expect_success 'excluded region at beginning' '

45 + for_each_ref__exclude refs/heads refs/heads/bar >actual &&

46 + for_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&

47 +

48 + test_cmp expect actual

49 +'

50 +

51 +test_expect_success 'excluded region at end' '

52 + for_each_ref__exclude refs/heads refs/heads/quux >actual &&

53 + for_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&

54 +

55 + test_cmp expect actual

56 +'

57 +

58 +test_expect_success 'disjoint excluded regions' '

59 + for_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&

60 + for_each_ref refs/heads/baz refs/heads/foo >expect &&

61 +

62 + test_cmp expect actual

63 +'

64 +

65 +test_expect_success 'adjacent, non-overlapping excluded regions' '

66 + for_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&

67 + for_each_ref refs/heads/foo refs/heads/quux >expect &&

68 +

69 + test_cmp expect actual

70 +'

71 +

72 +test_expect_success 'overlapping excluded regions' '

73 + for_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&

74 + for_each_ref refs/heads/foo refs/heads/quux >expect &&

75 +

76 + test_cmp expect actual

77 +'

78 +

79 +test_expect_success 'several overlapping excluded regions' '

80 + for_each_ref__exclude refs/heads \

81 + refs/heads/bar refs/heads/baz refs/heads/foo >actual &&

82 + for_each_ref refs/heads/quux >expect &&

83 +

84 + test_cmp expect actual

85 +'

86 +

87 +test_expect_success 'non-matching excluded section' '

88 + for_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&

89 + for_each_ref >expect &&

90 +

91 + test_cmp expect actual

92 +'

93 +

94 +test_expect_success 'meta-characters are discarded' '

95 + for_each_ref__exclude refs/heads "refs/heads/ba*" >actual &&

96 + for_each_ref >expect &&

97 +

98 + test_cmp expect actual

99 +'

100 +

101 +test_done