@matheusbsbs / matheus-bagliano / commits / fe7df03a9a

fetch: speed up lookup of want refs via commit-graph

When updating our local refs based on the refs fetched from the remote, we need to iterate through all requested refs and load their respective commits such that we can determine whether they need to be appended to FETCH_HEAD or not. In cases where we're fetching from a remote with exceedingly many refs, resolving these refs can be quite expensive given that we repeatedly need to unpack object headers for each of the referenced objects. Speed this up by opportunistically trying to resolve object IDs via the commit graph. We only do so for any refs which are not in "refs/tags": more likely than not, these are going to be a commit anyway, and this lets us avoid having to unpack object headers completely in case the object is a commit that is part of the commit-graph. This significantly speeds up mirror-fetches in a real-world repository with 2.3M refs: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 56.482 s ± 0.384 s [User: 53.340 s, System: 5.365 s] Range (min … max): 56.050 s … 57.045 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 33.727 s ± 0.170 s [User: 30.252 s, System: 5.194 s] Range (min … max): 33.452 s … 33.871 s 5 runs Summary 'HEAD: git-fetch' ran 1.67 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Patrick Steinhardt committed Sep 1, 2021 at 15:09 UTC fe7df03a9a2fa434ebce38b2cd5e6da42f8b2692

1 file changed +18 -6

builtin/fetch.c

+18 -6

@@ -1074,7 +1074,6 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,

1074 int connectivity_checked, struct ref *ref_map)

1075 {

1076 struct fetch_head fetch_head;

1077 - struct commit *commit;

1077 int url_len, i, rc = 0;

1078 struct strbuf note = STRBUF_INIT, err = STRBUF_INIT;

1079 struct ref_transaction *transaction = NULL;

@@ -1122,6 +1121,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,

1121 want_status <= FETCH_HEAD_IGNORE;

1122 want_status++) {

1123 for (rm = ref_map; rm; rm = rm->next) {

1124 + struct commit *commit = NULL;

1125 struct ref *ref = NULL;

1126

1127 if (rm->status == REF_STATUS_REJECT_SHALLOW) {

@@ -1131,11 +1131,23 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,

1131 continue;

1132 }

1133

1134 - commit = lookup_commit_reference_gently(the_repository,

1135 - &rm->old_oid,

1136 - 1);

1137 - if (!commit)

1138 - rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE;

1134 + /*

1135 + * References in "refs/tags/" are often going to point

1136 + * to annotated tags, which are not part of the

1137 + * commit-graph. We thus only try to look up refs in

1138 + * the graph which are not in that namespace to not

1139 + * regress performance in repositories with many

1140 + * annotated tags.

1141 + */

1142 + if (!starts_with(rm->name, "refs/tags/"))

1143 + commit = lookup_commit_in_graph(the_repository, &rm->old_oid);

1144 + if (!commit) {

1145 + commit = lookup_commit_reference_gently(the_repository,

1146 + &rm->old_oid,

1147 + 1);

1148 + if (!commit)

1149 + rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE;

1150 + }

1151

1152 if (rm->fetch_head_status != want_status)

1153 continue;