pack-objects: break out of want_object loop early

When pack-objects collects the list of objects to pack (either from stdin, or via its internal rev-list), it filters each one through want_object_in_pack(). This function loops through each existing packfile, looking for the object. When we find it, we mark the pack/offset combo for later use. However, we can't just return "yes, we want it" at that point. If --honor-pack-keep is in effect, we must keep looking to find it in _all_ packs, to make sure none of them has a .keep. Likewise, if --local is in effect, we must make sure it is not present in any non-local pack. As a result, the sum effort of these calls is effectively O(nr_objects * nr_packs). In an ordinary repository, we have only a handful of packs, and this doesn't make a big difference. But in pathological cases, it can slow the counting phase to a crawl. This patch notices the case that we have neither "--local" nor "--honor-pack-keep" in effect and breaks out of the loop early, after finding the first instance. Note that our worst case is still "objects * packs" (i.e., we might find each object in the last pack we look in), but in practice we will often break out early. On an "average" repo, my git.git with 8 packs, this shows a modest 2% (a few dozen milliseconds) improvement in the counting-objects phase of "git pack-objects --all <foo" (hackily instrumented by sticking exit(0) right after list_objects). But in a much more pathological case, it makes a bigger difference. I ran the same command on a real-world example with ~9 million objects across 1300 packs. The counting time dropped from 413s to 45s, an improvement of about 89%. Note that this patch won't do anything by itself for a normal "git gc", as it uses both --honor-pack-keep and --local. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Jeff King committed Jul 29, 2016 at 00:10 UTC cd3799679533328dcf262549c9d6466b07628df1
1 file changed +16
builtin/pack-objects.c
+16
@@ -977,6 +977,22 @@ static int want_object_in_pack(const unsigned char *sha1,
977 return 1;
978 if (incremental)
979 return 0;
980 +
981 + /*
982 + * When asked to do --local (do not include an
983 + * object that appears in a pack we borrow
984 + * from elsewhere) or --honor-pack-keep (do not
985 + * include an object that appears in a pack marked
986 + * with .keep), we need to make sure no copy of this
987 + * object come from in _any_ pack that causes us to
988 + * omit it, and need to complete this loop. When
989 + * neither option is in effect, we know the object
990 + * we just found is going to be packed, so break
991 + * out of the loop to return 1 now.
992 + */
993 + if (!ignore_packed_keep && !local)
994 + break;
995 +
996 if (local && !p->pack_local)
997 return 0;
998 if (ignore_packed_keep && p->pack_local && p->pack_keep)