@matheusbsbs / matheus-bagliano / commits / ab6eea6f7b

receive-pack: use oidset to de-duplicate .have lines

If you have an alternate object store with a very large number of refs, the peak memory usage of the sha1_array can grow high, even if most of them are duplicates that end up not being printed at all. The similar for_each_alternate_ref() code-paths in fetch-pack solve this by using flags in "struct object" to de-duplicate (and so are relying on obj_hash at the core). But we don't have a "struct object" at all in this case. We could call lookup_unknown_object() to get one, but if our goal is reducing memory footprint, it's not great: - an unknown object is as large as the largest object type (a commit), which is bigger than an oidset entry - we can free the memory after our ref advertisement, but "struct object" entries persist forever (and the receive-pack may hang around for a long time, as the bottleneck is often client upload bandwidth). So let's use an oidset. Note that unlike a sha1-array it doesn't sort the output as a side effect. However, our output is at least stable, because for_each_alternate_ref() will give us the sha1s in ref-sorted order. In one particularly pathological case with an alternate that has 60,000 unique refs out of 80 million total, this reduced the peak heap usage of "git receive-pack . </dev/null" from 13GB to 14MB. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Jeff King committed Feb 8, 2017 at 15:53 UTC ab6eea6f7b9a5289d72c05476da19ab2bb457fd3

1 file changed +12 -14

builtin/receive-pack.c

+12 -14

@@ -21,6 +21,7 @@

21 #include "sigchain.h"

22 #include "fsck.h"

23 #include "tmp-objdir.h"

24 +#include "oidset.h"

26 static const char * const receive_pack_usage[] = {

27 N_("git receive-pack <git-dir>"),

@@ -271,27 +272,24 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,

272 return 0;

273 }

274

274 -static int show_one_alternate_sha1(const unsigned char sha1[20], void *unused)

275 +static void show_one_alternate_ref(const char *refname,

276 + const struct object_id *oid,

277 + void *data)

278 {

276 - show_ref(".have", sha1);

277 - return 0;

278 -}

279 + struct oidset *seen = data;

280

280 -static void collect_one_alternate_ref(const char *refname,

281 - const struct object_id *oid,

282 - void *data)

283 -{

284 - struct sha1_array *sa = data;

285 - sha1_array_append(sa, oid->hash);

281 + if (oidset_insert(seen, oid))

282 + return;

283 +

284 + show_ref(".have", oid->hash);

285 }

286

287 static void write_head_info(void)

288 {

290 - struct sha1_array sa = SHA1_ARRAY_INIT;

289 + static struct oidset seen = OIDSET_INIT;

290

292 - for_each_alternate_ref(collect_one_alternate_ref, &sa);

293 - sha1_array_for_each_unique(&sa, show_one_alternate_sha1, NULL);

294 - sha1_array_clear(&sa);

291 + for_each_alternate_ref(show_one_alternate_ref, &seen);

292 + oidset_clear(&seen);

293 for_each_ref(show_ref_cb, NULL);

294 if (!sent_capabilities)

295 show_ref("capabilities^{}", null_sha1);