@matheusbsbs / matheus-bagliano / commits / 6fdfaf15a0

reftable/stack: use stat info to avoid re-reading stack list

Whenever we call into the refs interfaces we potentially have to reload refs in case they have been concurrently modified, either in-process or externally. While this happens somewhat automatically for loose refs because we simply try to re-read the files, the "packed" backend will reload its snapshot of the packed-refs file in case its stat info has changed since last reading it. In the reftable backend we have a similar mechanism that is provided by `reftable_stack_reload()`. This function will read the list of stacks from "tables.list" and, if they have changed from the currently stored list, reload the stacks. This is heavily inefficient though, as we have to check whether the stack is up-to-date on basically every read and thus keep on re-reading the file all the time even if it didn't change at all. We can do better and use the same stat(3P)-based mechanism that the "packed" backend uses. Instead of reading the file, we will only open the file descriptor, fstat(3P) it, and then compare the info against the cached value from the last time we have updated the stack. This should always work alright because "tables.list" is updated atomically via a rename, so even if the ctime or mtime wasn't granular enough to identify a change, at least the inode number or file size should have changed. This change significantly speeds up operations where many refs are read, like when using git-update-ref(1). The following benchmark creates N refs in an otherwise-empty repository via `git update-ref --stdin`: Benchmark 1: update-ref: create many refs (refcount = 1, revision = HEAD~) Time (mean ± σ): 5.1 ms ± 0.2 ms [User: 2.4 ms, System: 2.6 ms] Range (min … max): 4.8 ms … 7.2 ms 109 runs Benchmark 2: update-ref: create many refs (refcount = 100, revision = HEAD~) Time (mean ± σ): 19.1 ms ± 0.9 ms [User: 8.9 ms, System: 9.9 ms] Range (min … max): 18.4 ms … 26.7 ms 72 runs Benchmark 3: update-ref: create many refs (refcount = 10000, revision = HEAD~) Time (mean ± σ): 1.336 s ± 0.018 s [User: 0.590 s, System: 0.724 s] Range (min … max): 1.314 s … 1.373 s 10 runs Benchmark 4: update-ref: create many refs (refcount = 1, revision = HEAD) Time (mean ± σ): 5.1 ms ± 0.2 ms [User: 2.4 ms, System: 2.6 ms] Range (min … max): 4.8 ms … 7.2 ms 109 runs Benchmark 5: update-ref: create many refs (refcount = 100, revision = HEAD) Time (mean ± σ): 14.8 ms ± 0.2 ms [User: 7.1 ms, System: 7.5 ms] Range (min … max): 14.2 ms … 15.2 ms 82 runs Benchmark 6: update-ref: create many refs (refcount = 10000, revision = HEAD) Time (mean ± σ): 927.6 ms ± 5.3 ms [User: 437.8 ms, System: 489.5 ms] Range (min … max): 919.4 ms … 936.4 ms 10 runs Summary update-ref: create many refs (refcount = 1, revision = HEAD) ran 1.00 ± 0.07 times faster than update-ref: create many refs (refcount = 1, revision = HEAD~) 2.89 ± 0.14 times faster than update-ref: create many refs (refcount = 100, revision = HEAD) 3.74 ± 0.25 times faster than update-ref: create many refs (refcount = 100, revision = HEAD~) 181.26 ± 8.30 times faster than update-ref: create many refs (refcount = 10000, revision = HEAD) 261.01 ± 12.35 times faster than update-ref: create many refs (refcount = 10000, revision = HEAD~) Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Patrick Steinhardt committed Jan 11, 2024 at 11:06 UTC 6fdfaf15a0c60572ac58c979a46e34634051e12f

3 files changed +13 -1

reftable/stack.c

+11 -1

@@ -175,6 +175,7 @@ void reftable_stack_destroy(struct reftable_stack *st)

175 st->readers_len = 0;

176 FREE_AND_NULL(st->readers);

177 }

178 + stat_validity_clear(&st->list_validity);

179 FREE_AND_NULL(st->list_file);

180 FREE_AND_NULL(st->reftable_dir);

181 reftable_free(st);

@@ -374,7 +375,11 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,

375 sleep_millisec(delay);

376 }

377

378 + stat_validity_update(&st->list_validity, fd);

379 +

380 out:

381 + if (err)

382 + stat_validity_clear(&st->list_validity);

383 if (fd >= 0)

384 close(fd);

385 free_names(names);

@@ -388,8 +393,13 @@ out:

393 static int stack_uptodate(struct reftable_stack *st)

394 {

395 char **names = NULL;

391 - int err = read_lines(st->list_file, &names);

396 + int err;

397 int i = 0;

398 +

399 + if (stat_validity_check(&st->list_validity, st->list_file))

400 + return 0;

401 +

402 + err = read_lines(st->list_file, &names);

403 if (err < 0)

404 return err;

405

reftable/stack.h

@@ -14,6 +14,7 @@ https://developers.google.com/open-source/licenses/bsd

14 #include "reftable-stack.h"

16 struct reftable_stack {

17 + struct stat_validity list_validity;

18 char *list_file;

19 char *reftable_dir;

20 int disable_auto_compact;

reftable/system.h

@@ -12,6 +12,7 @@ https://developers.google.com/open-source/licenses/bsd

12 /* This header glues the reftable library to the rest of Git */

14 #include "git-compat-util.h"

15 +#include "statinfo.h"

16 #include "strbuf.h"

17 #include "hash-ll.h" /* hash ID, sizes.*/

18 #include "dir.h" /* remove_dir_recursively, for tests.*/