@matheusbsbs / matheus-bagliano / commits / ba47d88795

t/perf: add performance tests for multi-pack reuse

To ensure that we don't regress either the size or runtime performance of multi-pack reuse, add a performance test to measure both of these. The test partitions the objects in GIT_TEST_PERF_LARGE_REPO into 1, 10, and 100 packs, and then tries to perform a "clone" at each stage with both single- and multi-pack reuse enabled. Note that the `repack_into_n_chunks()` function in this new test script differs from the existing `repack_into_n()`. The former partitions the repository into N equal-sized chunks, while the latter produces N packs of five commits each (plus their objects), and then another pack with the remainder. On git.git, I can produce the following results on my machine: Test this tree -------------------------------------------------------------------------------- 5332.3: clone for 1-pack scenario (single-pack reuse) 1.57(2.99+0.15) 5332.4: clone size for 1-pack scenario (single-pack reuse) 231.8M 5332.5: clone for 1-pack scenario (multi-pack reuse) 1.79(2.96+0.21) 5332.6: clone size for 1-pack scenario (multi-pack reuse) 231.7M 5332.9: clone for 10-pack scenario (single-pack reuse) 3.89(16.75+0.35) 5332.10: clone size for 10-pack scenario (single-pack reuse) 209.9M 5332.11: clone for 10-pack scenario (multi-pack reuse) 1.56(2.99+0.17) 5332.12: clone size for 10-pack scenario (multi-pack reuse) 224.4M 5332.15: clone for 100-pack scenario (single-pack reuse) 8.24(54.31+0.59) 5332.16: clone size for 100-pack scenario (single-pack reuse) 278.3M 5332.17: clone for 100-pack scenario (multi-pack reuse) 2.13(2.44+0.33) 5332.18: clone size for 100-pack scenario (multi-pack reuse) 357.9M Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Taylor Blau committed Dec 14, 2023 at 17:24 UTC ba47d88795e12193e7b0fffc5130757a5517a5da
1 file changed +81
t/perf/p5332-multi-pack-reuse.sh new
+81
@@ -0,0 +1,81 @@
1 +#!/bin/sh
2 +
3 +test_description='tests pack performance with multi-pack reuse'
4 +
5 +. ./perf-lib.sh
6 +. "${TEST_DIRECTORY}/perf/lib-pack.sh"
7 +
8 +packdir=.git/objects/pack
9 +
10 +test_perf_large_repo
11 +
12 +find_pack () {
13 + for idx in $packdir/pack-*.idx
14 + do
15 + if git show-index <$idx | grep -q "$1"
16 + then
17 + basename $idx
18 + fi || return 1
19 + done
20 +}
21 +
22 +repack_into_n_chunks () {
23 + git repack -adk &&
24 +
25 + test "$1" -eq 1 && return ||
26 +
27 + find $packdir -type f | sort >packs.before &&
28 +
29 + # partition the repository into $1 chunks of consecutive commits, and
30 + # then create $1 packs with the objects reachable from each chunk
31 + # (excluding any objects reachable from the previous chunks)
32 + sz="$(($(git rev-list --count --all) / $1))"
33 + for rev in $(git rev-list --all | awk "NR % $sz == 0" | tac)
34 + do
35 + pack="$(echo "$rev" | git pack-objects --revs \
36 + --honor-pack-keep --delta-base-offset $packdir/pack)" &&
37 + touch $packdir/pack-$pack.keep || return 1
38 + done
39 +
40 + # grab any remaining objects not packed by the previous step(s)
41 + git pack-objects --revs --all --honor-pack-keep --delta-base-offset \
42 + $packdir/pack &&
43 +
44 + find $packdir -type f | sort >packs.after &&
45 +
46 + # and install the whole thing
47 + for f in $(comm -12 packs.before packs.after)
48 + do
49 + rm -f "$f" || return 1
50 + done
51 + rm -fr $packdir/*.keep
52 +}
53 +
54 +for nr_packs in 1 10 100
55 +do
56 + test_expect_success "create $nr_packs-pack scenario" '
57 + repack_into_n_chunks $nr_packs
58 + '
59 +
60 + test_expect_success "setup bitmaps for $nr_packs-pack scenario" '
61 + find $packdir -type f -name "*.idx" | sed -e "s/.*\/\(.*\)$/+\1/g" |
62 + git multi-pack-index write --stdin-packs --bitmap \
63 + --preferred-pack="$(find_pack $(git rev-parse HEAD))"
64 + '
65 +
66 + for reuse in single multi
67 + do
68 + test_perf "clone for $nr_packs-pack scenario ($reuse-pack reuse)" "
69 + git for-each-ref --format='%(objectname)' refs/heads refs/tags >in &&
70 + git -c pack.allowPackReuse=$reuse pack-objects \
71 + --revs --delta-base-offset --use-bitmap-index \
72 + --stdout <in >result
73 + "
74 +
75 + test_size "clone size for $nr_packs-pack scenario ($reuse-pack reuse)" '
76 + wc -c <result
77 + '
78 + done
79 +done
80 +
81 +test_done