@matheusbsbs / matheus-bagliano / commits / ca5381d43e
Fork 0

pack-objects: finishing touches.

This introduces --no-reuse-delta option to disable reusing of existing delta, which is a large part of the optimization introduced by this series. This may become necessary if repeated repacking makes delta chain too long. With this, the output of the command becomes identical to that of the older implementation. But the performance suffers greatly. It still allows reusing non-deltified representations; there is no point uncompressing and recompressing the whole text. It also adds a couple more statistics output, while squelching it under -q flag, which the last round forgot to do. $ time old-git-pack-objects --stdout >/dev/null <RL Generating pack... Done counting 184141 objects. Packing 184141 objects.................... real 12m8.530s user 11m1.450s sys 0m57.920s $ time git-pack-objects --stdout >/dev/null <RL Generating pack... Done counting 184141 objects. Packing 184141 objects..................... Total 184141, written 184141 (delta 138297), reused 178833 (delta 134081) real 0m59.549s user 0m56.670s sys 0m2.400s $ time git-pack-objects --stdout --no-reuse-delta >/dev/null <RL Generating pack... Done counting 184141 objects. Packing 184141 objects..................... Total 184141, written 184141 (delta 134833), reused 47904 (delta 0) real 11m13.830s user 9m45.240s sys 0m44.330s There is one remaining issue when --no-reuse-delta option is not used. It can create delta chains that are deeper than specified. A<--B<--C<--D E F G Suppose we have a delta chain A to D (A is stored in full either in a pack or as a loose object. B is depth1 delta relative to A, C is depth2 delta relative to B...) with loose objects E, F, G. And we are going to pack all of them. B, C and D are left as delta against A, B and C respectively. So A, E, F, and G are examined for deltification, and let's say we decided to keep E expanded, and store the rest as deltas like this: E<--F<--G<--A Oops. We ended up making D a bit too deep, didn't we? B, C and D form a chain on top of A! This is because we did not know what the final depth of A would be, when we checked objects and decided to keep the existing delta. Unfortunately, deferring the decision until just before the deltification is not an option. To be able to make B, C, and D candidates for deltification with the rest, we need to know the type and final unexpanded size of them, but the major part of the optimization comes from the fact that we do not read the delta data to do so -- getting the final size is quite an expensive operation. To prevent this from happening, we should keep A from being deltified. But how would we tell that, cheaply? To do this most precisely, after check_object() runs, each object that is used as the base object of some existing delta needs to be marked with the maximum depth of the objects we decided to keep deltified (in this case, D is depth 3 relative to A, so if no other delta chain that is longer than 3 based on A exists, mark A with 3). Then when attempting to deltify A, we would take that number into account to see if the final delta chain that leads to D becomes too deep. However, this is a bit cumbersome to compute, so we would cheat and reduce the maximum depth for A arbitrarily to depth/4 in this implementation. Signed-off-by: Junio C Hamano <junkio@cox.net>

Junio C Hamano committed Feb 16, 2006 at 11:55 UTC ca5381d43e8595f592d8d7ecfc9bb0bfa5e52f6d
2 files changed +91 -32
Documentation/git-pack-objects.txt
+20 -1
@@ -8,7 +8,10 @@ git-pack-objects - Create a packed archive of objects.
8
9 SYNOPSIS
10 --------
11 -'git-pack-objects' [--non-empty] [--local] [--incremental] [--window=N] [--depth=N] {--stdout | base-name} < object-list
11 +[verse]
12 +'git-pack-objects' [-q] [--no-reuse-delta] [--non-empty]
13 + [--local] [--incremental] [--window=N] [--depth=N]
14 + {--stdout | base-name} < object-list
15
16
17 DESCRIPTION
@@ -32,6 +35,10 @@ Placing both in the pack/ subdirectory of $GIT_OBJECT_DIRECTORY (or
35 any of the directories on $GIT_ALTERNATE_OBJECT_DIRECTORIES)
36 enables git to read from such an archive.
37
38 +In a packed archive, an object is either stored as a compressed
39 +whole, or as a difference from some other object. The latter is
40 +often called a delta.
41 +
42
43 OPTIONS
44 -------
@@ -74,6 +81,18 @@ base-name::
81 Only create a packed archive if it would contain at
82 least one object.
83
84 +-q::
85 + This flag makes the command not to report its progress
86 + on the standard error stream.
87 +
88 +--no-reuse-delta::
89 + When creating a packed archive in a repository that
90 + has existing packs, the command reuses existing deltas.
91 + This sometimes results in a slightly suboptimal pack.
92 + This flag tells the command not to reuse existing deltas
93 + but compute them from scratch.
94 +
95 +
96 Author
97 ------
98 Written by Linus Torvalds <torvalds@osdl.org>
pack-objects.c
+71 -31
@@ -5,7 +5,7 @@
5 #include "csum-file.h"
6 #include <sys/time.h>
7
8 -static const char pack_usage[] = "git-pack-objects [-q] [--non-empty] [--local] [--incremental] [--window=N] [--depth=N] {--stdout | base-name} < object-list";
8 +static const char pack_usage[] = "git-pack-objects [-q] [--no-reuse-delta] [--non-empty] [--local] [--incremental] [--window=N] [--depth=N] {--stdout | base-name} < object-list";
9
10 struct object_entry {
11 unsigned char sha1[20];
@@ -14,10 +14,11 @@ struct object_entry {
14 unsigned int depth; /* delta depth */
15 unsigned int hash; /* name hint hash */
16 enum object_type type;
17 + unsigned char edge; /* reused delta chain points at this entry. */
18 + enum object_type in_pack_type; /* could be delta */
19 unsigned long delta_size; /* delta data size (uncompressed) */
20 struct object_entry *delta; /* delta base object */
21 struct packed_git *in_pack; /* already in pack */
20 - enum object_type in_pack_type; /* could be delta */
22 unsigned int in_pack_offset;
23 };
24
@@ -36,6 +37,7 @@ struct object_entry {
37
38 static unsigned char object_list_sha1[20];
39 static int non_empty = 0;
40 +static int no_reuse_delta = 0;
41 static int local = 0;
42 static int incremental = 0;
43 static struct object_entry **sorted_by_sha, **sorted_by_type;
@@ -75,7 +77,9 @@ static int pack_revindex_hashsz = 0;
77 * stats
78 */
79 static int written = 0;
80 +static int written_delta = 0;
81 static int reused = 0;
82 +static int reused_delta = 0;
83
84 static int pack_revindex_ix(struct packed_git *p)
85 {
@@ -227,10 +231,23 @@ static unsigned long write_object(struct sha1file *f, struct object_entry *entry
231 unsigned char header[10];
232 unsigned hdrlen, datalen;
233 enum object_type obj_type;
234 + int to_reuse = 0;
235
236 obj_type = entry->type;
232 - if (!entry->in_pack ||
233 - (obj_type != entry->in_pack_type)) {
237 + if (! entry->in_pack)
238 + to_reuse = 0; /* can't reuse what we don't have */
239 + else if (obj_type == OBJ_DELTA)
240 + to_reuse = 1; /* check_object() decided it for us */
241 + else if (obj_type != entry->in_pack_type)
242 + to_reuse = 0; /* pack has delta which is unusable */
243 + else if (entry->delta)
244 + to_reuse = 0; /* we want to pack afresh */
245 + else
246 + to_reuse = 1; /* we have it in-pack undeltified,
247 + * and we do not need to deltify it.
248 + */
249 +
250 + if (! to_reuse) {
251 buf = read_sha1_file(entry->sha1, type, &size);
252 if (!buf)
253 die("unable to read %s", sha1_to_hex(entry->sha1));
@@ -266,8 +283,12 @@ static unsigned long write_object(struct sha1file *f, struct object_entry *entry
283 sha1write(f, buf, datalen);
284 unuse_packed_git(p);
285 hdrlen = 0; /* not really */
286 + if (obj_type == OBJ_DELTA)
287 + reused_delta++;
288 reused++;
289 }
290 + if (obj_type == OBJ_DELTA)
291 + written_delta++;
292 written++;
293 return hdrlen + datalen;
294 }
@@ -294,7 +315,6 @@ static void write_pack_file(void)
315 int i;
316 struct sha1file *f;
317 unsigned long offset;
297 - unsigned long mb;
318 struct pack_header hdr;
319
320 if (!base_name)
@@ -357,10 +377,9 @@ static int add_object_entry(unsigned char *sha1, unsigned int hash)
377 unsigned int idx = nr_objects;
378 struct object_entry *entry;
379 struct packed_git *p;
360 - unsigned int found_offset;
361 - struct packed_git *found_pack;
380 + unsigned int found_offset = 0;
381 + struct packed_git *found_pack = NULL;
382
363 - found_pack = NULL;
383 for (p = packed_git; p; p = p->next) {
384 struct pack_entry e;
385 if (find_pack_entry_one(sha1, &e, p)) {
@@ -420,32 +439,39 @@ static void check_object(struct object_entry *entry)
439 char type[20];
440
441 if (entry->in_pack) {
442 + unsigned char base[20];
443 + unsigned long size;
444 + struct object_entry *base_entry;
445 +
446 + /* We want in_pack_type even if we do not reuse delta.
447 + * There is no point not reusing non-delta representations.
448 + */
449 + check_reuse_pack_delta(entry->in_pack,
450 + entry->in_pack_offset,
451 + base, &size,
452 + &entry->in_pack_type);
453 +
454 /* Check if it is delta, and the base is also an object
455 * we are going to pack. If so we will reuse the existing
456 * delta.
457 */
427 - unsigned char base[20];
428 - unsigned long size;
429 - struct object_entry *base_entry;
430 - if (!check_reuse_pack_delta(entry->in_pack,
431 - entry->in_pack_offset,
432 - base, &size,
433 - &entry->in_pack_type) &&
458 + if (!no_reuse_delta &&
459 + entry->in_pack_type == OBJ_DELTA &&
460 (base_entry = locate_object_entry(base))) {
435 - /* We do not know depth at this point, but it
436 - * does not matter. Getting delta_chain_length
437 - * with packed_object_info_detail() is not so
438 - * expensive, so we could do that later if we
439 - * wanted to. Calling sha1_object_info to get
440 - * the true size (and later an uncompressed
441 - * representation) of deeply deltified object
442 - * is quite expensive.
461 +
462 + /* Depth value does not matter - find_deltas()
463 + * will never consider reused delta as the
464 + * base object to deltify other objects
465 + * against, in order to avoid circular deltas.
466 */
444 - entry->depth = 1;
445 - /* uncompressed size */
467 +
468 + /* uncompressed size of the delta data */
469 entry->size = entry->delta_size = size;
470 entry->delta = base_entry;
471 entry->type = OBJ_DELTA;
472 +
473 + base_entry->edge = 1;
474 +
475 return;
476 }
477 /* Otherwise we would do the usual */
@@ -568,6 +594,13 @@ static int try_delta(struct unpacked *cur, struct unpacked *old, unsigned max_de
594 if (cur_entry->type != old_entry->type)
595 return -1;
596
597 + /* If the current object is at edge, take the depth the objects
598 + * that depend on the current object into account -- otherwise
599 + * they would become too deep.
600 + */
601 + if (cur_entry->edge)
602 + max_depth /= 4;
603 +
604 size = cur_entry->size;
605 if (size < 50)
606 return -1;
@@ -627,7 +660,7 @@ static void find_deltas(struct object_entry **list, int window, int depth)
660
661 if (entry->delta)
662 /* This happens if we decided to reuse existing
630 - * delta from a pack.
663 + * delta from a pack. "!no_reuse_delta &&" is implied.
664 */
665 continue;
666
@@ -636,6 +669,7 @@ static void find_deltas(struct object_entry **list, int window, int depth)
669 n->data = read_sha1_file(entry->sha1, type, &size);
670 if (size != entry->size)
671 die("object %s inconsistent object length (%lu vs %lu)", sha1_to_hex(entry->sha1), size, entry->size);
672 +
673 j = window;
674 while (--j > 0) {
675 unsigned int other_idx = idx + j;
@@ -664,7 +698,7 @@ static void prepare_pack(int window, int depth)
698 fprintf(stderr, "Packing %d objects", nr_objects);
699 get_object_details();
700 if (progress)
667 - fprintf(stderr, ".");
701 + fputc('.', stderr);
702
703 sorted_by_type = create_sorted_list(type_size_sort);
704 if (window && depth)
@@ -694,8 +728,9 @@ static int reuse_cached_pack(unsigned char *sha1, int pack_to_stdout)
728 }
729 }
730
697 - fprintf(stderr, "Reusing %d objects pack %s\n", nr_objects,
698 - sha1_to_hex(sha1));
731 + if (progress)
732 + fprintf(stderr, "Reusing %d objects pack %s\n", nr_objects,
733 + sha1_to_hex(sha1));
734
735 if (pack_to_stdout) {
736 if (copy_fd(ifd, 1))
@@ -775,6 +810,10 @@ int main(int argc, char **argv)
810 progress = 0;
811 continue;
812 }
813 + if (!strcmp("--no-reuse-delta", arg)) {
814 + no_reuse_delta = 1;
815 + continue;
816 + }
817 if (!strcmp("--stdout", arg)) {
818 pack_to_stdout = 1;
819 continue;
@@ -850,7 +889,8 @@ int main(int argc, char **argv)
889 puts(sha1_to_hex(object_list_sha1));
890 }
891 }
853 - fprintf(stderr, "Total %d, written %d, reused %d\n",
854 - nr_objects, written, reused);
892 + if (progress)
893 + fprintf(stderr, "Total %d, written %d (delta %d), reused %d (delta %d)\n",
894 + nr_objects, written, written_delta, reused, reused_delta);
895 return 0;
896 }