@matheusbsbs / matheus-bagliano / commits / 7b0f229222

commit-graph write: add progress output

Before this change the "commit-graph write" command didn't report any progress. On my machine this command takes more than 10 seconds to write the graph for linux.git, and around 1m30s on the 2015-04-03-1M-git.git[1] test repository (a test case for a large monorepository). Furthermore, since the gc.writeCommitGraph setting was added in d5d5d7b641 ("gc: automatically write commit-graph files", 2018-06-27), there was no indication at all from a "git gc" run that anything was different. This why one of the progress bars being added here uses start_progress() instead of start_delayed_progress(), so that it's guaranteed to be seen. E.g. on my tiny 867 commit dotfiles.git repository: $ git -c gc.writeCommitGraph=true gc Enumerating objects: 2821, done. [...] Computing commit graph generation numbers: 100% (867/867), done. On larger repositories, such as linux.git the delayed progress bar(s) will kick in, and we'll show what's going on instead of, as was previously happening, printing nothing while we write the graph: $ git -c gc.writeCommitGraph=true gc [...] Annotating commits in commit graph: 1565573, done. Computing commit graph generation numbers: 100% (782484/782484), done. Note that here we don't show "Finding commits for commit graph", this is because under "git gc" we seed the search with the commit references in the repository, and that set is too small to show any progress, but would e.g. on a smaller repo such as git.git with --stdin-commits: $ git rev-list --all | git -c gc.writeCommitGraph=true write --stdin-commits Finding commits for commit graph: 100% (162576/162576), done. Computing commit graph generation numbers: 100% (162576/162576), done. With --stdin-packs we don't show any estimation of how much is left to do. This is because we might be processing more than one pack. We could be less lazy here and show progress, either by detecting that we're only processing one pack, or by first looping over the packs to discover how many commits they have. I don't see the point in doing that work. So instead we get (on 2015-04-03-1M-git.git): $ echo pack-<HASH>.idx | git -c gc.writeCommitGraph=true --exec-path=$PWD commit-graph write --stdin-packs Finding commits for commit graph: 13064614, done. Annotating commits in commit graph: 3001341, done. Computing commit graph generation numbers: 100% (1000447/1000447), done. No GC mode uses --stdin-packs. It's what they use at Microsoft to manually compute the generation numbers for their collection of large packs which are never coalesced. The reason we need a "report_progress" variable passed down from "git gc" is so that we don't report this output when we're running in the process "git gc --auto" detaches from the terminal. Since we write the commit graph from the "git gc" process itself (as opposed to what we do with say the "git repack" phase), we'd end up writing the output to .git/gc.log and reporting it to the user next time as part of the "The last gc run reported the following[...]" error, see 329e6e8794 ("gc: save log from daemonized gc --auto and print it next time", 2015-09-19). So we must keep track of whether or not we're running in that demonized mode, and if so print no progress. See [2] and subsequent replies for a discussion of an approach not taken in compute_generation_numbers(). I.e. we're saying "Computing commit graph generation numbers", even though on an established history we're mostly skipping over all the work we did in the past. This is similar to the white lie we tell in the "Writing objects" phase (not all are objects being written). Always showing progress is considered more important than accuracy. I.e. on a repository like 2015-04-03-1M-git.git we'd hang for 6 seconds with no output on the second "git gc" if no changes were made to any objects in the interim if we'd take the approach in [2]. 1. https://github.com/avar/2015-04-03-1M-git 2. <c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com> (https://public-inbox.org/git/c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com/) Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Ævar Arnfjörð Bjarmason committed Sep 17, 2018 at 15:33 UTC 7b0f2292224cec841de78acc911e4c378ea79faa

4 files changed +60 -13

builtin/commit-graph.c

+3 -2

@@ -151,7 +151,7 @@ static int graph_write(int argc, const char **argv)

151 opts.obj_dir = get_object_directory();

152

153 if (opts.reachable) {

154 - write_commit_graph_reachable(opts.obj_dir, opts.append);

154 + write_commit_graph_reachable(opts.obj_dir, opts.append, 1);

155 return 0;

156 }

157

@@ -171,7 +171,8 @@ static int graph_write(int argc, const char **argv)

171 write_commit_graph(opts.obj_dir,

172 pack_indexes,

173 commit_hex,

174 - opts.append);

174 + opts.append,

175 + 1);

176

177 string_list_clear(&lines, 0);

178 return 0;

builtin/gc.c

+2 -1

@@ -646,7 +646,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)

646 clean_pack_garbage();

647

648 if (gc_write_commit_graph)

649 - write_commit_graph_reachable(get_object_directory(), 0);

649 + write_commit_graph_reachable(get_object_directory(), 0,

650 + !daemonized);

651

652 if (auto_gc && too_many_loose_objects())

653 warning(_("There are too many unreachable loose objects; "

commit-graph.c

+52 -8

@@ -13,6 +13,7 @@

13 #include "commit-graph.h"

14 #include "object-store.h"

15 #include "alloc.h"

16 +#include "progress.h"

18 #define GRAPH_SIGNATURE 0x43475048 /* "CGPH" */

19 #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* "OIDF" */

@@ -548,6 +549,8 @@ struct packed_oid_list {

549 struct object_id *list;

550 int nr;

551 int alloc;

552 + struct progress *progress;

553 + int progress_done;

554 };

555

556 static int add_packed_commits(const struct object_id *oid,

@@ -560,6 +563,9 @@ static int add_packed_commits(const struct object_id *oid,

563 off_t offset = nth_packed_object_offset(pack, pos);

564 struct object_info oi = OBJECT_INFO_INIT;

565

566 + if (list->progress)

567 + display_progress(list->progress, ++list->progress_done);

568 +

569 oi.typep = &type;

570 if (packed_object_info(the_repository, pack, offset, &oi) < 0)

571 die(_("unable to get type of object %s"), oid_to_hex(oid));

@@ -587,12 +593,18 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com

593 }

594 }

595

590 -static void close_reachable(struct packed_oid_list *oids)

596 +static void close_reachable(struct packed_oid_list *oids, int report_progress)

597 {

598 int i;

599 struct commit *commit;

600 + struct progress *progress = NULL;

601 + int j = 0;

602

603 + if (report_progress)

604 + progress = start_delayed_progress(

605 + _("Annotating commits in commit graph"), 0);

606 for (i = 0; i < oids->nr; i++) {

607 + display_progress(progress, ++j);

608 commit = lookup_commit(the_repository, &oids->list[i]);

609 if (commit)

610 commit->object.flags |= UNINTERESTING;

@@ -604,6 +616,7 @@ static void close_reachable(struct packed_oid_list *oids)

616 * closure.

617 */

618 for (i = 0; i < oids->nr; i++) {

619 + display_progress(progress, ++j);

620 commit = lookup_commit(the_repository, &oids->list[i]);

621

622 if (commit && !parse_commit(commit))

@@ -611,19 +624,28 @@ static void close_reachable(struct packed_oid_list *oids)

624 }

625

626 for (i = 0; i < oids->nr; i++) {

627 + display_progress(progress, ++j);

628 commit = lookup_commit(the_repository, &oids->list[i]);

629

630 if (commit)

631 commit->object.flags &= ~UNINTERESTING;

632 }

633 + stop_progress(&progress);

634 }

635

621 -static void compute_generation_numbers(struct packed_commit_list* commits)

636 +static void compute_generation_numbers(struct packed_commit_list* commits,

637 + int report_progress)

638 {

639 int i;

640 struct commit_list *list = NULL;

641 + struct progress *progress = NULL;

642

643 + if (report_progress)

644 + progress = start_progress(

645 + _("Computing commit graph generation numbers"),

646 + commits->nr);

647 for (i = 0; i < commits->nr; i++) {

648 + display_progress(progress, i + 1);

649 if (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&

650 commits->list[i]->generation != GENERATION_NUMBER_ZERO)

651 continue;

@@ -655,6 +677,7 @@ static void compute_generation_numbers(struct packed_commit_list* commits)

677 }

678 }

679 }

680 + stop_progress(&progress);

681 }

682

683 static int add_ref_to_list(const char *refname,

@@ -667,19 +690,20 @@ static int add_ref_to_list(const char *refname,

690 return 0;

691 }

692

670 -void write_commit_graph_reachable(const char *obj_dir, int append)

693 +void write_commit_graph_reachable(const char *obj_dir, int append,

694 + int report_progress)

695 {

696 struct string_list list;

697

698 string_list_init(&list, 1);

699 for_each_ref(add_ref_to_list, &list);

676 - write_commit_graph(obj_dir, NULL, &list, append);

700 + write_commit_graph(obj_dir, NULL, &list, append, report_progress);

701 }

702

703 void write_commit_graph(const char *obj_dir,

704 struct string_list *pack_indexes,

705 struct string_list *commit_hex,

682 - int append)

706 + int append, int report_progress)

707 {

708 struct packed_oid_list oids;

709 struct packed_commit_list commits;

@@ -692,9 +716,12 @@ void write_commit_graph(const char *obj_dir,

716 int num_chunks;

717 int num_extra_edges;

718 struct commit_list *parent;

719 + struct progress *progress = NULL;

720

721 oids.nr = 0;

722 oids.alloc = approximate_object_count() / 4;

723 + oids.progress = NULL;

724 + oids.progress_done = 0;

725

726 if (append) {

727 prepare_commit_graph_one(the_repository, obj_dir);

@@ -721,6 +748,11 @@ void write_commit_graph(const char *obj_dir,

748 int dirlen;

749 strbuf_addf(&packname, "%s/pack/", obj_dir);

750 dirlen = packname.len;

751 + if (report_progress) {

752 + oids.progress = start_delayed_progress(

753 + _("Finding commits for commit graph"), 0);

754 + oids.progress_done = 0;

755 + }

756 for (i = 0; i < pack_indexes->nr; i++) {

757 struct packed_git *p;

758 strbuf_setlen(&packname, dirlen);

@@ -733,15 +765,21 @@ void write_commit_graph(const char *obj_dir,

765 for_each_object_in_pack(p, add_packed_commits, &oids, 0);

766 close_pack(p);

767 }

768 + stop_progress(&oids.progress);

769 strbuf_release(&packname);

770 }

771

772 if (commit_hex) {

773 + if (report_progress)

774 + progress = start_delayed_progress(

775 + _("Finding commits for commit graph"),

776 + commit_hex->nr);

777 for (i = 0; i < commit_hex->nr; i++) {

778 const char *end;

779 struct object_id oid;

780 struct commit *result;

781

782 + display_progress(progress, i + 1);

783 if (commit_hex->items[i].string &&

784 parse_oid_hex(commit_hex->items[i].string, &oid, &end))

785 continue;

@@ -754,12 +792,18 @@ void write_commit_graph(const char *obj_dir,

792 oids.nr++;

793 }

794 }

795 + stop_progress(&progress);

796 }

797

759 - if (!pack_indexes && !commit_hex)

798 + if (!pack_indexes && !commit_hex) {

799 + if (report_progress)

800 + oids.progress = start_delayed_progress(

801 + _("Finding commits for commit graph"), 0);

802 for_each_packed_object(add_packed_commits, &oids, 0);

803 + stop_progress(&oids.progress);

804 + }

805

762 - close_reachable(&oids);

806 + close_reachable(&oids, report_progress);

807

808 QSORT(oids.list, oids.nr, commit_compare);

809

@@ -799,7 +843,7 @@ void write_commit_graph(const char *obj_dir,

843 if (commits.nr >= GRAPH_PARENT_MISSING)

844 die(_("too many commits to write graph"));

845

802 - compute_generation_numbers(&commits);

846 + compute_generation_numbers(&commits, report_progress);

847

848 graph_name = get_commit_graph_filename(obj_dir);

849 if (safe_create_leading_directories(graph_name))

commit-graph.h

+3 -2

@@ -52,11 +52,12 @@ struct commit_graph {

53 struct commit_graph *load_commit_graph_one(const char *graph_file);

55 -void write_commit_graph_reachable(const char *obj_dir, int append);

55 +void write_commit_graph_reachable(const char *obj_dir, int append,

56 + int report_progress);

57 void write_commit_graph(const char *obj_dir,

58 struct string_list *pack_indexes,

59 struct string_list *commit_hex,

59 - int append);

60 + int append, int report_progress);

62 int verify_commit_graph(struct repository *r, struct commit_graph *g);