sparse-checkout: integrate with sparse index

When modifying the sparse-checkout definition, the sparse-checkout builtin calls update_sparsity() to modify the SKIP_WORKTREE bits of all cache entries in the index. Before, we needed the index to be fully expanded in order to ensure we had the full list of files necessary that match the new patterns. Insert a call to reset_sparse_directories() that expands sparse directories that are within the new pattern list, but only far enough that every necessary file path now exists as a cache entry. The remaining logic within update_sparsity() will modify the SKIP_WORKTREE bits appropriately. This allows us to disable command_requires_full_index within the sparse-checkout builtin. Add tests that demonstrate that we are not expanding to a full index unnecessarily. We can see the improved performance in the p2000 test script: Test HEAD~1 HEAD ------------------------------------------------------------------------ 2000.24: git ... (sparse-v3) 2.14(1.55+0.58) 1.57(1.03+0.53) -26.6% 2000.25: git ... (sparse-v4) 2.20(1.62+0.57) 1.58(0.98+0.59) -28.2% These reductions of 26-28% are small compared to most examples, but the time is dominated by writing a new copy of the base repository to the worktree and then deleting it again. The fact that the previous index expansion was such a large portion of the time is telling how important it is to complete this sparse index integration. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Derrick Stolee committed May 23, 2022 at 13:48 UTC 598b1e7d0982fd71a25d861dccc1d580ef14ac90
3 files changed +32
builtin/sparse-checkout.c
+3
@@ -937,6 +937,9 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)
937
938 git_config(git_default_config, NULL);
939
940 + prepare_repo_settings(the_repository);
941 + the_repository->settings.command_requires_full_index = 0;
942 +
943 if (argc > 0) {
944 if (!strcmp(argv[0], "list"))
945 return sparse_checkout_list(argc, argv);
t/t1092-sparse-checkout-compatibility.sh
+25
@@ -1548,6 +1548,31 @@ test_expect_success 'ls-files' '
1548 ensure_not_expanded ls-files --sparse
1549 '
1550
1551 +test_expect_success 'sparse index is not expanded: sparse-checkout' '
1552 + init_repos &&
1553 +
1554 + ensure_not_expanded sparse-checkout set deep/deeper2 &&
1555 + ensure_not_expanded sparse-checkout set deep/deeper1 &&
1556 + ensure_not_expanded sparse-checkout set deep &&
1557 + ensure_not_expanded sparse-checkout add folder1 &&
1558 + ensure_not_expanded sparse-checkout set deep/deeper1 &&
1559 + ensure_not_expanded sparse-checkout set folder2 &&
1560 +
1561 + # Demonstrate that the checks that "folder1/a" is a file
1562 + # do not cause a sparse-index expansion (since it is in the
1563 + # sparse-checkout cone).
1564 + echo >>sparse-index/folder2/a &&
1565 + git -C sparse-index add folder2/a &&
1566 +
1567 + ensure_not_expanded sparse-checkout add folder1 &&
1568 +
1569 + # Skip checks here, since deep/deeper1 is inside a sparse directory
1570 + # that must be expanded to check whether `deep/deeper1` is a file
1571 + # or not.
1572 + ensure_not_expanded sparse-checkout set --skip-checks deep/deeper1 &&
1573 + ensure_not_expanded sparse-checkout set
1574 +'
1575 +
1576 # NEEDSWORK: a sparse-checkout behaves differently from a full checkout
1577 # in this scenario, but it shouldn't.
1578 test_expect_success 'reset mixed and checkout orphan' '
unpack-trees.c
+4
@@ -18,6 +18,7 @@
18 #include "promisor-remote.h"
19 #include "entry.h"
20 #include "parallel-checkout.h"
21 +#include "sparse-index.h"
22
23 /*
24 * Error messages expected by scripts out of plumbing commands such as
@@ -2018,6 +2019,9 @@ enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)
2019 goto skip_sparse_checkout;
2020 }
2021
2022 + /* Expand sparse directories as needed */
2023 + expand_index(o->src_index, o->pl);
2024 +
2025 /* Set NEW_SKIP_WORKTREE on existing entries. */
2026 mark_all_ce_unused(o->src_index);
2027 mark_new_skip_worktree(o->pl, o->src_index, 0,