index-pack: check .gitmodules files with --strict

Now that the internal fsck code has all of the plumbing we need, we can start checking incoming .gitmodules files. Naively, it seems like we would just need to add a call to fsck_finish() after we've processed all of the objects. And that would be enough to cover the initial test included here. But there are two extra bits: 1. We currently don't bother calling fsck_object() at all for blobs, since it has traditionally been a noop. We'd actually catch these blobs in fsck_finish() at the end, but it's more efficient to check them when we already have the object loaded in memory. 2. The second pass done by fsck_finish() needs to access the objects, but we're actually indexing the pack in this process. In theory we could give the fsck code a special callback for accessing the in-pack data, but it's actually quite tricky: a. We don't have an internal efficient index mapping oids to packfile offsets. We only generate it on the fly as part of writing out the .idx file. b. We'd still have to reconstruct deltas, which means we'd basically have to replicate all of the reading logic in packfile.c. Instead, let's avoid running fsck_finish() until after we've written out the .idx file, and then just add it to our internal packed_git list. This does mean that the objects are "in the repository" before we finish our fsck checks. But unpack-objects already exhibits this same behavior, and it's an acceptable tradeoff here for the same reason: the quarantine mechanism means that pushes will be fully protected. In addition to a basic push test in t7415, we add a sneaky pack that reverses the usual object order in the pack, requiring that index-pack access the tree and blob during the "finish" step. This already works for unpack-objects (since it will have written out loose objects), but we'll check it with this sneaky pack for good measure. Signed-off-by: Jeff King <peff@peff.net>

Jeff King committed May 4, 2018 at 19:45 UTC 73c3f0f704a91b6792e0199a3f3ab6e3a1971675
3 files changed +60
builtin/index-pack.c
+10
@@ -836,6 +836,9 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,
836 blob->object.flags |= FLAG_CHECKED;
837 else
838 die(_("invalid blob object %s"), oid_to_hex(oid));
839 + if (do_fsck_object &&
840 + fsck_object(&blob->object, (void *)data, size, &fsck_options))
841 + die(_("fsck error in packed object"));
842 } else {
843 struct object *obj;
844 int eaten;
@@ -1477,6 +1480,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
1480 } else
1481 chmod(final_index_name, 0444);
1482
1483 + if (do_fsck_object)
1484 + add_packed_git(final_index_name, strlen(final_index_name), 0);
1485 +
1486 if (!from_stdin) {
1487 printf("%s\n", sha1_to_hex(hash));
1488 } else {
@@ -1818,6 +1824,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
1824 pack_hash);
1825 else
1826 close(input_fd);
1827 +
1828 + if (do_fsck_object && fsck_finish(&fsck_options))
1829 + die(_("fsck error in pack objects"));
1830 +
1831 free(objects);
1832 strbuf_release(&index_name_buf);
1833 if (pack_name == NULL)
t/lib-pack.sh
+12
@@ -79,6 +79,18 @@ pack_obj () {
79 ;;
80 esac
81
82 + # If it's not a delta, we can convince pack-objects to generate a pack
83 + # with just our entry, and then strip off the header (12 bytes) and
84 + # trailer (20 bytes).
85 + if test -z "$2"
86 + then
87 + echo "$1" | git pack-objects --stdout >pack_obj.tmp &&
88 + size=$(wc -c <pack_obj.tmp) &&
89 + dd if=pack_obj.tmp bs=1 count=$((size - 20 - 12)) skip=12 &&
90 + rm -f pack_obj.tmp
91 + return
92 + fi
93 +
94 echo >&2 "BUG: don't know how to print $1${2:+ (from $2)}"
95 return 1
96 }
t/t7415-submodule-names.sh
+38
@@ -6,6 +6,7 @@ Exercise the name-checking function on a variety of names, and then give a
6 real-world setup that confirms we catch this in practice.
7 '
8 . ./test-lib.sh
9 +. "$TEST_DIRECTORY"/lib-pack.sh
10
11 test_expect_success 'check names' '
12 cat >expect <<-\EOF &&
@@ -84,4 +85,41 @@ test_expect_success 'transfer.fsckObjects detects evil superproject (unpack)' '
85 test_must_fail git push dst.git HEAD
86 '
87
88 +test_expect_success 'transfer.fsckObjects detects evil superproject (index)' '
89 + rm -rf dst.git &&
90 + git init --bare dst.git &&
91 + git -C dst.git config transfer.fsckObjects true &&
92 + git -C dst.git config transfer.unpackLimit 1 &&
93 + test_must_fail git push dst.git HEAD
94 +'
95 +
96 +# Normally our packs contain commits followed by trees followed by blobs. This
97 +# reverses the order, which requires backtracking to find the context of a
98 +# blob. We'll start with a fresh gitmodules-only tree to make it simpler.
99 +test_expect_success 'create oddly ordered pack' '
100 + git checkout --orphan odd &&
101 + git rm -rf --cached . &&
102 + git add .gitmodules &&
103 + git commit -m odd &&
104 + {
105 + pack_header 3 &&
106 + pack_obj $(git rev-parse HEAD:.gitmodules) &&
107 + pack_obj $(git rev-parse HEAD^{tree}) &&
108 + pack_obj $(git rev-parse HEAD)
109 + } >odd.pack &&
110 + pack_trailer odd.pack
111 +'
112 +
113 +test_expect_success 'transfer.fsckObjects handles odd pack (unpack)' '
114 + rm -rf dst.git &&
115 + git init --bare dst.git &&
116 + test_must_fail git -C dst.git unpack-objects --strict <odd.pack
117 +'
118 +
119 +test_expect_success 'transfer.fsckObjects handles odd pack (index)' '
120 + rm -rf dst.git &&
121 + git init --bare dst.git &&
122 + test_must_fail git -C dst.git index-pack --strict --stdin <odd.pack
123 +'
124 +
125 test_done