fast-import: add 'ls' command
Lazy fast-import frontend authors that want to rely on the backend to keep track of the content of the imported trees _almost_ have what they need in the 'cat-blob' command (v1.7.4-rc0~30^2~3, 2010-11-28). But it is not quite enough, since (1) cat-blob can be used to retrieve the content of files, but not their mode, and (2) using cat-blob requires the frontend to keep track of a name (mark number or object id) for each blob to be retrieved Introduce an 'ls' command to complement cat-blob and take care of the remaining needs. The 'ls' command finds what is at a given path within a given tree-ish (tag, commit, or tree): 'ls' SP <dataref> SP <path> LF or in fast-import's active commit: 'ls' SP <path> LF The response is a single line sent through the cat-blob channel, imitating ls-tree output. So for example: FE> ls :1 Documentation gfi> 040000 tree 9e6c2b599341d28a2a375f8207507e0a2a627fe9 Documentation FE> ls 9e6c2b599341d28a2a375f8207507e0a2a627fe9 git-fast-import.txt gfi> 100644 blob 4f92954396e3f0f97e75b6838a5635b583708870 git-fast-import.txt FE> ls :1 RelNotes gfi> 120000 blob b942e499449d97aeb50c73ca2bdc1c6e6d528743 RelNotes FE> cat-blob b942e499449d97aeb50c73ca2bdc1c6e6d528743 gfi> b942e499449d97aeb50c73ca2bdc1c6e6d528743 blob 32 gfi> Documentation/RelNotes/1.7.4.txt The most interesting parts of the reply are the first word, which is a 6-digit octal mode (regular file, executable, symlink, directory, or submodule), and the part from the second space to the tab, which is a <dataref> that can be used in later cat-blob, ls, and filemodify (M) commands to refer to the content (blob, tree, or commit) at that path. If there is nothing there, the response is "missing some/path". The intent is for this command to be used to read files from the active commit, so a frontend can apply patches to them, and to copy files and directories from previous revisions. For example, proposed updates to svn-fe use this command in place of its internal representation of the repository directory structure. This simplifies the frontend a great deal and means support for resuming an import in a separate fast-import run (i.e., incremental import) is basically free. Signed-off-by: David Barr <david.barr@cordelta.com> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Improved-by: Junio C Hamano <gitster@pobox.com> Improved-by: Sverre Rabbelier <srabbelier@gmail.com>
David Barr committed
Dec 2, 2010 at 21:40 UTC
8dc6a373d201839859fe7924b63e57526ee2fc22
3 files changed
+303
-14
Documentation/git-fast-import.txt
+60
-3
@@ -196,7 +196,8 @@ especially when a higher level language such as Perl, Python or
196
Ruby is being used.
197
198
fast-import is very strict about its input. Where we say SP below we mean
199
-*exactly* one space. Likewise LF means one (and only one) linefeed.
199
+*exactly* one space. Likewise LF means one (and only one) linefeed
200
+and HT one (and only one) horizontal tab.
201
Supplying additional whitespace characters will cause unexpected
202
results, such as branch names or file names with leading or trailing
203
spaces in their name, or early termination of fast-import when it encounters
@@ -334,6 +335,11 @@ and control the current import process. More detailed discussion
335
format to the file descriptor set with `--cat-blob-fd` or
336
`stdout` if unspecified.
337
338
+`ls`::
339
+ Causes fast-import to print a line describing a directory
340
+ entry in 'ls-tree' format to the file descriptor set with
341
+ `--cat-blob-fd` or `stdout` if unspecified.
342
+
343
`feature`::
344
Require that fast-import supports the specified feature, or
345
abort if it does not.
@@ -919,6 +925,55 @@ This command can be used anywhere in the stream that comments are
925
accepted. In particular, the `cat-blob` command can be used in the
926
middle of a commit but not in the middle of a `data` command.
927
928
+`ls`
929
+~~~~
930
+Prints information about the object at a path to a file descriptor
931
+previously arranged with the `--cat-blob-fd` argument. This allows
932
+printing a blob from the active commit (with `cat-blob`) or copying a
933
+blob or tree from a previous commit for use in the current one (with
934
+`filemodify`).
935
+
936
+The `ls` command can be used anywhere in the stream that comments are
937
+accepted, including the middle of a commit.
938
+
939
+Reading from the active commit::
940
+ This form can only be used in the middle of a `commit`.
941
+ The path names a directory entry within fast-import's
942
+ active commit. The path must be quoted in this case.
943
++
944
+....
945
+ 'ls' SP <path> LF
946
+....
947
+
948
+Reading from a named tree::
949
+ The `<dataref>` can be a mark reference (`:<idnum>`) or the
950
+ full 40-byte SHA-1 of a Git tag, commit, or tree object,
951
+ preexisting or waiting to be written.
952
+ The path is relative to the top level of the tree
953
+ named by `<dataref>`.
954
++
955
+....
956
+ 'ls' SP <dataref> SP <path> LF
957
+....
958
+
959
+See `filemodify` above for a detailed description of `<path>`.
960
+
961
+Output uses the same format as `git ls-tree <tree> {litdd} <path>`:
962
+
963
+====
964
+ <mode> SP ('blob' | 'tree' | 'commit') SP <dataref> HT <path> LF
965
+====
966
+
967
+The <dataref> represents the blob, tree, or commit object at <path>
968
+and can be used in later 'cat-blob', 'filemodify', or 'ls' commands.
969
+
970
+If there is no file or subtree at that path, 'git fast-import' will
971
+instead report
972
+
973
+====
974
+ missing SP <path> LF
975
+====
976
+
977
`feature`
978
~~~~~~~~~
979
Require that fast-import supports the specified feature, or abort if
@@ -946,8 +1001,10 @@ import-marks::
1001
any "feature import-marks" command in the stream.
1002
1003
cat-blob::
949
- Ignored. Versions of fast-import not supporting the
950
- "cat-blob" command will exit with a message indicating so.
1004
+ls::
1005
+ Require that the backend support the 'cat-blob' or 'ls' command.
1006
+ Versions of fast-import not supporting the specified command
1007
+ will exit with a message indicating so.
1008
This lets the import error out early with a clear message,
1009
rather than wasting time on the early part of an import
1010
before the unsupported command is detected.
fast-import.c
+159
-3
@@ -24,10 +24,12 @@ Format of STDIN stream:
24
commit_msg
25
('from' sp committish lf)?
26
('merge' sp committish lf)*
27
- file_change*
27
+ (file_change | ls)*
28
lf?;
29
commit_msg ::= data;
30
31
+ ls ::= 'ls' sp '"' quoted(path) '"' lf;
32
+
33
file_change ::= file_clr
34
| file_del
35
| file_rnm
@@ -132,7 +134,7 @@ Format of STDIN stream:
134
ts ::= # time since the epoch in seconds, ascii base10 notation;
135
tz ::= # GIT style timezone;
136
135
- # note: comments and cat requests may appear anywhere
137
+ # note: comments, ls and cat requests may appear anywhere
138
# in the input, except within a data command. Any form
139
# of the data command always escapes the related input
140
# from comment processing.
@@ -141,7 +143,9 @@ Format of STDIN stream:
143
# must be the first character on that line (an lf
144
# preceded it).
145
#
146
+
147
cat_blob ::= 'cat-blob' sp (hexsha1 | idnum) lf;
148
+ ls_tree ::= 'ls' sp (hexsha1 | idnum) sp path_str lf;
149
150
comment ::= '#' not_lf* lf;
151
not_lf ::= # Any byte that is not ASCII newline (LF);
@@ -374,6 +378,7 @@ static int cat_blob_fd = STDOUT_FILENO;
378
379
static void parse_argv(void);
380
static void parse_cat_blob(void);
381
+static void parse_ls(struct branch *b);
382
383
static void write_branch_report(FILE *rpt, struct branch *b)
384
{
@@ -2614,6 +2619,8 @@ static void parse_new_commit(void)
2619
note_change_n(b, prev_fanout);
2620
else if (!strcmp("deleteall", command_buf.buf))
2621
file_change_deleteall(b);
2622
+ else if (!prefixcmp(command_buf.buf, "ls "))
2623
+ parse_ls(b);
2624
else {
2625
unread_command_buf = 1;
2626
break;
@@ -2837,6 +2844,153 @@ static void parse_cat_blob(void)
2844
cat_blob(oe, sha1);
2845
}
2846
2847
+static struct object_entry *dereference(struct object_entry *oe,
2848
+ unsigned char sha1[20])
2849
+{
2850
+ unsigned long size;
2851
+ void *buf = NULL;
2852
+ if (!oe) {
2853
+ enum object_type type = sha1_object_info(sha1, NULL);
2854
+ if (type < 0)
2855
+ die("object not found: %s", sha1_to_hex(sha1));
2856
+ /* cache it! */
2857
+ oe = insert_object(sha1);
2858
+ oe->type = type;
2859
+ oe->pack_id = MAX_PACK_ID;
2860
+ oe->idx.offset = 1;
2861
+ }
2862
+ switch (oe->type) {
2863
+ case OBJ_TREE: /* easy case. */
2864
+ return oe;
2865
+ case OBJ_COMMIT:
2866
+ case OBJ_TAG:
2867
+ break;
2868
+ default:
2869
+ die("Not a treeish: %s", command_buf.buf);
2870
+ }
2871
+
2872
+ if (oe->pack_id != MAX_PACK_ID) { /* in a pack being written */
2873
+ buf = gfi_unpack_entry(oe, &size);
2874
+ } else {
2875
+ enum object_type unused;
2876
+ buf = read_sha1_file(sha1, &unused, &size);
2877
+ }
2878
+ if (!buf)
2879
+ die("Can't load object %s", sha1_to_hex(sha1));
2880
+
2881
+ /* Peel one layer. */
2882
+ switch (oe->type) {
2883
+ case OBJ_TAG:
2884
+ if (size < 40 + strlen("object ") ||
2885
+ get_sha1_hex(buf + strlen("object "), sha1))
2886
+ die("Invalid SHA1 in tag: %s", command_buf.buf);
2887
+ break;
2888
+ case OBJ_COMMIT:
2889
+ if (size < 40 + strlen("tree ") ||
2890
+ get_sha1_hex(buf + strlen("tree "), sha1))
2891
+ die("Invalid SHA1 in commit: %s", command_buf.buf);
2892
+ }
2893
+
2894
+ free(buf);
2895
+ return find_object(sha1);
2896
+}
2897
+
2898
+static struct object_entry *parse_treeish_dataref(const char **p)
2899
+{
2900
+ unsigned char sha1[20];
2901
+ struct object_entry *e;
2902
+
2903
+ if (**p == ':') { /* <mark> */
2904
+ char *endptr;
2905
+ e = find_mark(strtoumax(*p + 1, &endptr, 10));
2906
+ if (endptr == *p + 1)
2907
+ die("Invalid mark: %s", command_buf.buf);
2908
+ if (!e)
2909
+ die("Unknown mark: %s", command_buf.buf);
2910
+ *p = endptr;
2911
+ hashcpy(sha1, e->idx.sha1);
2912
+ } else { /* <sha1> */
2913
+ if (get_sha1_hex(*p, sha1))
2914
+ die("Invalid SHA1: %s", command_buf.buf);
2915
+ e = find_object(sha1);
2916
+ *p += 40;
2917
+ }
2918
+
2919
+ while (!e || e->type != OBJ_TREE)
2920
+ e = dereference(e, sha1);
2921
+ return e;
2922
+}
2923
+
2924
+static void print_ls(int mode, const unsigned char *sha1, const char *path)
2925
+{
2926
+ static struct strbuf line = STRBUF_INIT;
2927
+
2928
+ /* See show_tree(). */
2929
+ const char *type =
2930
+ S_ISGITLINK(mode) ? commit_type :
2931
+ S_ISDIR(mode) ? tree_type :
2932
+ blob_type;
2933
+
2934
+ if (!mode) {
2935
+ /* missing SP path LF */
2936
+ strbuf_reset(&line);
2937
+ strbuf_addstr(&line, "missing ");
2938
+ quote_c_style(path, &line, NULL, 0);
2939
+ strbuf_addch(&line, '\n');
2940
+ } else {
2941
+ /* mode SP type SP object_name TAB path LF */
2942
+ strbuf_reset(&line);
2943
+ strbuf_addf(&line, "%06o %s %s\t",
2944
+ mode, type, sha1_to_hex(sha1));
2945
+ quote_c_style(path, &line, NULL, 0);
2946
+ strbuf_addch(&line, '\n');
2947
+ }
2948
+ cat_blob_write(line.buf, line.len);
2949
+}
2950
+
2951
+static void parse_ls(struct branch *b)
2952
+{
2953
+ const char *p;
2954
+ struct tree_entry *root = NULL;
2955
+ struct tree_entry leaf = {0};
2956
+
2957
+ /* ls SP (<treeish> SP)? <path> */
2958
+ p = command_buf.buf + strlen("ls ");
2959
+ if (*p == '"') {
2960
+ if (!b)
2961
+ die("Not in a commit: %s", command_buf.buf);
2962
+ root = &b->branch_tree;
2963
+ } else {
2964
+ struct object_entry *e = parse_treeish_dataref(&p);
2965
+ root = new_tree_entry();
2966
+ hashcpy(root->versions[1].sha1, e->idx.sha1);
2967
+ load_tree(root);
2968
+ if (*p++ != ' ')
2969
+ die("Missing space after tree-ish: %s", command_buf.buf);
2970
+ }
2971
+ if (*p == '"') {
2972
+ static struct strbuf uq = STRBUF_INIT;
2973
+ const char *endp;
2974
+ strbuf_reset(&uq);
2975
+ if (unquote_c_style(&uq, p, &endp))
2976
+ die("Invalid path: %s", command_buf.buf);
2977
+ if (*endp)
2978
+ die("Garbage after path in: %s", command_buf.buf);
2979
+ p = uq.buf;
2980
+ }
2981
+ tree_content_get(root, p, &leaf);
2982
+ /*
2983
+ * A directory in preparation would have a sha1 of zero
2984
+ * until it is saved. Save, for simplicity.
2985
+ */
2986
+ if (S_ISDIR(leaf.versions[1].mode))
2987
+ store_tree(&leaf);
2988
+
2989
+ print_ls(leaf.versions[1].mode, leaf.versions[1].sha1, p);
2990
+ if (!b || root != &b->branch_tree)
2991
+ release_tree_entry(root);
2992
+}
2993
+
2994
static void checkpoint(void)
2995
{
2996
checkpoint_requested = 0;
@@ -3001,7 +3155,7 @@ static int parse_one_feature(const char *feature, int from_stream)
3155
relative_marks_paths = 0;
3156
} else if (!prefixcmp(feature, "force")) {
3157
force_update = 1;
3004
- } else if (!strcmp(feature, "notes")) {
3158
+ } else if (!strcmp(feature, "notes") || !strcmp(feature, "ls")) {
3159
; /* do nothing; we have the feature */
3160
} else {
3161
return 0;
@@ -3142,6 +3296,8 @@ int main(int argc, const char **argv)
3296
while (read_next_command() != EOF) {
3297
if (!strcmp("blob", command_buf.buf))
3298
parse_new_blob();
3299
+ else if (!prefixcmp(command_buf.buf, "ls "))
3300
+ parse_ls(NULL);
3301
else if (!prefixcmp(command_buf.buf, "commit "))
3302
parse_new_commit();
3303
else if (!prefixcmp(command_buf.buf, "tag "))
t/t9300-fast-import.sh
+84
-8
@@ -42,6 +42,14 @@ echo "$@"'
42
43
>empty
44
45
+test_expect_success 'setup: have pipes?' '
46
+ rm -f frob &&
47
+ if mkfifo frob
48
+ then
49
+ test_set_prereq PIPE
50
+ fi
51
+'
52
+
53
###
54
### series A
55
###
@@ -898,6 +906,77 @@ test_expect_success \
906
git diff-tree -C --find-copies-harder -r N4^ N4 >actual &&
907
compare_diff_raw expect actual'
908
909
+test_expect_success PIPE 'N: read and copy directory' '
910
+ cat >expect <<-\EOF
911
+ :100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100 file2/newf file3/newf
912
+ :100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100 file2/oldf file3/oldf
913
+ EOF
914
+ git update-ref -d refs/heads/N4 &&
915
+ rm -f backflow &&
916
+ mkfifo backflow &&
917
+ (
918
+ exec <backflow &&
919
+ cat <<-EOF &&
920
+ commit refs/heads/N4
921
+ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
922
+ data <<COMMIT
923
+ copy by tree hash, part 2
924
+ COMMIT
925
+
926
+ from refs/heads/branch^0
927
+ ls "file2"
928
+ EOF
929
+ read mode type tree filename &&
930
+ echo "M 040000 $tree file3"
931
+ ) |
932
+ git fast-import --cat-blob-fd=3 3>backflow &&
933
+ git diff-tree -C --find-copies-harder -r N4^ N4 >actual &&
934
+ compare_diff_raw expect actual
935
+'
936
+
937
+test_expect_success PIPE 'N: empty directory reads as missing' '
938
+ cat <<-\EOF >expect &&
939
+ OBJNAME
940
+ :000000 100644 OBJNAME OBJNAME A unrelated
941
+ EOF
942
+ echo "missing src" >expect.response &&
943
+ git update-ref -d refs/heads/read-empty &&
944
+ rm -f backflow &&
945
+ mkfifo backflow &&
946
+ (
947
+ exec <backflow &&
948
+ cat <<-EOF &&
949
+ commit refs/heads/read-empty
950
+ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
951
+ data <<COMMIT
952
+ read "empty" (missing) directory
953
+ COMMIT
954
+
955
+ M 100644 inline src/greeting
956
+ data <<BLOB
957
+ hello
958
+ BLOB
959
+ C src/greeting dst1/non-greeting
960
+ C src/greeting unrelated
961
+ # leave behind "empty" src directory
962
+ D src/greeting
963
+ ls "src"
964
+ EOF
965
+ read -r line &&
966
+ printf "%s\n" "$line" >response &&
967
+ cat <<-\EOF
968
+ D dst1
969
+ D dst2
970
+ EOF
971
+ ) |
972
+ git fast-import --cat-blob-fd=3 3>backflow &&
973
+ test_cmp expect.response response &&
974
+ git rev-list read-empty |
975
+ git diff-tree -r --root --stdin |
976
+ sed "s/$_x40/OBJNAME/g" >actual &&
977
+ test_cmp expect actual
978
+'
979
+
980
test_expect_success \
981
'N: copy root directory by tree hash' \
982
'cat >expect <<-\EOF &&
@@ -1861,6 +1940,11 @@ test_expect_success 'R: feature no-relative-marks should be honoured' '
1940
test_cmp marks.new non-relative.out
1941
'
1942
1943
+test_expect_success 'R: feature ls supported' '
1944
+ echo "feature ls" |
1945
+ git fast-import
1946
+'
1947
+
1948
test_expect_success 'R: feature cat-blob supported' '
1949
echo "feature cat-blob" |
1950
git fast-import
@@ -1986,14 +2070,6 @@ test_expect_success 'R: print two blobs to stdout' '
2070
test_cmp expect actual
2071
'
2072
1989
-test_expect_success 'setup: have pipes?' '
1990
- rm -f frob &&
1991
- if mkfifo frob
1992
- then
1993
- test_set_prereq PIPE
1994
- fi
1995
-'
1996
-
2073
test_expect_success PIPE 'R: copy using cat-file' '
2074
expect_id=$(git hash-object big) &&
2075
expect_len=$(wc -c <big) &&