Allow frontends to bidirectionally communicate with fast-import
The existing checkpoint command is very useful to force fast-import to dump the branches out to disk so that standard Git tools can access them and the objects they refer to. However there was not a way to know when fast-import had finished executing the checkpoint and it was safe to read those refs. The progress command can be used to make fast-import output any message of the frontend's choosing to standard out. The frontend can scan for these messages using select() or poll() to monitor a pipe connected to the standard output of fast-import. Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce committed
Aug 1, 2007 at 10:23 UTC
ac053c02029d88c7ed4d7e92949a1586eb3f7704
3 files changed
+84
Documentation/git-fast-import.txt
+39
@@ -298,6 +298,11 @@ and control the current import process. More detailed discussion
298
This command is optional and is not needed to perform
299
an import.
300
301
+`progress`::
302
+ Causes fast-import to echo the entire line to its own
303
+ standard output. This command is optional and is not needed
304
+ to perform an import.
305
+
306
`commit`
307
~~~~~~~~
308
Create or update a branch with a new commit, recording one logical
@@ -775,6 +780,31 @@ explicit checkpointing may not be necessary.
780
781
The `LF` after the command is optional (it used to be required).
782
783
+`progress`
784
+~~~~~~~~~~
785
+Causes fast-import to print the entire `progress` line unmodified to
786
+its standard output channel (file descriptor 1) when the command is
787
+processed from the input stream. The command otherwise has no impact
788
+on the current import, or on any of fast-import's internal state.
789
+
790
+....
791
+ 'progress' SP <any> LF
792
+ LF?
793
+....
794
+
795
+The `<any>` part of the command may contain any sequence of bytes
796
+that does not contain `LF`. The `LF` after the command is optional.
797
+Callers may wish to process the output through a tool such as sed to
798
+remove the leading part of the line, for example:
799
+
800
+====
801
+ frontend | git-fast-import | sed 's/^progress //'
802
+====
803
+
804
+Placing a `progress` command immediately after a `checkpoint` will
805
+inform the reader when the `checkpoint` has been completed and it
806
+can safely access the refs that fast-import updated.
807
+
808
Tips and Tricks
809
---------------
810
The following tips and tricks have been collected from various
@@ -867,6 +897,15 @@ This will take longer, but will also produce a smaller packfile.
897
You only need to expend the effort once, and everyone using your
898
project will benefit from the smaller repository.
899
900
+Include Some Progress Messages
901
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
902
+Every once in a while have your frontend emit a `progress` message
903
+to fast-import. The contents of the messages are entirely free-form,
904
+so one suggestion would be to output the current month and year
905
+each time the current commit date moves into the next month.
906
+Your users will feel better knowing how much of the data stream
907
+has been processed.
908
+
909
910
Packfile Optimization
911
---------------------
fast-import.c
+14
@@ -8,6 +8,7 @@ Format of STDIN stream:
8
| new_tag
9
| reset_branch
10
| checkpoint
11
+ | progress
12
;
13
14
new_blob ::= 'blob' lf
@@ -53,6 +54,9 @@ Format of STDIN stream:
54
checkpoint ::= 'checkpoint' lf
55
lf?;
56
57
+ progress ::= 'progress' sp not_lf* lf
58
+ lf?;
59
+
60
# note: the first idnum in a stream should be 1 and subsequent
61
# idnums should not have gaps between values as this will cause
62
# the stream parser to reserve space for the gapped values. An
@@ -2125,6 +2129,14 @@ static void cmd_checkpoint(void)
2129
skip_optional_lf();
2130
}
2131
2132
+static void cmd_progress(void)
2133
+{
2134
+ fwrite(command_buf.buf, 1, command_buf.len - 1, stdout);
2135
+ fputc('\n', stdout);
2136
+ fflush(stdout);
2137
+ skip_optional_lf();
2138
+}
2139
+
2140
static void import_marks(const char *input_file)
2141
{
2142
char line[512];
@@ -2235,6 +2247,8 @@ int main(int argc, const char **argv)
2247
cmd_reset_branch();
2248
else if (!strcmp("checkpoint", command_buf.buf))
2249
cmd_checkpoint();
2250
+ else if (!prefixcmp(command_buf.buf, "progress "))
2251
+ cmd_progress();
2252
else
2253
die("Unsupported command: %s", command_buf.buf);
2254
}
t/t9300-fast-import.sh
+31
@@ -885,4 +885,35 @@ test_expect_success \
885
git log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&
886
git diff expect actual'
887
888
+cat >input <<INPUT_END
889
+commit refs/heads/O4
890
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
891
+data <<COMMIT
892
+zstring
893
+COMMIT
894
+commit refs/heads/O4
895
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
896
+data <<COMMIT
897
+zof
898
+COMMIT
899
+progress Two commits down, 2 to go!
900
+commit refs/heads/O4
901
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
902
+data <<COMMIT
903
+zempty
904
+COMMIT
905
+progress Three commits down, 1 to go!
906
+commit refs/heads/O4
907
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
908
+data <<COMMIT
909
+zcommits
910
+COMMIT
911
+progress I'm done!
912
+INPUT_END
913
+test_expect_success \
914
+ 'O: progress outputs as requested by input' \
915
+ 'git-fast-import <input >actual &&
916
+ grep "progress " <input >expect &&
917
+ git diff expect actual'
918
+
919
test_done