@matheusbsbs / matheus-bagliano / commits / ac053c0202
Fork 0

Allow frontends to bidirectionally communicate with fast-import

The existing checkpoint command is very useful to force fast-import to dump the branches out to disk so that standard Git tools can access them and the objects they refer to. However there was not a way to know when fast-import had finished executing the checkpoint and it was safe to read those refs. The progress command can be used to make fast-import output any message of the frontend's choosing to standard out. The frontend can scan for these messages using select() or poll() to monitor a pipe connected to the standard output of fast-import. Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Shawn O. Pearce committed Aug 1, 2007 at 10:23 UTC ac053c02029d88c7ed4d7e92949a1586eb3f7704
3 files changed +84
Documentation/git-fast-import.txt
+39
@@ -298,6 +298,11 @@ and control the current import process. More detailed discussion
298 This command is optional and is not needed to perform
299 an import.
300
301 +`progress`::
302 + Causes fast-import to echo the entire line to its own
303 + standard output. This command is optional and is not needed
304 + to perform an import.
305 +
306 `commit`
307 ~~~~~~~~
308 Create or update a branch with a new commit, recording one logical
@@ -775,6 +780,31 @@ explicit checkpointing may not be necessary.
780
781 The `LF` after the command is optional (it used to be required).
782
783 +`progress`
784 +~~~~~~~~~~
785 +Causes fast-import to print the entire `progress` line unmodified to
786 +its standard output channel (file descriptor 1) when the command is
787 +processed from the input stream. The command otherwise has no impact
788 +on the current import, or on any of fast-import's internal state.
789 +
790 +....
791 + 'progress' SP <any> LF
792 + LF?
793 +....
794 +
795 +The `<any>` part of the command may contain any sequence of bytes
796 +that does not contain `LF`. The `LF` after the command is optional.
797 +Callers may wish to process the output through a tool such as sed to
798 +remove the leading part of the line, for example:
799 +
800 +====
801 + frontend | git-fast-import | sed 's/^progress //'
802 +====
803 +
804 +Placing a `progress` command immediately after a `checkpoint` will
805 +inform the reader when the `checkpoint` has been completed and it
806 +can safely access the refs that fast-import updated.
807 +
808 Tips and Tricks
809 ---------------
810 The following tips and tricks have been collected from various
@@ -867,6 +897,15 @@ This will take longer, but will also produce a smaller packfile.
897 You only need to expend the effort once, and everyone using your
898 project will benefit from the smaller repository.
899
900 +Include Some Progress Messages
901 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
902 +Every once in a while have your frontend emit a `progress` message
903 +to fast-import. The contents of the messages are entirely free-form,
904 +so one suggestion would be to output the current month and year
905 +each time the current commit date moves into the next month.
906 +Your users will feel better knowing how much of the data stream
907 +has been processed.
908 +
909
910 Packfile Optimization
911 ---------------------
fast-import.c
+14
@@ -8,6 +8,7 @@ Format of STDIN stream:
8 | new_tag
9 | reset_branch
10 | checkpoint
11 + | progress
12 ;
13
14 new_blob ::= 'blob' lf
@@ -53,6 +54,9 @@ Format of STDIN stream:
54 checkpoint ::= 'checkpoint' lf
55 lf?;
56
57 + progress ::= 'progress' sp not_lf* lf
58 + lf?;
59 +
60 # note: the first idnum in a stream should be 1 and subsequent
61 # idnums should not have gaps between values as this will cause
62 # the stream parser to reserve space for the gapped values. An
@@ -2125,6 +2129,14 @@ static void cmd_checkpoint(void)
2129 skip_optional_lf();
2130 }
2131
2132 +static void cmd_progress(void)
2133 +{
2134 + fwrite(command_buf.buf, 1, command_buf.len - 1, stdout);
2135 + fputc('\n', stdout);
2136 + fflush(stdout);
2137 + skip_optional_lf();
2138 +}
2139 +
2140 static void import_marks(const char *input_file)
2141 {
2142 char line[512];
@@ -2235,6 +2247,8 @@ int main(int argc, const char **argv)
2247 cmd_reset_branch();
2248 else if (!strcmp("checkpoint", command_buf.buf))
2249 cmd_checkpoint();
2250 + else if (!prefixcmp(command_buf.buf, "progress "))
2251 + cmd_progress();
2252 else
2253 die("Unsupported command: %s", command_buf.buf);
2254 }
t/t9300-fast-import.sh
+31
@@ -885,4 +885,35 @@ test_expect_success \
885 git log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&
886 git diff expect actual'
887
888 +cat >input <<INPUT_END
889 +commit refs/heads/O4
890 +committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
891 +data <<COMMIT
892 +zstring
893 +COMMIT
894 +commit refs/heads/O4
895 +committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
896 +data <<COMMIT
897 +zof
898 +COMMIT
899 +progress Two commits down, 2 to go!
900 +commit refs/heads/O4
901 +committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
902 +data <<COMMIT
903 +zempty
904 +COMMIT
905 +progress Three commits down, 1 to go!
906 +commit refs/heads/O4
907 +committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
908 +data <<COMMIT
909 +zcommits
910 +COMMIT
911 +progress I'm done!
912 +INPUT_END
913 +test_expect_success \
914 + 'O: progress outputs as requested by input' \
915 + 'git-fast-import <input >actual &&
916 + grep "progress " <input >expect &&
917 + git diff expect actual'
918 +
919 test_done