@matheusbsbs / matheus-bagliano / commits / 2841e8f81c

convert: add "status=delayed" to filter process protocol

Some `clean` / `smudge` filters may require a significant amount of time to process a single blob (e.g. the Git LFS smudge filter might perform network requests). During this process the Git checkout operation is blocked and Git needs to wait until the filter is done to continue with the checkout. Teach the filter process protocol, introduced in edcc8581 ("convert: add filter.<driver>.process option", 2016-10-16), to accept the status "delayed" as response to a filter request. Upon this response Git continues with the checkout operation. After the checkout operation Git calls "finish_delayed_checkout" which queries the filter for remaining blobs. If the filter is still working on the completion, then the filter is expected to block. If the filter has completed all remaining blobs then an empty response is expected. Git has a multiple code paths that checkout a blob. Support delayed checkouts only in `clone` (in unpack-trees.c) and `checkout` operations for now. The optimization is most effective in these code paths as all files of the tree are processed. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Lars Schneider committed Jun 30, 2017 at 22:41 UTC 2841e8f81cb2820024804b9341577be1d0ce1240
9 files changed +575 -90
Documentation/gitattributes.txt
+65 -4
@@ -425,8 +425,8 @@ packet: git< capability=clean
425 packet: git< capability=smudge
426 packet: git< 0000
427 ------------------------
428 -Supported filter capabilities in version 2 are "clean" and
429 -"smudge".
428 +Supported filter capabilities in version 2 are "clean", "smudge",
429 +and "delay".
430
431 Afterwards Git sends a list of "key=value" pairs terminated with
432 a flush packet. The list will contain at least the filter command
@@ -512,12 +512,73 @@ the protocol then Git will stop the filter process and restart it
512 with the next file that needs to be processed. Depending on the
513 `filter.<driver>.required` flag Git will interpret that as error.
514
515 -After the filter has processed a blob it is expected to wait for
516 -the next "key=value" list containing a command. Git will close
515 +After the filter has processed a command it is expected to wait for
516 +a "key=value" list containing the next command. Git will close
517 the command pipe on exit. The filter is expected to detect EOF
518 and exit gracefully on its own. Git will wait until the filter
519 process has stopped.
520
521 +Delay
522 +^^^^^
523 +
524 +If the filter supports the "delay" capability, then Git can send the
525 +flag "can-delay" after the filter command and pathname. This flag
526 +denotes that the filter can delay filtering the current blob (e.g. to
527 +compensate network latencies) by responding with no content but with
528 +the status "delayed" and a flush packet.
529 +------------------------
530 +packet: git> command=smudge
531 +packet: git> pathname=path/testfile.dat
532 +packet: git> can-delay=1
533 +packet: git> 0000
534 +packet: git> CONTENT
535 +packet: git> 0000
536 +packet: git< status=delayed
537 +packet: git< 0000
538 +------------------------
539 +
540 +If the filter supports the "delay" capability then it must support the
541 +"list_available_blobs" command. If Git sends this command, then the
542 +filter is expected to return a list of pathnames representing blobs
543 +that have been delayed earlier and are now available.
544 +The list must be terminated with a flush packet followed
545 +by a "success" status that is also terminated with a flush packet. If
546 +no blobs for the delayed paths are available, yet, then the filter is
547 +expected to block the response until at least one blob becomes
548 +available. The filter can tell Git that it has no more delayed blobs
549 +by sending an empty list. As soon as the filter responds with an empty
550 +list, Git stops asking. All blobs that Git has not received at this
551 +point are considered missing and will result in an error.
552 +
553 +------------------------
554 +packet: git> command=list_available_blobs
555 +packet: git> 0000
556 +packet: git< pathname=path/testfile.dat
557 +packet: git< pathname=path/otherfile.dat
558 +packet: git< 0000
559 +packet: git< status=success
560 +packet: git< 0000
561 +------------------------
562 +
563 +After Git received the pathnames, it will request the corresponding
564 +blobs again. These requests contain a pathname and an empty content
565 +section. The filter is expected to respond with the smudged content
566 +in the usual way as explained above.
567 +------------------------
568 +packet: git> command=smudge
569 +packet: git> pathname=path/testfile.dat
570 +packet: git> 0000
571 +packet: git> 0000 # empty content!
572 +packet: git< status=success
573 +packet: git< 0000
574 +packet: git< SMUDGED_CONTENT
575 +packet: git< 0000
576 +packet: git< 0000 # empty list, keep "status=success" unchanged!
577 +------------------------
578 +
579 +Example
580 +^^^^^^^
581 +
582 A long running filter demo implementation can be found in
583 `contrib/long-running-filter/example.pl` located in the Git
584 core repository. If you develop your own long running filter
builtin/checkout.c
+3
@@ -376,6 +376,8 @@ static int checkout_paths(const struct checkout_opts *opts,
376 state.force = 1;
377 state.refresh_cache = 1;
378 state.istate = &the_index;
379 +
380 + enable_delayed_checkout(&state);
381 for (pos = 0; pos < active_nr; pos++) {
382 struct cache_entry *ce = active_cache[pos];
383 if (ce->ce_flags & CE_MATCHED) {
@@ -390,6 +392,7 @@ static int checkout_paths(const struct checkout_opts *opts,
392 pos = skip_same_name(ce, pos) - 1;
393 }
394 }
395 + errs |= finish_delayed_checkout(&state);
396
397 if (write_locked_index(&the_index, lock_file, COMMIT_LOCK))
398 die(_("unable to write new index file"));
cache.h
+3
@@ -1544,6 +1544,7 @@ struct checkout {
1544 struct index_state *istate;
1545 const char *base_dir;
1546 int base_dir_len;
1547 + struct delayed_checkout *delayed_checkout;
1548 unsigned force:1,
1549 quiet:1,
1550 not_new:1,
@@ -1553,6 +1554,8 @@ struct checkout {
1554
1555 #define TEMPORARY_FILENAME_LENGTH 25
1556 extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);
1557 +extern void enable_delayed_checkout(struct checkout *state);
1558 +extern int finish_delayed_checkout(struct checkout *state);
1559
1560 struct cache_def {
1561 struct strbuf path;
convert.c
+94 -16
@@ -496,6 +496,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le
496
497 #define CAP_CLEAN (1u<<0)
498 #define CAP_SMUDGE (1u<<1)
499 +#define CAP_DELAY (1u<<2)
500
501 struct cmd2process {
502 struct subprocess_entry subprocess; /* must be the first member! */
@@ -521,6 +522,7 @@ static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)
522 } known_caps[] = {
523 { "clean", CAP_CLEAN },
524 { "smudge", CAP_SMUDGE },
525 + { "delay", CAP_DELAY },
526 };
527
528 sigchain_push(SIGPIPE, SIG_IGN);
@@ -605,9 +607,11 @@ static void handle_filter_error(const struct strbuf *filter_status,
607
608 static int apply_multi_file_filter(const char *path, const char *src, size_t len,
609 int fd, struct strbuf *dst, const char *cmd,
608 - const unsigned int wanted_capability)
610 + const unsigned int wanted_capability,
611 + struct delayed_checkout *dco)
612 {
613 int err;
614 + int can_delay = 0;
615 struct cmd2process *entry;
616 struct child_process *process;
617 struct strbuf nbuf = STRBUF_INIT;
@@ -662,6 +666,14 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len
666 if (err)
667 goto done;
668
669 + if ((entry->supported_capabilities & CAP_DELAY) &&
670 + dco && dco->state == CE_CAN_DELAY) {
671 + can_delay = 1;
672 + err = packet_write_fmt_gently(process->in, "can-delay=1\n");
673 + if (err)
674 + goto done;
675 + }
676 +
677 err = packet_flush_gently(process->in);
678 if (err)
679 goto done;
@@ -677,14 +689,73 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len
689 if (err)
690 goto done;
691
680 - err = strcmp(filter_status.buf, "success");
692 + if (can_delay && !strcmp(filter_status.buf, "delayed")) {
693 + string_list_insert(&dco->filters, cmd);
694 + string_list_insert(&dco->paths, path);
695 + } else {
696 + /* The filter got the blob and wants to send us a response. */
697 + err = strcmp(filter_status.buf, "success");
698 + if (err)
699 + goto done;
700 +
701 + err = read_packetized_to_strbuf(process->out, &nbuf) < 0;
702 + if (err)
703 + goto done;
704 +
705 + err = subprocess_read_status(process->out, &filter_status);
706 + if (err)
707 + goto done;
708 +
709 + err = strcmp(filter_status.buf, "success");
710 + }
711 +
712 +done:
713 + sigchain_pop(SIGPIPE);
714 +
715 + if (err)
716 + handle_filter_error(&filter_status, entry, wanted_capability);
717 + else
718 + strbuf_swap(dst, &nbuf);
719 + strbuf_release(&nbuf);
720 + return !err;
721 +}
722 +
723 +
724 +int async_query_available_blobs(const char *cmd, struct string_list *available_paths)
725 +{
726 + int err;
727 + char *line;
728 + struct cmd2process *entry;
729 + struct child_process *process;
730 + struct strbuf filter_status = STRBUF_INIT;
731 +
732 + assert(subprocess_map_initialized);
733 + entry = (struct cmd2process *)subprocess_find_entry(&subprocess_map, cmd);
734 + if (!entry) {
735 + error("external filter '%s' is not available anymore although "
736 + "not all paths have been filtered", cmd);
737 + return 0;
738 + }
739 + process = &entry->subprocess.process;
740 + sigchain_push(SIGPIPE, SIG_IGN);
741 +
742 + err = packet_write_fmt_gently(
743 + process->in, "command=list_available_blobs\n");
744 if (err)
745 goto done;
746
684 - err = read_packetized_to_strbuf(process->out, &nbuf) < 0;
747 + err = packet_flush_gently(process->in);
748 if (err)
749 goto done;
750
751 + while ((line = packet_read_line(process->out, NULL))) {
752 + const char *path;
753 + if (skip_prefix(line, "pathname=", &path))
754 + string_list_insert(available_paths, xstrdup(path));
755 + else
756 + ; /* ignore unknown keys */
757 + }
758 +
759 err = subprocess_read_status(process->out, &filter_status);
760 if (err)
761 goto done;
@@ -695,10 +766,7 @@ done:
766 sigchain_pop(SIGPIPE);
767
768 if (err)
698 - handle_filter_error(&filter_status, entry, wanted_capability);
699 - else
700 - strbuf_swap(dst, &nbuf);
701 - strbuf_release(&nbuf);
769 + handle_filter_error(&filter_status, entry, 0);
770 return !err;
771 }
772
@@ -713,7 +781,8 @@ static struct convert_driver {
781
782 static int apply_filter(const char *path, const char *src, size_t len,
783 int fd, struct strbuf *dst, struct convert_driver *drv,
716 - const unsigned int wanted_capability)
784 + const unsigned int wanted_capability,
785 + struct delayed_checkout *dco)
786 {
787 const char *cmd = NULL;
788
@@ -731,7 +800,8 @@ static int apply_filter(const char *path, const char *src, size_t len,
800 if (cmd && *cmd)
801 return apply_single_file_filter(path, src, len, fd, dst, cmd);
802 else if (drv->process && *drv->process)
734 - return apply_multi_file_filter(path, src, len, fd, dst, drv->process, wanted_capability);
803 + return apply_multi_file_filter(path, src, len, fd, dst,
804 + drv->process, wanted_capability, dco);
805
806 return 0;
807 }
@@ -1072,7 +1142,7 @@ int would_convert_to_git_filter_fd(const char *path)
1142 if (!ca.drv->required)
1143 return 0;
1144
1075 - return apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN);
1145 + return apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL);
1146 }
1147
1148 const char *get_convert_attr_ascii(const char *path)
@@ -1109,7 +1179,7 @@ int convert_to_git(const char *path, const char *src, size_t len,
1179
1180 convert_attrs(&ca, path);
1181
1112 - ret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN);
1182 + ret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN, NULL);
1183 if (!ret && ca.drv && ca.drv->required)
1184 die("%s: clean filter '%s' failed", path, ca.drv->name);
1185
@@ -1134,7 +1204,7 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,
1204 assert(ca.drv);
1205 assert(ca.drv->clean || ca.drv->process);
1206
1137 - if (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN))
1207 + if (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))
1208 die("%s: clean filter '%s' failed", path, ca.drv->name);
1209
1210 crlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);
@@ -1143,7 +1213,7 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,
1213
1214 static int convert_to_working_tree_internal(const char *path, const char *src,
1215 size_t len, struct strbuf *dst,
1146 - int normalizing)
1216 + int normalizing, struct delayed_checkout *dco)
1217 {
1218 int ret = 0, ret_filter = 0;
1219 struct conv_attrs ca;
@@ -1168,21 +1238,29 @@ static int convert_to_working_tree_internal(const char *path, const char *src,
1238 }
1239 }
1240
1171 - ret_filter = apply_filter(path, src, len, -1, dst, ca.drv, CAP_SMUDGE);
1241 + ret_filter = apply_filter(
1242 + path, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);
1243 if (!ret_filter && ca.drv && ca.drv->required)
1244 die("%s: smudge filter %s failed", path, ca.drv->name);
1245
1246 return ret | ret_filter;
1247 }
1248
1249 +int async_convert_to_working_tree(const char *path, const char *src,
1250 + size_t len, struct strbuf *dst,
1251 + void *dco)
1252 +{
1253 + return convert_to_working_tree_internal(path, src, len, dst, 0, dco);
1254 +}
1255 +
1256 int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)
1257 {
1180 - return convert_to_working_tree_internal(path, src, len, dst, 0);
1258 + return convert_to_working_tree_internal(path, src, len, dst, 0, NULL);
1259 }
1260
1261 int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst)
1262 {
1185 - int ret = convert_to_working_tree_internal(path, src, len, dst, 1);
1263 + int ret = convert_to_working_tree_internal(path, src, len, dst, 1, NULL);
1264 if (ret) {
1265 src = dst->buf;
1266 len = dst->len;
convert.h
+26
@@ -4,6 +4,8 @@
4 #ifndef CONVERT_H
5 #define CONVERT_H
6
7 +#include "string-list.h"
8 +
9 enum safe_crlf {
10 SAFE_CRLF_FALSE = 0,
11 SAFE_CRLF_FAIL = 1,
@@ -32,6 +34,26 @@ enum eol {
34 #endif
35 };
36
37 +enum ce_delay_state {
38 + CE_NO_DELAY = 0,
39 + CE_CAN_DELAY = 1,
40 + CE_RETRY = 2
41 +};
42 +
43 +struct delayed_checkout {
44 + /*
45 + * State of the currently processed cache entry. If the state is
46 + * CE_CAN_DELAY, then the filter can delay the current cache entry.
47 + * If the state is CE_RETRY, then this signals the filter that the
48 + * cache entry was requested before.
49 + */
50 + enum ce_delay_state state;
51 + /* List of filter drivers that signaled delayed blobs. */
52 + struct string_list filters;
53 + /* List of delayed blobs identified by their path. */
54 + struct string_list paths;
55 +};
56 +
57 extern enum eol core_eol;
58 extern const char *get_cached_convert_stats_ascii(const char *path);
59 extern const char *get_wt_convert_stats_ascii(const char *path);
@@ -42,6 +64,10 @@ extern int convert_to_git(const char *path, const char *src, size_t len,
64 struct strbuf *dst, enum safe_crlf checksafe);
65 extern int convert_to_working_tree(const char *path, const char *src,
66 size_t len, struct strbuf *dst);
67 +extern int async_convert_to_working_tree(const char *path, const char *src,
68 + size_t len, struct strbuf *dst,
69 + void *dco);
70 +extern int async_query_available_blobs(const char *cmd, struct string_list *available_paths);
71 extern int renormalize_buffer(const char *path, const char *src, size_t len,
72 struct strbuf *dst);
73 static inline int would_convert_to_git(const char *path)
entry.c
+127 -5
@@ -137,6 +137,105 @@ static int streaming_write_entry(const struct cache_entry *ce, char *path,
137 return result;
138 }
139
140 +void enable_delayed_checkout(struct checkout *state)
141 +{
142 + if (!state->delayed_checkout) {
143 + state->delayed_checkout = xmalloc(sizeof(*state->delayed_checkout));
144 + state->delayed_checkout->state = CE_CAN_DELAY;
145 + string_list_init(&state->delayed_checkout->filters, 0);
146 + string_list_init(&state->delayed_checkout->paths, 0);
147 + }
148 +}
149 +
150 +static int remove_available_paths(struct string_list_item *item, void *cb_data)
151 +{
152 + struct string_list *available_paths = cb_data;
153 + struct string_list_item *available;
154 +
155 + available = string_list_lookup(available_paths, item->string);
156 + if (available)
157 + available->util = (void *)item->string;
158 + return !available;
159 +}
160 +
161 +int finish_delayed_checkout(struct checkout *state)
162 +{
163 + int errs = 0;
164 + struct string_list_item *filter, *path;
165 + struct delayed_checkout *dco = state->delayed_checkout;
166 +
167 + if (!state->delayed_checkout)
168 + return errs;
169 +
170 + dco->state = CE_RETRY;
171 + while (dco->filters.nr > 0) {
172 + for_each_string_list_item(filter, &dco->filters) {
173 + struct string_list available_paths = STRING_LIST_INIT_NODUP;
174 +
175 + if (!async_query_available_blobs(filter->string, &available_paths)) {
176 + /* Filter reported an error */
177 + errs = 1;
178 + filter->string = "";
179 + continue;
180 + }
181 + if (available_paths.nr <= 0) {
182 + /*
183 + * Filter responded with no entries. That means
184 + * the filter is done and we can remove the
185 + * filter from the list (see
186 + * "string_list_remove_empty_items" call below).
187 + */
188 + filter->string = "";
189 + continue;
190 + }
191 +
192 + /*
193 + * In dco->paths we store a list of all delayed paths.
194 + * The filter just send us a list of available paths.
195 + * Remove them from the list.
196 + */
197 + filter_string_list(&dco->paths, 0,
198 + &remove_available_paths, &available_paths);
199 +
200 + for_each_string_list_item(path, &available_paths) {
201 + struct cache_entry* ce;
202 +
203 + if (!path->util) {
204 + error("external filter '%s' signaled that '%s' "
205 + "is now available although it has not been "
206 + "delayed earlier",
207 + filter->string, path->string);
208 + errs |= 1;
209 +
210 + /*
211 + * Do not ask the filter for available blobs,
212 + * again, as the filter is likely buggy.
213 + */
214 + filter->string = "";
215 + continue;
216 + }
217 + ce = index_file_exists(state->istate, path->string,
218 + strlen(path->string), 0);
219 + errs |= (ce ? checkout_entry(ce, state, NULL) : 1);
220 + }
221 + }
222 + string_list_remove_empty_items(&dco->filters, 0);
223 + }
224 + string_list_clear(&dco->filters, 0);
225 +
226 + /* At this point we should not have any delayed paths anymore. */
227 + errs |= dco->paths.nr;
228 + for_each_string_list_item(path, &dco->paths) {
229 + error("'%s' was not filtered properly", path->string);
230 + }
231 + string_list_clear(&dco->paths, 0);
232 +
233 + free(dco);
234 + state->delayed_checkout = NULL;
235 +
236 + return errs;
237 +}
238 +
239 static int write_entry(struct cache_entry *ce,
240 char *path, const struct checkout *state, int to_tempfile)
241 {
@@ -179,11 +278,34 @@ static int write_entry(struct cache_entry *ce,
278 /*
279 * Convert from git internal format to working tree format
280 */
182 - if (ce_mode_s_ifmt == S_IFREG &&
183 - convert_to_working_tree(ce->name, new, size, &buf)) {
184 - free(new);
185 - new = strbuf_detach(&buf, &newsize);
186 - size = newsize;
281 + if (ce_mode_s_ifmt == S_IFREG) {
282 + struct delayed_checkout *dco = state->delayed_checkout;
283 + if (dco && dco->state != CE_NO_DELAY) {
284 + /* Do not send the blob in case of a retry. */
285 + if (dco->state == CE_RETRY) {
286 + new = NULL;
287 + size = 0;
288 + }
289 + ret = async_convert_to_working_tree(
290 + ce->name, new, size, &buf, dco);
291 + if (ret && string_list_has_string(&dco->paths, ce->name)) {
292 + free(new);
293 + goto finish;
294 + }
295 + } else
296 + ret = convert_to_working_tree(
297 + ce->name, new, size, &buf);
298 +
299 + if (ret) {
300 + free(new);
301 + new = strbuf_detach(&buf, &newsize);
302 + size = newsize;
303 + }
304 + /*
305 + * No "else" here as errors from convert are OK at this
306 + * point. If the error would have been fatal (e.g.
307 + * filter is required), then we would have died already.
308 + */
309 }
310
311 fd = open_output_fd(path, ce, to_tempfile);
t/t0021-conversion.sh
+116
@@ -701,4 +701,120 @@ test_expect_success PERL 'invalid process filter must fail (and not hang!)' '
701 )
702 '
703
704 +test_expect_success PERL 'delayed checkout in process filter' '
705 + test_config_global filter.a.process "rot13-filter.pl a.log clean smudge delay" &&
706 + test_config_global filter.a.required true &&
707 + test_config_global filter.b.process "rot13-filter.pl b.log clean smudge delay" &&
708 + test_config_global filter.b.required true &&
709 +
710 + rm -rf repo &&
711 + mkdir repo &&
712 + (
713 + cd repo &&
714 + git init &&
715 + echo "*.a filter=a" >.gitattributes &&
716 + echo "*.b filter=b" >>.gitattributes &&
717 + cp "$TEST_ROOT/test.o" test.a &&
718 + cp "$TEST_ROOT/test.o" test-delay10.a &&
719 + cp "$TEST_ROOT/test.o" test-delay11.a &&
720 + cp "$TEST_ROOT/test.o" test-delay20.a &&
721 + cp "$TEST_ROOT/test.o" test-delay10.b &&
722 + git add . &&
723 + git commit -m "test commit"
724 + ) &&
725 +
726 + S=$(file_size "$TEST_ROOT/test.o") &&
727 + cat >a.exp <<-EOF &&
728 + START
729 + init handshake complete
730 + IN: smudge test.a $S [OK] -- OUT: $S . [OK]
731 + IN: smudge test-delay10.a $S [OK] -- [DELAYED]
732 + IN: smudge test-delay11.a $S [OK] -- [DELAYED]
733 + IN: smudge test-delay20.a $S [OK] -- [DELAYED]
734 + IN: list_available_blobs test-delay10.a test-delay11.a [OK]
735 + IN: smudge test-delay10.a 0 [OK] -- OUT: $S . [OK]
736 + IN: smudge test-delay11.a 0 [OK] -- OUT: $S . [OK]
737 + IN: list_available_blobs test-delay20.a [OK]
738 + IN: smudge test-delay20.a 0 [OK] -- OUT: $S . [OK]
739 + IN: list_available_blobs [OK]
740 + STOP
741 + EOF
742 + cat >b.exp <<-EOF &&
743 + START
744 + init handshake complete
745 + IN: smudge test-delay10.b $S [OK] -- [DELAYED]
746 + IN: list_available_blobs test-delay10.b [OK]
747 + IN: smudge test-delay10.b 0 [OK] -- OUT: $S . [OK]
748 + IN: list_available_blobs [OK]
749 + STOP
750 + EOF
751 +
752 + rm -rf repo-cloned &&
753 + filter_git clone repo repo-cloned &&
754 + test_cmp_count a.exp repo-cloned/a.log &&
755 + test_cmp_count b.exp repo-cloned/b.log &&
756 +
757 + (
758 + cd repo-cloned &&
759 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.a &&
760 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.a &&
761 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay11.a &&
762 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay20.a &&
763 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.b &&
764 +
765 + rm *.a *.b &&
766 + filter_git checkout . &&
767 + test_cmp_count ../a.exp a.log &&
768 + test_cmp_count ../b.exp b.log &&
769 +
770 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.a &&
771 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.a &&
772 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay11.a &&
773 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay20.a &&
774 + test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.b
775 + )
776 +'
777 +
778 +test_expect_success PERL 'missing file in delayed checkout' '
779 + test_config_global filter.bug.process "rot13-filter.pl bug.log clean smudge delay" &&
780 + test_config_global filter.bug.required true &&
781 +
782 + rm -rf repo &&
783 + mkdir repo &&
784 + (
785 + cd repo &&
786 + git init &&
787 + echo "*.a filter=bug" >.gitattributes &&
788 + cp "$TEST_ROOT/test.o" missing-delay.a
789 + git add . &&
790 + git commit -m "test commit"
791 + ) &&
792 +
793 + rm -rf repo-cloned &&
794 + test_must_fail git clone repo repo-cloned 2>git-stderr.log &&
795 + cat git-stderr.log &&
796 + grep "error: .missing-delay\.a. was not filtered properly" git-stderr.log
797 +'
798 +
799 +test_expect_success PERL 'invalid file in delayed checkout' '
800 + test_config_global filter.bug.process "rot13-filter.pl bug.log clean smudge delay" &&
801 + test_config_global filter.bug.required true &&
802 +
803 + rm -rf repo &&
804 + mkdir repo &&
805 + (
806 + cd repo &&
807 + git init &&
808 + echo "*.a filter=bug" >.gitattributes &&
809 + cp "$TEST_ROOT/test.o" invalid-delay.a &&
810 + cp "$TEST_ROOT/test.o" unfiltered
811 + git add . &&
812 + git commit -m "test commit"
813 + ) &&
814 +
815 + rm -rf repo-cloned &&
816 + test_must_fail git clone repo repo-cloned 2>git-stderr.log &&
817 + grep "error: external filter .* signaled that .unfiltered. is now available although it has not been delayed earlier" git-stderr.log
818 +'
819 +
820 test_done
t/t0021/rot13-filter.pl
+139 -65
@@ -18,6 +18,16 @@
18 # operation then the filter signals that it cannot or does not want
19 # to process the file and any file after that is processed with the
20 # same command.
21 +# (5) If data with a pathname that is a key in the DELAY hash is
22 +# requested (e.g. "test-delay10.a") then the filter responds with
23 +# a "delay" status and sets the "requested" field in the DELAY hash.
24 +# The filter will signal the availability of this object after
25 +# "count" (field in DELAY hash) "list_available_blobs" commands.
26 +# (6) If data with the pathname "missing-delay.a" is processed that the
27 +# filter will drop the path from the "list_available_blobs" response.
28 +# (7) If data with the pathname "invalid-delay.a" is processed that the
29 +# filter will add the path "unfiltered" which was not delayed before
30 +# to the "list_available_blobs" response.
31 #
32
33 use strict;
@@ -30,6 +40,15 @@ my @capabilities = @ARGV;
40
41 open my $debug, ">>", $log_file or die "cannot open log file: $!";
42
43 +my %DELAY = (
44 + 'test-delay10.a' => { "requested" => 0, "count" => 1 },
45 + 'test-delay11.a' => { "requested" => 0, "count" => 1 },
46 + 'test-delay20.a' => { "requested" => 0, "count" => 2 },
47 + 'test-delay10.b' => { "requested" => 0, "count" => 1 },
48 + 'missing-delay.a' => { "requested" => 0, "count" => 1 },
49 + 'invalid-delay.a' => { "requested" => 0, "count" => 1 },
50 +);
51 +
52 sub rot13 {
53 my $str = shift;
54 $str =~ y/A-Za-z/N-ZA-Mn-za-m/;
@@ -66,7 +85,7 @@ sub packet_bin_read {
85
86 sub packet_txt_read {
87 my ( $res, $buf ) = packet_bin_read();
69 - unless ( $buf =~ s/\n$// ) {
88 + unless ( $buf eq '' or $buf =~ s/\n$// ) {
89 die "A non-binary line MUST be terminated by an LF.";
90 }
91 return ( $res, $buf );
@@ -101,6 +120,7 @@ packet_flush();
120
121 ( packet_txt_read() eq ( 0, "capability=clean" ) ) || die "bad capability";
122 ( packet_txt_read() eq ( 0, "capability=smudge" ) ) || die "bad capability";
123 +( packet_txt_read() eq ( 0, "capability=delay" ) ) || die "bad capability";
124 ( packet_bin_read() eq ( 1, "" ) ) || die "bad capability end";
125
126 foreach (@capabilities) {
@@ -111,88 +131,142 @@ print $debug "init handshake complete\n";
131 $debug->flush();
132
133 while (1) {
114 - my ($command) = packet_txt_read() =~ /^command=(.+)$/;
134 + my ( $command ) = packet_txt_read() =~ /^command=(.+)$/;
135 print $debug "IN: $command";
136 $debug->flush();
137
118 - my ($pathname) = packet_txt_read() =~ /^pathname=(.+)$/;
119 - print $debug " $pathname";
120 - $debug->flush();
121 -
122 - if ( $pathname eq "" ) {
123 - die "bad pathname '$pathname'";
124 - }
138 + if ( $command eq "list_available_blobs" ) {
139 + # Flush
140 + packet_bin_read();
141
126 - # Flush
127 - packet_bin_read();
128 -
129 - my $input = "";
130 - {
131 - binmode(STDIN);
132 - my $buffer;
133 - my $done = 0;
134 - while ( !$done ) {
135 - ( $done, $buffer ) = packet_bin_read();
136 - $input .= $buffer;
142 + foreach my $pathname ( sort keys %DELAY ) {
143 + if ( $DELAY{$pathname}{"requested"} >= 1 ) {
144 + $DELAY{$pathname}{"count"} = $DELAY{$pathname}{"count"} - 1;
145 + if ( $pathname eq "invalid-delay.a" ) {
146 + # Send Git a pathname that was not delayed earlier
147 + packet_txt_write("pathname=unfiltered");
148 + }
149 + if ( $pathname eq "missing-delay.a" ) {
150 + # Do not signal Git that this file is available
151 + } elsif ( $DELAY{$pathname}{"count"} == 0 ) {
152 + print $debug " $pathname";
153 + packet_txt_write("pathname=$pathname");
154 + }
155 + }
156 }
138 - print $debug " " . length($input) . " [OK] -- ";
139 - $debug->flush();
140 - }
141 -
142 - my $output;
143 - if ( $pathname eq "error.r" or $pathname eq "abort.r" ) {
144 - $output = "";
145 - }
146 - elsif ( $command eq "clean" and grep( /^clean$/, @capabilities ) ) {
147 - $output = rot13($input);
148 - }
149 - elsif ( $command eq "smudge" and grep( /^smudge$/, @capabilities ) ) {
150 - $output = rot13($input);
151 - }
152 - else {
153 - die "bad command '$command'";
154 - }
157
156 - if ( $pathname eq "error.r" ) {
157 - print $debug "[ERROR]\n";
158 - $debug->flush();
159 - packet_txt_write("status=error");
158 packet_flush();
161 - }
162 - elsif ( $pathname eq "abort.r" ) {
163 - print $debug "[ABORT]\n";
159 +
160 + print $debug " [OK]\n";
161 $debug->flush();
165 - packet_txt_write("status=abort");
162 + packet_txt_write("status=success");
163 packet_flush();
164 }
165 else {
169 - packet_txt_write("status=success");
170 - packet_flush();
166 + my ( $pathname ) = packet_txt_read() =~ /^pathname=(.+)$/;
167 + print $debug " $pathname";
168 + $debug->flush();
169 +
170 + if ( $pathname eq "" ) {
171 + die "bad pathname '$pathname'";
172 + }
173 +
174 + # Read until flush
175 + my ( $done, $buffer ) = packet_txt_read();
176 + while ( $buffer ne '' ) {
177 + if ( $buffer eq "can-delay=1" ) {
178 + if ( exists $DELAY{$pathname} and $DELAY{$pathname}{"requested"} == 0 ) {
179 + $DELAY{$pathname}{"requested"} = 1;
180 + }
181 + } else {
182 + die "Unknown message '$buffer'";
183 + }
184
172 - if ( $pathname eq "${command}-write-fail.r" ) {
173 - print $debug "[WRITE FAIL]\n";
185 + ( $done, $buffer ) = packet_txt_read();
186 + }
187 +
188 + my $input = "";
189 + {
190 + binmode(STDIN);
191 + my $buffer;
192 + my $done = 0;
193 + while ( !$done ) {
194 + ( $done, $buffer ) = packet_bin_read();
195 + $input .= $buffer;
196 + }
197 + print $debug " " . length($input) . " [OK] -- ";
198 $debug->flush();
175 - die "${command} write error";
199 }
200
178 - print $debug "OUT: " . length($output) . " ";
179 - $debug->flush();
201 + my $output;
202 + if ( exists $DELAY{$pathname} and exists $DELAY{$pathname}{"output"} ) {
203 + $output = $DELAY{$pathname}{"output"}
204 + }
205 + elsif ( $pathname eq "error.r" or $pathname eq "abort.r" ) {
206 + $output = "";
207 + }
208 + elsif ( $command eq "clean" and grep( /^clean$/, @capabilities ) ) {
209 + $output = rot13($input);
210 + }
211 + elsif ( $command eq "smudge" and grep( /^smudge$/, @capabilities ) ) {
212 + $output = rot13($input);
213 + }
214 + else {
215 + die "bad command '$command'";
216 + }
217 +
218 + if ( $pathname eq "error.r" ) {
219 + print $debug "[ERROR]\n";
220 + $debug->flush();
221 + packet_txt_write("status=error");
222 + packet_flush();
223 + }
224 + elsif ( $pathname eq "abort.r" ) {
225 + print $debug "[ABORT]\n";
226 + $debug->flush();
227 + packet_txt_write("status=abort");
228 + packet_flush();
229 + }
230 + elsif ( $command eq "smudge" and
231 + exists $DELAY{$pathname} and
232 + $DELAY{$pathname}{"requested"} == 1
233 + ) {
234 + print $debug "[DELAYED]\n";
235 + $debug->flush();
236 + packet_txt_write("status=delayed");
237 + packet_flush();
238 + $DELAY{$pathname}{"requested"} = 2;
239 + $DELAY{$pathname}{"output"} = $output;
240 + }
241 + else {
242 + packet_txt_write("status=success");
243 + packet_flush();
244
181 - while ( length($output) > 0 ) {
182 - my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );
183 - packet_bin_write($packet);
184 - # dots represent the number of packets
185 - print $debug ".";
186 - if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {
187 - $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );
245 + if ( $pathname eq "${command}-write-fail.r" ) {
246 + print $debug "[WRITE FAIL]\n";
247 + $debug->flush();
248 + die "${command} write error";
249 }
189 - else {
190 - $output = "";
250 +
251 + print $debug "OUT: " . length($output) . " ";
252 + $debug->flush();
253 +
254 + while ( length($output) > 0 ) {
255 + my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );
256 + packet_bin_write($packet);
257 + # dots represent the number of packets
258 + print $debug ".";
259 + if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {
260 + $output = substr( $output, $MAX_PACKET_CONTENT_SIZE );
261 + }
262 + else {
263 + $output = "";
264 + }
265 }
266 + packet_flush();
267 + print $debug " [OK]\n";
268 + $debug->flush();
269 + packet_flush();
270 }
193 - packet_flush();
194 - print $debug " [OK]\n";
195 - $debug->flush();
196 - packet_flush();
271 }
272 }
unpack-trees.c
+2
@@ -379,6 +379,7 @@ static int check_updates(struct unpack_trees_options *o)
379 if (should_update_submodules() && o->update && !o->dry_run)
380 reload_gitmodules_file(index, &state);
381
382 + enable_delayed_checkout(&state);
383 for (i = 0; i < index->cache_nr; i++) {
384 struct cache_entry *ce = index->cache[i];
385
@@ -393,6 +394,7 @@ static int check_updates(struct unpack_trees_options *o)
394 }
395 }
396 }
397 + errs |= finish_delayed_checkout(&state);
398 stop_progress(&progress);
399 if (o->update)
400 git_attr_set_direction(GIT_ATTR_CHECKIN, NULL);