convert: avoid malloc of original file size

We write the output of a "clean" filter into a strbuf. Rather than growing the strbuf dynamically as we read its output, we make the initial allocation as large as the original input file. This is a good guess when the filter is just tweaking a few bytes, but it's disastrous when the point of the filter is to condense a very large file into a short identifier (e.g., the way git-lfs and git-annex do). We may ask to allocate many gigabytes, causing the allocation to fail and Git to die(). Instead, let's just let strbuf do its usual growth. When the clean filter does output something around the same size as the worktree file, the buffer will need to be reallocated until it fits, starting at 8192 and doubling in size. Benchmarking indicates that reallocation is not a significant overhead for outputs up to a few MB in size. Signed-off-by: Joey Hess <id@joeyh.name> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Joey Hess committed Mar 7, 2019 at 14:56 UTC 02156ab031e430bc45ce6984dfc712de9962dec8
1 file changed +1 -1
convert.c
+1 -1
@@ -732,7 +732,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le
732 if (start_async(&async))
733 return 0; /* error was already reported */
734
735 - if (strbuf_read(&nbuf, async.out, len) < 0) {
735 + if (strbuf_read(&nbuf, async.out, 0) < 0) {
736 err = error(_("read from external filter '%s' failed"), cmd);
737 }
738 if (close(async.out)) {