@matheusbsbs / matheus-bagliano / commits / 0b6806b9e4

xread, xwrite: limit size of IO to 8MB

Checking out 2GB or more through an external filter (see test) fails on Mac OS X 10.8.4 (12E55) for a 64-bit executable with: error: read from external filter cat failed error: cannot feed the input to external filter cat error: cat died of signal 13 error: external filter cat failed 141 error: external filter cat failed The reason is that read() immediately returns with EINVAL when asked to read more than 2GB. According to POSIX [1], if the value of nbyte passed to read() is greater than SSIZE_MAX, the result is implementation-defined. The write function has the same restriction [2]. Since OS X still supports running 32-bit executables, the 32-bit limit (SSIZE_MAX = INT_MAX = 2GB - 1) seems to be also imposed on 64-bit executables under certain conditions. For write, the problem has been addressed earlier [6c642a]. Address the problem for read() and write() differently, by limiting size of IO chunks unconditionally on all platforms in xread() and xwrite(). Large chunks only cause problems, like causing latencies when killing the process, even if OS X was not buggy. Doing IO in reasonably sized smaller chunks should have no negative impact on performance. The compat wrapper clipped_write() introduced earlier [6c642a] is not needed anymore. It will be reverted in a separate commit. The new test catches read and write problems. Note that 'git add' exits with 0 even if it prints filtering errors to stderr. The test, therefore, checks stderr. 'git add' should probably be changed (sometime in another commit) to exit with nonzero if filtering fails. The test could then be changed to use test_must_fail. Thanks to the following people for suggestions and testing: Johannes Sixt <j6t@kdbg.org> John Keeping <john@keeping.me.uk> Jonathan Nieder <jrnieder@gmail.com> Kyle J. McKay <mackyle@gmail.com> Linus Torvalds <torvalds@linux-foundation.org> Torsten Bögershausen <tboegi@web.de> [1] http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html [2] http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html [6c642a] commit 6c642a878688adf46b226903858b53e2d31ac5c3 compate/clipped-write.c: large write(2) fails on Mac OS X/XNU Signed-off-by: Steffen Prohaska <prohaska@zib.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Steffen Prohaska committed Aug 20, 2013 at 08:43 UTC 0b6806b9e45c659d25b87fb5713c920a3081eac8

2 files changed +26

t/t0021-conversion.sh

+14

@@ -190,4 +190,18 @@ test_expect_success 'required filter clean failure' '

190 test_must_fail git add test.fc

191 '

192

193 +test -n "$GIT_TEST_LONG" && test_set_prereq EXPENSIVE

194 +

195 +test_expect_success EXPENSIVE 'filter large file' '

196 + git config filter.largefile.smudge cat &&

197 + git config filter.largefile.clean cat &&

198 + for i in $(test_seq 1 2048); do printf "%1048576d" 1; done >2GB &&

199 + echo "2GB filter=largefile" >.gitattributes &&

200 + git add 2GB 2>err &&

201 + ! test -s err &&

202 + rm -f 2GB &&

203 + git checkout -- 2GB 2>err &&

204 + ! test -s err

205 +'

206 +

207 test_done

wrapper.c

+12

@@ -130,6 +130,14 @@ void *xcalloc(size_t nmemb, size_t size)

130 return ret;

131 }

132

133 +/*

134 + * Limit size of IO chunks, because huge chunks only cause pain. OS X

135 + * 64-bit is buggy, returning EINVAL if len >= INT_MAX; and even in

136 + * the absense of bugs, large chunks can result in bad latencies when

137 + * you decide to kill the process.

138 + */

139 +#define MAX_IO_SIZE (8*1024*1024)

140 +

141 /*

142 * xread() is the same a read(), but it automatically restarts read()

143 * operations with a recoverable error (EAGAIN and EINTR). xread()

@@ -138,6 +146,8 @@ void *xcalloc(size_t nmemb, size_t size)

146 ssize_t xread(int fd, void *buf, size_t len)

147 {

148 ssize_t nr;

149 + if (len > MAX_IO_SIZE)

150 + len = MAX_IO_SIZE;

151 while (1) {

152 nr = read(fd, buf, len);

153 if ((nr < 0) && (errno == EAGAIN || errno == EINTR))

@@ -154,6 +164,8 @@ ssize_t xread(int fd, void *buf, size_t len)

164 ssize_t xwrite(int fd, const void *buf, size_t len)

165 {

166 ssize_t nr;

167 + if (len > MAX_IO_SIZE)

168 + len = MAX_IO_SIZE;

169 while (1) {

170 nr = write(fd, buf, len);

171 if ((nr < 0) && (errno == EAGAIN || errno == EINTR))