harden REALLOC_ARRAY and xcalloc against size_t overflow

REALLOC_ARRAY inherently involves a multiplication which can overflow size_t, resulting in a much smaller buffer than we think we've allocated. We can easily harden it by using st_mult() to check for overflow. Likewise, we can add ALLOC_ARRAY to do the same thing for xmalloc calls. xcalloc() should already be fine, because it takes the two factors separately, assuming the system calloc actually checks for overflow. However, before we even hit the system calloc(), we do our memory_limit_check, which involves a multiplication. Let's check for overflow ourselves so that this limit cannot be bypassed. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Jeff King committed Feb 22, 2016 at 17:43 UTC e7792a74bcf7fcd554b4546fc91122b8c2af7d75
2 files changed +5 -1
git-compat-util.h
+2 -1
@@ -779,7 +779,8 @@ extern int odb_pack_keep(char *name, size_t namesz, const unsigned char *sha1);
779 extern char *xgetcwd(void);
780 extern FILE *fopen_for_writing(const char *path);
781
782 -#define REALLOC_ARRAY(x, alloc) (x) = xrealloc((x), (alloc) * sizeof(*(x)))
782 +#define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))
783 +#define REALLOC_ARRAY(x, alloc) (x) = xrealloc((x), st_mult(sizeof(*(x)), (alloc)))
784
785 static inline char *xstrdup_or_null(const char *str)
786 {
wrapper.c
+3
@@ -152,6 +152,9 @@ void *xcalloc(size_t nmemb, size_t size)
152 {
153 void *ret;
154
155 + if (unsigned_mult_overflows(nmemb, size))
156 + die("data too large to fit into virtual memory space");
157 +
158 memory_limit_check(size * nmemb, 0);
159 ret = calloc(nmemb, size);
160 if (!ret && (!nmemb || !size))