reftable/merged: fix zero-sized allocation when there are no readers

It was reported [1] that Git started to fail with an out-of-memory error when initializing repositories with the reftable backend on NonStop platforms. A bisect led to 802c0646ac (reftable/merged: handle allocation failures in `merged_table_init_iter()`, 2024-10-02), which changed how we allocate memory when initializing a merged table. The root cause of this seems to be that NonStop returns a `NULL` pointer when doing a zero-sized allocation. This would've already happened before the above change, but we never noticed because we did not check the result. Now we do notice and thus return an out-of-memory error to the caller. Fix the issue by skipping the allocation altogether in case there are no readers. [1]: <00ad01db5017$aa9ce340$ffd6a9c0$@nexbridge.com> Reported-by: Randall S. Becker <rsbecker@nexbridge.com> Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Patrick Steinhardt committed Dec 22, 2024 at 08:24 UTC 5ab83521cfe687e4295f5748f2c5d2aa7477efe5
1 file changed +7 -5
reftable/merged.c
+7 -5
@@ -238,14 +238,16 @@ int merged_table_init_iter(struct reftable_merged_table *mt,
238 struct reftable_iterator *it,
239 uint8_t typ)
240 {
241 - struct merged_subiter *subiters;
241 + struct merged_subiter *subiters = NULL;
242 struct merged_iter *mi = NULL;
243 int ret;
244
245 - REFTABLE_CALLOC_ARRAY(subiters, mt->readers_len);
246 - if (!subiters) {
247 - ret = REFTABLE_OUT_OF_MEMORY_ERROR;
248 - goto out;
245 + if (mt->readers_len) {
246 + REFTABLE_CALLOC_ARRAY(subiters, mt->readers_len);
247 + if (!subiters) {
248 + ret = REFTABLE_OUT_OF_MEMORY_ERROR;
249 + goto out;
250 + }
251 }
252
253 for (size_t i = 0; i < mt->readers_len; i++) {