@leroysheep / rag-system / commits / 740d24e541

fast-import: use writev(3p) to send cat-blob responses

When answering a `cat-blob` command, `cat_blob()` issues three separate calls to write(3p) on the cat-blob fd: one for the header line, one for the full blob payload, and one for the trailing newline. Frontends like git-filter-repo issue these commands in bulk, once per rewritten blob, so the syscall overhead adds up. Use `writev_in_full()` to send all three parts with a single syscall. This can be benchmarked with the following setup: $ git cat-file --unordered --filter=object:type=blob --batch-check='cat-blob %(objectname)' --batch-all-objects >request $ git fast-import --cat-blob-fd=3 <request Executing this with 100,000 objects in linux.git: Benchmark 1: HEAD~ Time (mean ± σ): 1.320 s ± 0.003 s [User: 1.154 s, System: 0.161 s] Range (min … max): 1.314 s … 1.324 s 10 runs Benchmark 2: HEAD Time (mean ± σ): 1.270 s ± 0.022 s [User: 1.133 s, System: 0.132 s] Range (min … max): 1.209 s … 1.282 s 10 runs Summary HEAD ran 1.04 ± 0.02 times faster than HEAD~ Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Patrick Steinhardt committed Jul 16, 2026 at 09:52 UTC 740d24e5416182d45cecfdf5eaa1074ee6005ec1

1 file changed +15 -3

builtin/fast-import.c

+15 -3

index aa656c5195..48fda01c94 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -3332,6 +3332,7 @@ static void cat_blob_write(const char *buf, unsigned long size) static void cat_blob(struct object_entry *oe, struct object_id *oid) { struct strbuf line = STRBUF_INIT; + struct iovec iov[3]; unsigned long size; enum object_type type = 0; char *buf; @@ -3365,10 +3366,21 @@ static void cat_blob(struct object_entry *oe, struct object_id *oid) strbuf_reset(&line); strbuf_addf(&line, "%s %s %"PRIuMAX"\n", oid_to_hex(oid), type_name(type), (uintmax_t)size); - cat_blob_write(line.buf, line.len); + + /* + * Write the header, the payload and the trailing newline with a + * single writev(3p) call instead of three separate write(3p) calls. + */ + iov[0].iov_base = line.buf; + iov[0].iov_len = line.len; + iov[1].iov_base = buf; + iov[1].iov_len = size; + iov[2].iov_base = (void *) "\n"; + iov[2].iov_len = 1; + + if (writev_in_full(cat_blob_fd, iov, ARRAY_SIZE(iov)) < 0) + die_errno(_("write to frontend failed")); strbuf_release(&line); - cat_blob_write(buf, size); - cat_blob_write("\n", 1); if (oe && oe->pack_id == pack_id) { last_blob.offset = oe->idx.offset; strbuf_attach(&last_blob.data, buf, size, size + 1);