diffcore-rename: filter rename_src list when possible

We have to look at each entry in rename_src a total of rename_dst_nr times. When we're not detecting copies, any exact renames or ignorable rename paths will just be skipped over. While checking that these can be skipped over is a relatively cheap check, it's still a waste of time to do that check more than once, let alone rename_dst_nr times. When rename_src_nr is a few thousand times bigger than the number of relevant sources (such as when cherry-picking a commit that only touched a handful of files, but from a side of history that has different names for some high level directories), this time can add up. First make an initial pass over the rename_src array and move all the relevant entries to the front, so that we can iterate over just those relevant entries. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 14.119 s ± 0.101 s 13.815 s ± 0.062 s mega-renames: 1802.044 s ± 0.828 s 1799.937 s ± 0.493 s just-one-mega: 51.391 s ± 0.028 s 51.289 s ± 0.019 s Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Elijah Newren committed Feb 14, 2021 at 07:35 UTC 829514c5151deee1b37cdeaf451bf28219602126
1 file changed +51 -8
diffcore-rename.c
+51 -8
@@ -454,6 +454,54 @@ static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, i
454 return count;
455 }
456
457 +static void remove_unneeded_paths_from_src(int detecting_copies)
458 +{
459 + int i, new_num_src;
460 +
461 + if (detecting_copies)
462 + return; /* nothing to remove */
463 + if (break_idx)
464 + return; /* culling incompatible with break detection */
465 +
466 + /*
467 + * Note on reasons why we cull unneeded sources but not destinations:
468 + * 1) Pairings are stored in rename_dst (not rename_src), which we
469 + * need to keep around. So, we just can't cull rename_dst even
470 + * if we wanted to. But doing so wouldn't help because...
471 + *
472 + * 2) There is a matrix pairwise comparison that follows the
473 + * "Performing inexact rename detection" progress message.
474 + * Iterating over the destinations is done in the outer loop,
475 + * hence we only iterate over each of those once and we can
476 + * easily skip the outer loop early if the destination isn't
477 + * relevant. That's only one check per destination path to
478 + * skip.
479 + *
480 + * By contrast, the sources are iterated in the inner loop; if
481 + * we check whether a source can be skipped, then we'll be
482 + * checking it N separate times, once for each destination.
483 + * We don't want to have to iterate over known-not-needed
484 + * sources N times each, so avoid that by removing the sources
485 + * from rename_src here.
486 + */
487 + for (i = 0, new_num_src = 0; i < rename_src_nr; i++) {
488 + /*
489 + * renames are stored in rename_dst, so if a rename has
490 + * already been detected using this source, we can just
491 + * remove the source knowing rename_dst has its info.
492 + */
493 + if (rename_src[i].p->one->rename_used)
494 + continue;
495 +
496 + if (new_num_src < i)
497 + memcpy(&rename_src[new_num_src], &rename_src[i],
498 + sizeof(struct diff_rename_src));
499 + new_num_src++;
500 + }
501 +
502 + rename_src_nr = new_num_src;
503 +}
504 +
505 void diffcore_rename(struct diff_options *options)
506 {
507 int detect_rename = options->detect_rename;
@@ -529,14 +577,10 @@ void diffcore_rename(struct diff_options *options)
577 if (minimum_score == MAX_SCORE)
578 goto cleanup;
579
532 - /*
533 - * Calculate how many renames are left (but all the source
534 - * files still remain as options for rename/copies!)
535 - */
580 + /* Calculate how many renames are left */
581 num_destinations = (rename_dst_nr - rename_count);
582 + remove_unneeded_paths_from_src(want_copies);
583 num_sources = rename_src_nr;
538 - if (!want_copies)
539 - num_sources -= rename_count;
584
585 /* All done? */
586 if (!num_destinations || !num_sources)
@@ -578,8 +622,7 @@ void diffcore_rename(struct diff_options *options)
622 struct diff_filespec *one = rename_src[j].p->one;
623 struct diff_score this_src;
624
581 - if (one->rename_used && !want_copies)
582 - continue;
625 + assert(!one->rename_used || want_copies || break_idx);
626
627 if (skip_unmodified &&
628 diff_unmodified_pair(rename_src[j].p))