@matheusbsbs / matheus-bagliano / commits / 1d028dc682

blame: add a fingerprint heuristic to match ignored lines

This algorithm will replace the heuristic used to identify lines from ignored commits with one that finds likely candidate lines in the parent's version of the file. The actual replacement occurs in an upcoming commit. The old heuristic simply assigned lines in the target to the same line number (plus offset) in the parent. The new function uses a fingerprinting algorithm to detect similarity between lines. The new heuristic is designed to accurately match changes made mechanically by formatting tools such as clang-format and clang-tidy. These tools make changes such as breaking up lines to fit within a character limit or changing identifiers to fit with a naming convention. The heuristic is not intended to match more extensive refactoring changes and may give misleading results in such cases. In most cases formatting tools preserve line ordering, so the heuristic is optimised for such cases. (Some types of changes do reorder lines e.g. sorting keep the line content identical, the git blame -M option can already be used to address this). The reason that it is advantageous to rely on ordering is due to source code repeating the same character sequences often e.g. declaring an identifier on one line and using that identifier on several subsequent lines. This means that lines can look very similar to each other which presents a problem when doing fuzzy matching. Relying on ordering gives us extra clues to point towards the true match. The heuristic operates on a single diff chunk change at a time. It creates a “fingerprint” for each line on each side of the change. Fingerprints are described in detail in the comment for `struct fingerprint`, but essentially are a multiset of the character pairs in a line. The heuristic first identifies the line in the target entry whose fingerprint is most clearly matched to a line fingerprint in the parent entry. Where fingerprints match identically, the position of the lines is used as a tie-break. The heuristic locks in the best match, and subtracts the fingerprint of the line in the target entry from the fingerprint of the line in the parent entry to prevent other lines being matched on the same parts of that line. It then repeats the process recursively on the section of the chunk before the match, and then the section of the chunk after the match. Here's an example of the difference the fingerprinting makes. Consider a file with two commits: commit-a 1) void func_1(void *x, void *y); commit-b 2) void func_2(void *x, void *y); After a commit 'X', we have: commit-X 1) void func_1(void *x, commit-X 2) void *y); commit-X 3) void func_2(void *x, commit-X 4) void *y); When we blame-ignored with the old algorithm, we get: commit-a 1) void func_1(void *x, commit-b 2) void *y); commit-X 3) void func_2(void *x, commit-X 4) void *y); Where commit-b is blamed for 2 instead of 3. With the fingerprint algorithm, we get: commit-a 1) void func_1(void *x, commit-a 2) void *y); commit-b 3) void func_2(void *x, commit-b 4) void *y); Note line 2 could be matched with either commit-a or commit-b as it is equally similar to both lines, but is matched with commit-a because its position as a fraction of the new line range is more similar to commit-a as a fraction of the old line range. Line 4 is also equally similar to both lines, but as it appears after line 3 which will be matched first it cannot be matched with an earlier line. For many more examples, see t/t8014-blame-ignore-fuzzy.sh which contains example parent and target files and the line numbers in the parent that must be matched. Signed-off-by: Michael Platings <michael@platin.gs> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Michael Platings committed Jun 20, 2019 at 12:38 UTC 1d028dc682d0cb4420fb124419ebc60e913d421c

2 files changed +1082

blame.c

+642

@@ -339,6 +339,648 @@ static int find_line_starts(int **line_starts, const char *buf,

339 return num;

340 }

341

342 +struct fingerprint_entry;

343 +

344 +/* A fingerprint is intended to loosely represent a string, such that two

345 + * fingerprints can be quickly compared to give an indication of the similarity

346 + * of the strings that they represent.

347 + *

348 + * A fingerprint is represented as a multiset of the lower-cased byte pairs in

349 + * the string that it represents. Whitespace is added at each end of the

350 + * string. Whitespace pairs are ignored. Whitespace is converted to '\0'.

351 + * For example, the string "Darth Radar" will be converted to the following

352 + * fingerprint:

353 + * {"\0d", "da", "da", "ar", "ar", "rt", "th", "h\0", "\0r", "ra", "ad", "r\0"}

354 + *

355 + * The similarity between two fingerprints is the size of the intersection of

356 + * their multisets, including repeated elements. See fingerprint_similarity for

357 + * examples.

358 + *

359 + * For ease of implementation, the fingerprint is implemented as a map

360 + * of byte pairs to the count of that byte pair in the string, instead of

361 + * allowing repeated elements in a set.

362 + */

363 +struct fingerprint {

364 + struct hashmap map;

365 + /* As we know the maximum number of entries in advance, it's

366 + * convenient to store the entries in a single array instead of having

367 + * the hashmap manage the memory.

368 + */

369 + struct fingerprint_entry *entries;

370 +};

371 +

372 +/* A byte pair in a fingerprint. Stores the number of times the byte pair

373 + * occurs in the string that the fingerprint represents.

374 + */

375 +struct fingerprint_entry {

376 + /* The hashmap entry - the hash represents the byte pair in its

377 + * entirety so we don't need to store the byte pair separately.

378 + */

379 + struct hashmap_entry entry;

380 + /* The number of times the byte pair occurs in the string that the

381 + * fingerprint represents.

382 + */

383 + int count;

384 +};

385 +

386 +/* See `struct fingerprint` for an explanation of what a fingerprint is.

387 + * \param result the fingerprint of the string is stored here. This must be

388 + * freed later using free_fingerprint.

389 + * \param line_begin the start of the string

390 + * \param line_end the end of the string

391 + */

392 +static void get_fingerprint(struct fingerprint *result,

393 + const char *line_begin,

394 + const char *line_end)

395 +{

396 + unsigned int hash, c0 = 0, c1;

397 + const char *p;

398 + int max_map_entry_count = 1 + line_end - line_begin;

399 + struct fingerprint_entry *entry = xcalloc(max_map_entry_count,

400 + sizeof(struct fingerprint_entry));

401 + struct fingerprint_entry *found_entry;

402 +

403 + hashmap_init(&result->map, NULL, NULL, max_map_entry_count);

404 + result->entries = entry;

405 + for (p = line_begin; p <= line_end; ++p, c0 = c1) {

406 + /* Always terminate the string with whitespace.

407 + * Normalise whitespace to 0, and normalise letters to

408 + * lower case. This won't work for multibyte characters but at

409 + * worst will match some unrelated characters.

410 + */

411 + if ((p == line_end) || isspace(*p))

412 + c1 = 0;

413 + else

414 + c1 = tolower(*p);

415 + hash = c0 | (c1 << 8);

416 + /* Ignore whitespace pairs */

417 + if (hash == 0)

418 + continue;

419 + hashmap_entry_init(entry, hash);

420 +

421 + found_entry = hashmap_get(&result->map, entry, NULL);

422 + if (found_entry) {

423 + found_entry->count += 1;

424 + } else {

425 + entry->count = 1;

426 + hashmap_add(&result->map, entry);

427 + ++entry;

428 + }

429 + }

430 +}

431 +

432 +static void free_fingerprint(struct fingerprint *f)

433 +{

434 + hashmap_free(&f->map, 0);

435 + free(f->entries);

436 +}

437 +

438 +/* Calculates the similarity between two fingerprints as the size of the

439 + * intersection of their multisets, including repeated elements. See

440 + * `struct fingerprint` for an explanation of the fingerprint representation.

441 + * The similarity between "cat mat" and "father rather" is 2 because "at" is

442 + * present twice in both strings while the similarity between "tim" and "mit"

443 + * is 0.

444 + */

445 +static int fingerprint_similarity(struct fingerprint *a, struct fingerprint *b)

446 +{

447 + int intersection = 0;

448 + struct hashmap_iter iter;

449 + const struct fingerprint_entry *entry_a, *entry_b;

450 +

451 + hashmap_iter_init(&b->map, &iter);

452 +

453 + while ((entry_b = hashmap_iter_next(&iter))) {

454 + if ((entry_a = hashmap_get(&a->map, entry_b, NULL))) {

455 + intersection += entry_a->count < entry_b->count ?

456 + entry_a->count : entry_b->count;

457 + }

458 + }

459 + return intersection;

460 +}

461 +

462 +/* Subtracts byte-pair elements in B from A, modifying A in place.

463 + */

464 +static void fingerprint_subtract(struct fingerprint *a, struct fingerprint *b)

465 +{

466 + struct hashmap_iter iter;

467 + struct fingerprint_entry *entry_a;

468 + const struct fingerprint_entry *entry_b;

469 +

470 + hashmap_iter_init(&b->map, &iter);

471 +

472 + while ((entry_b = hashmap_iter_next(&iter))) {

473 + if ((entry_a = hashmap_get(&a->map, entry_b, NULL))) {

474 + if (entry_a->count <= entry_b->count)

475 + hashmap_remove(&a->map, entry_b, NULL);

476 + else

477 + entry_a->count -= entry_b->count;

478 + }

479 + }

480 +}

481 +

482 +/* Calculate fingerprints for a series of lines.

483 + * Puts the fingerprints in the fingerprints array, which must have been

484 + * preallocated to allow storing line_count elements.

485 + */

486 +static void get_line_fingerprints(struct fingerprint *fingerprints,

487 + const char *content, const int *line_starts,

488 + long first_line, long line_count)

489 +{

490 + int i;

491 + const char *linestart, *lineend;

492 +

493 + line_starts += first_line;

494 + for (i = 0; i < line_count; ++i) {

495 + linestart = content + line_starts[i];

496 + lineend = content + line_starts[i + 1];

497 + get_fingerprint(fingerprints + i, linestart, lineend);

498 + }

499 +}

500 +

501 +static void free_line_fingerprints(struct fingerprint *fingerprints,

502 + int nr_fingerprints)

503 +{

504 + int i;

505 +

506 + for (i = 0; i < nr_fingerprints; i++)

507 + free_fingerprint(&fingerprints[i]);

508 +}

509 +

510 +/* This contains the data necessary to linearly map a line number in one half

511 + * of a diff chunk to the line in the other half of the diff chunk that is

512 + * closest in terms of its position as a fraction of the length of the chunk.

513 + */

514 +struct line_number_mapping {

515 + int destination_start, destination_length,

516 + source_start, source_length;

517 +};

518 +

519 +/* Given a line number in one range, offset and scale it to map it onto the

520 + * other range.

521 + * Essentially this mapping is a simple linear equation but the calculation is

522 + * more complicated to allow performing it with integer operations.

523 + * Another complication is that if a line could map onto many lines in the

524 + * destination range then we want to choose the line at the center of those

525 + * possibilities.

526 + * Example: if the chunk is 2 lines long in A and 10 lines long in B then the

527 + * first 5 lines in B will map onto the first line in the A chunk, while the

528 + * last 5 lines will all map onto the second line in the A chunk.

529 + * Example: if the chunk is 10 lines long in A and 2 lines long in B then line

530 + * 0 in B will map onto line 2 in A, and line 1 in B will map onto line 7 in A.

531 + */

532 +static int map_line_number(int line_number,

533 + const struct line_number_mapping *mapping)

534 +{

535 + return ((line_number - mapping->source_start) * 2 + 1) *

536 + mapping->destination_length /

537 + (mapping->source_length * 2) +

538 + mapping->destination_start;

539 +}

540 +

541 +/* Get a pointer to the element storing the similarity between a line in A

542 + * and a line in B.

543 + *

544 + * The similarities are stored in a 2-dimensional array. Each "row" in the

545 + * array contains the similarities for a line in B. The similarities stored in

546 + * a row are the similarities between the line in B and the nearby lines in A.

547 + * To keep the length of each row the same, it is padded out with values of -1

548 + * where the search range extends beyond the lines in A.

549 + * For example, if max_search_distance_a is 2 and the two sides of a diff chunk

550 + * look like this:

551 + * a | m

552 + * b | n

553 + * c | o

554 + * d | p

555 + * e | q

556 + * Then the similarity array will contain:

557 + * [-1, -1, am, bm, cm,

558 + * -1, an, bn, cn, dn,

559 + * ao, bo, co, do, eo,

560 + * bp, cp, dp, ep, -1,

561 + * cq, dq, eq, -1, -1]

562 + * Where similarities are denoted either by -1 for invalid, or the

563 + * concatenation of the two lines in the diff being compared.

564 + *

565 + * \param similarities array of similarities between lines in A and B

566 + * \param line_a the index of the line in A, in the same frame of reference as

567 + * closest_line_a.

568 + * \param local_line_b the index of the line in B, relative to the first line

569 + * in B that similarities represents.

570 + * \param closest_line_a the index of the line in A that is deemed to be

571 + * closest to local_line_b. This must be in the same

572 + * frame of reference as line_a. This value defines

573 + * where similarities is centered for the line in B.

574 + * \param max_search_distance_a maximum distance in lines from the closest line

575 + * in A for other lines in A for which

576 + * similarities may be calculated.

577 + */

578 +static int *get_similarity(int *similarities,

579 + int line_a, int local_line_b,

580 + int closest_line_a, int max_search_distance_a)

581 +{

582 + assert(abs(line_a - closest_line_a) <=

583 + max_search_distance_a);

584 + return similarities + line_a - closest_line_a +

585 + max_search_distance_a +

586 + local_line_b * (max_search_distance_a * 2 + 1);

587 +}

588 +

589 +#define CERTAIN_NOTHING_MATCHES -2

590 +#define CERTAINTY_NOT_CALCULATED -1

591 +

592 +/* Given a line in B, first calculate its similarities with nearby lines in A

593 + * if not already calculated, then identify the most similar and second most

594 + * similar lines. The "certainty" is calculated based on those two

595 + * similarities.

596 + *

597 + * \param start_a the index of the first line of the chunk in A

598 + * \param length_a the length in lines of the chunk in A

599 + * \param local_line_b the index of the line in B, relative to the first line

600 + * in the chunk.

601 + * \param fingerprints_a array of fingerprints for the chunk in A

602 + * \param fingerprints_b array of fingerprints for the chunk in B

603 + * \param similarities 2-dimensional array of similarities between lines in A

604 + * and B. See get_similarity() for more details.

605 + * \param certainties array of values indicating how strongly a line in B is

606 + * matched with some line in A.

607 + * \param second_best_result array of absolute indices in A for the second

608 + * closest match of a line in B.

609 + * \param result array of absolute indices in A for the closest match of a line

610 + * in B.

611 + * \param max_search_distance_a maximum distance in lines from the closest line

612 + * in A for other lines in A for which

613 + * similarities may be calculated.

614 + * \param map_line_number_in_b_to_a parameter to map_line_number().

615 + */

616 +static void find_best_line_matches(

617 + int start_a,

618 + int length_a,

619 + int start_b,

620 + int local_line_b,

621 + struct fingerprint *fingerprints_a,

622 + struct fingerprint *fingerprints_b,

623 + int *similarities,

624 + int *certainties,

625 + int *second_best_result,

626 + int *result,

627 + const int max_search_distance_a,

628 + const struct line_number_mapping *map_line_number_in_b_to_a)

629 +{

630 +

631 + int i, search_start, search_end, closest_local_line_a, *similarity,

632 + best_similarity = 0, second_best_similarity = 0,

633 + best_similarity_index = 0, second_best_similarity_index = 0;

634 +

635 + /* certainty has already been calculated so no need to redo the work */

636 + if (certainties[local_line_b] != CERTAINTY_NOT_CALCULATED)

637 + return;

638 +

639 + closest_local_line_a = map_line_number(

640 + local_line_b + start_b, map_line_number_in_b_to_a) - start_a;

641 +

642 + search_start = closest_local_line_a - max_search_distance_a;

643 + if (search_start < 0)

644 + search_start = 0;

645 +

646 + search_end = closest_local_line_a + max_search_distance_a + 1;

647 + if (search_end > length_a)

648 + search_end = length_a;

649 +

650 + for (i = search_start; i < search_end; ++i) {

651 + similarity = get_similarity(similarities,

652 + i, local_line_b,

653 + closest_local_line_a,

654 + max_search_distance_a);

655 + if (*similarity == -1) {

656 + /* This value will never exceed 10 but assert just in

657 + * case

658 + */

659 + assert(abs(i - closest_local_line_a) < 1000);

660 + /* scale the similarity by (1000 - distance from

661 + * closest line) to act as a tie break between lines

662 + * that otherwise are equally similar.

663 + */

664 + *similarity = fingerprint_similarity(

665 + fingerprints_b + local_line_b,

666 + fingerprints_a + i) *

667 + (1000 - abs(i - closest_local_line_a));

668 + }

669 + if (*similarity > best_similarity) {

670 + second_best_similarity = best_similarity;

671 + second_best_similarity_index = best_similarity_index;

672 + best_similarity = *similarity;

673 + best_similarity_index = i;

674 + } else if (*similarity > second_best_similarity) {

675 + second_best_similarity = *similarity;

676 + second_best_similarity_index = i;

677 + }

678 + }

679 +

680 + if (best_similarity == 0) {

681 + /* this line definitely doesn't match with anything. Mark it

682 + * with this special value so it doesn't get invalidated and

683 + * won't be recalculated.

684 + */

685 + certainties[local_line_b] = CERTAIN_NOTHING_MATCHES;

686 + result[local_line_b] = -1;

687 + } else {

688 + /* Calculate the certainty with which this line matches.

689 + * If the line matches well with two lines then that reduces

690 + * the certainty. However we still want to prioritise matching

691 + * a line that matches very well with two lines over matching a

692 + * line that matches poorly with one line, hence doubling

693 + * best_similarity.

694 + * This means that if we have

695 + * line X that matches only one line with a score of 3,

696 + * line Y that matches two lines equally with a score of 5,

697 + * and line Z that matches only one line with a score or 2,

698 + * then the lines in order of certainty are X, Y, Z.

699 + */

700 + certainties[local_line_b] = best_similarity * 2 -

701 + second_best_similarity;

702 +

703 + /* We keep both the best and second best results to allow us to

704 + * check at a later stage of the matching process whether the

705 + * result needs to be invalidated.

706 + */

707 + result[local_line_b] = start_a + best_similarity_index;

708 + second_best_result[local_line_b] =

709 + start_a + second_best_similarity_index;

710 + }

711 +}

712 +

713 +/*

714 + * This finds the line that we can match with the most confidence, and

715 + * uses it as a partition. It then calls itself on the lines on either side of

716 + * that partition. In this way we avoid lines appearing out of order, and

717 + * retain a sensible line ordering.

718 + * \param start_a index of the first line in A with which lines in B may be

719 + * compared.

720 + * \param start_b index of the first line in B for which matching should be

721 + * done.

722 + * \param length_a number of lines in A with which lines in B may be compared.

723 + * \param length_b number of lines in B for which matching should be done.

724 + * \param fingerprints_a mutable array of fingerprints in A. The first element

725 + * corresponds to the line at start_a.

726 + * \param fingerprints_b array of fingerprints in B. The first element

727 + * corresponds to the line at start_b.

728 + * \param similarities 2-dimensional array of similarities between lines in A

729 + * and B. See get_similarity() for more details.

730 + * \param certainties array of values indicating how strongly a line in B is

731 + * matched with some line in A.

732 + * \param second_best_result array of absolute indices in A for the second

733 + * closest match of a line in B.

734 + * \param result array of absolute indices in A for the closest match of a line

735 + * in B.

736 + * \param max_search_distance_a maximum distance in lines from the closest line

737 + * in A for other lines in A for which

738 + * similarities may be calculated.

739 + * \param max_search_distance_b an upper bound on the greatest possible

740 + * distance between lines in B such that they will

741 + * both be compared with the same line in A

742 + * according to max_search_distance_a.

743 + * \param map_line_number_in_b_to_a parameter to map_line_number().

744 + */

745 +static void fuzzy_find_matching_lines_recurse(

746 + int start_a, int start_b,

747 + int length_a, int length_b,

748 + struct fingerprint *fingerprints_a,

749 + struct fingerprint *fingerprints_b,

750 + int *similarities,

751 + int *certainties,

752 + int *second_best_result,

753 + int *result,

754 + int max_search_distance_a,

755 + int max_search_distance_b,

756 + const struct line_number_mapping *map_line_number_in_b_to_a)

757 +{

758 + int i, invalidate_min, invalidate_max, offset_b,

759 + second_half_start_a, second_half_start_b,

760 + second_half_length_a, second_half_length_b,

761 + most_certain_line_a, most_certain_local_line_b = -1,

762 + most_certain_line_certainty = -1,

763 + closest_local_line_a;

764 +

765 + for (i = 0; i < length_b; ++i) {

766 + find_best_line_matches(start_a,

767 + length_a,

768 + start_b,

769 + i,

770 + fingerprints_a,

771 + fingerprints_b,

772 + similarities,

773 + certainties,

774 + second_best_result,

775 + result,

776 + max_search_distance_a,

777 + map_line_number_in_b_to_a);

778 +

779 + if (certainties[i] > most_certain_line_certainty) {

780 + most_certain_line_certainty = certainties[i];

781 + most_certain_local_line_b = i;

782 + }

783 + }

784 +

785 + /* No matches. */

786 + if (most_certain_local_line_b == -1)

787 + return;

788 +

789 + most_certain_line_a = result[most_certain_local_line_b];

790 +

791 + /*

792 + * Subtract the most certain line's fingerprint in B from the matched

793 + * fingerprint in A. This means that other lines in B can't also match

794 + * the same parts of the line in A.

795 + */

796 + fingerprint_subtract(fingerprints_a + most_certain_line_a - start_a,

797 + fingerprints_b + most_certain_local_line_b);

798 +

799 + /* Invalidate results that may be affected by the choice of most

800 + * certain line.

801 + */

802 + invalidate_min = most_certain_local_line_b - max_search_distance_b;

803 + invalidate_max = most_certain_local_line_b + max_search_distance_b + 1;

804 + if (invalidate_min < 0)

805 + invalidate_min = 0;

806 + if (invalidate_max > length_b)

807 + invalidate_max = length_b;

808 +

809 + /* As the fingerprint in A has changed, discard previously calculated

810 + * similarity values with that fingerprint.

811 + */

812 + for (i = invalidate_min; i < invalidate_max; ++i) {

813 + closest_local_line_a = map_line_number(

814 + i + start_b, map_line_number_in_b_to_a) - start_a;

815 +

816 + /* Check that the lines in A and B are close enough that there

817 + * is a similarity value for them.

818 + */

819 + if (abs(most_certain_line_a - start_a - closest_local_line_a) >

820 + max_search_distance_a) {

821 + continue;

822 + }

823 +

824 + *get_similarity(similarities, most_certain_line_a - start_a,

825 + i, closest_local_line_a,

826 + max_search_distance_a) = -1;

827 + }

828 +

829 + /* More invalidating of results that may be affected by the choice of

830 + * most certain line.

831 + * Discard the matches for lines in B that are currently matched with a

832 + * line in A such that their ordering contradicts the ordering imposed

833 + * by the choice of most certain line.

834 + */

835 + for (i = most_certain_local_line_b - 1; i >= invalidate_min; --i) {

836 + /* In this loop we discard results for lines in B that are

837 + * before most-certain-line-B but are matched with a line in A

838 + * that is after most-certain-line-A.

839 + */

840 + if (certainties[i] >= 0 &&

841 + (result[i] >= most_certain_line_a ||

842 + second_best_result[i] >= most_certain_line_a)) {

843 + certainties[i] = CERTAINTY_NOT_CALCULATED;

844 + }

845 + }

846 + for (i = most_certain_local_line_b + 1; i < invalidate_max; ++i) {

847 + /* In this loop we discard results for lines in B that are

848 + * after most-certain-line-B but are matched with a line in A

849 + * that is before most-certain-line-A.

850 + */

851 + if (certainties[i] >= 0 &&

852 + (result[i] <= most_certain_line_a ||

853 + second_best_result[i] <= most_certain_line_a)) {

854 + certainties[i] = CERTAINTY_NOT_CALCULATED;

855 + }

856 + }

857 +

858 + /* Repeat the matching process for lines before the most certain line.

859 + */

860 + if (most_certain_local_line_b > 0) {

861 + fuzzy_find_matching_lines_recurse(

862 + start_a, start_b,

863 + most_certain_line_a + 1 - start_a,

864 + most_certain_local_line_b,

865 + fingerprints_a, fingerprints_b, similarities,

866 + certainties, second_best_result, result,

867 + max_search_distance_a,

868 + max_search_distance_b,

869 + map_line_number_in_b_to_a);

870 + }

871 + /* Repeat the matching process for lines after the most certain line.

872 + */

873 + if (most_certain_local_line_b + 1 < length_b) {

874 + second_half_start_a = most_certain_line_a;

875 + offset_b = most_certain_local_line_b + 1;

876 + second_half_start_b = start_b + offset_b;

877 + second_half_length_a =

878 + length_a + start_a - second_half_start_a;

879 + second_half_length_b =

880 + length_b + start_b - second_half_start_b;

881 + fuzzy_find_matching_lines_recurse(

882 + second_half_start_a, second_half_start_b,

883 + second_half_length_a, second_half_length_b,

884 + fingerprints_a + second_half_start_a - start_a,

885 + fingerprints_b + offset_b,

886 + similarities +

887 + offset_b * (max_search_distance_a * 2 + 1),

888 + certainties + offset_b,

889 + second_best_result + offset_b, result + offset_b,

890 + max_search_distance_a,

891 + max_search_distance_b,

892 + map_line_number_in_b_to_a);

893 + }

894 +}

895 +

896 +/* Find the lines in the parent line range that most closely match the lines in

897 + * the target line range. This is accomplished by matching fingerprints in each

898 + * blame_origin, and choosing the best matches that preserve the line ordering.

899 + * See struct fingerprint for details of fingerprint matching, and

900 + * fuzzy_find_matching_lines_recurse for details of preserving line ordering.

901 + *

902 + * The performance is believed to be O(n log n) in the typical case and O(n^2)

903 + * in a pathological case, where n is the number of lines in the target range.

904 + */

905 +static int *fuzzy_find_matching_lines(struct blame_origin *parent,

906 + struct blame_origin *target,

907 + int tlno, int parent_slno, int same,

908 + int parent_len)

909 +{

910 + /* We use the terminology "A" for the left hand side of the diff AKA

911 + * parent, and "B" for the right hand side of the diff AKA target. */

912 + int start_a = parent_slno;

913 + int length_a = parent_len;

914 + int start_b = tlno;

915 + int length_b = same - tlno;

916 +

917 + struct line_number_mapping map_line_number_in_b_to_a = {

918 + start_a, length_a, start_b, length_b

919 + };

920 +

921 + struct fingerprint *fingerprints_a = parent->fingerprints;

922 + struct fingerprint *fingerprints_b = target->fingerprints;

923 +

924 + int i, *result, *second_best_result,

925 + *certainties, *similarities, similarity_count;

926 +

927 + /*

928 + * max_search_distance_a means that given a line in B, compare it to

929 + * the line in A that is closest to its position, and the lines in A

930 + * that are no greater than max_search_distance_a lines away from the

931 + * closest line in A.

932 + *

933 + * max_search_distance_b is an upper bound on the greatest possible

934 + * distance between lines in B such that they will both be compared

935 + * with the same line in A according to max_search_distance_a.

936 + */

937 + int max_search_distance_a = 10, max_search_distance_b;

938 +

939 + if (length_a <= 0)

940 + return NULL;

941 +

942 + if (max_search_distance_a >= length_a)

943 + max_search_distance_a = length_a ? length_a - 1 : 0;

944 +

945 + max_search_distance_b = ((2 * max_search_distance_a + 1) * length_b

946 + - 1) / length_a;

947 +

948 + result = xcalloc(sizeof(int), length_b);

949 + second_best_result = xcalloc(sizeof(int), length_b);

950 + certainties = xcalloc(sizeof(int), length_b);

951 +

952 + /* See get_similarity() for details of similarities. */

953 + similarity_count = length_b * (max_search_distance_a * 2 + 1);

954 + similarities = xcalloc(sizeof(int), similarity_count);

955 +

956 + for (i = 0; i < length_b; ++i) {

957 + result[i] = -1;

958 + second_best_result[i] = -1;

959 + certainties[i] = CERTAINTY_NOT_CALCULATED;

960 + }

961 +

962 + for (i = 0; i < similarity_count; ++i)

963 + similarities[i] = -1;

964 +

965 + fuzzy_find_matching_lines_recurse(start_a, start_b,

966 + length_a, length_b,

967 + fingerprints_a + start_a,

968 + fingerprints_b + start_b,

969 + similarities,

970 + certainties,

971 + second_best_result,

972 + result,

973 + max_search_distance_a,

974 + max_search_distance_b,

975 + &map_line_number_in_b_to_a);

976 +

977 + free(similarities);

978 + free(certainties);

979 + free(second_best_result);

980 +

981 + return result;

982 +}

983 +

984 static void fill_origin_fingerprints(struct blame_origin *o, mmfile_t *file)

985 {

986 int *line_starts;

t/t8014-blame-ignore-fuzzy.sh new

+440

@@ -0,0 +1,440 @@

1 +#!/bin/sh

2 +

3 +test_description='git blame ignore fuzzy heuristic'

4 +. ./test-lib.sh

5 +

6 +# short circuit until blame has the fuzzy capabilities

7 +test_done

8 +

9 +pick_author='s/^[0-9a-f^]* *($[^ ]*$ .*/\1/'

10 +

11 +# Each test is composed of 4 variables:

12 +# titleN - the test name

13 +# aN - the initial content

14 +# bN - the final content

15 +# expectedN - the line numbers from aN that we expect git blame

16 +# on bN to identify, or "Final" if bN itself should

17 +# be identified as the origin of that line.

18 +

19 +# We start at test 2 because setup will show as test 1

20 +title2="Regression test for partially overlapping search ranges"

21 +cat <<EOF >a2

22 +1

23 +2

24 +3

25 +abcdef

26 +5

27 +6

28 +7

29 +ijkl

30 +9

31 +10

32 +11

33 +pqrs

34 +13

35 +14

36 +15

37 +wxyz

38 +17

39 +18

40 +19

41 +EOF

42 +cat <<EOF >b2

43 +abcde

44 +ijk

45 +pqr

46 +wxy

47 +EOF

48 +cat <<EOF >expected2

49 +4

50 +8

51 +12

52 +16

53 +EOF

54 +

55 +title3="Combine 3 lines into 2"

56 +cat <<EOF >a3

57 +if ((maxgrow==0) ||

58 + ( single_line_field && (field->dcols < maxgrow)) ||

59 + (!single_line_field && (field->drows < maxgrow)))

60 +EOF

61 +cat <<EOF >b3

62 +if ((maxgrow == 0) || (single_line_field && (field->dcols < maxgrow)) ||

63 + (!single_line_field && (field->drows < maxgrow))) {

64 +EOF

65 +cat <<EOF >expected3

66 +2

67 +3

68 +EOF

69 +

70 +title4="Add curly brackets"

71 +cat <<EOF >a4

72 + if (rows) *rows = field->rows;

73 + if (cols) *cols = field->cols;

74 + if (frow) *frow = field->frow;

75 + if (fcol) *fcol = field->fcol;

76 +EOF

77 +cat <<EOF >b4

78 + if (rows) {

79 + *rows = field->rows;

80 + }

81 + if (cols) {

82 + *cols = field->cols;

83 + }

84 + if (frow) {

85 + *frow = field->frow;

86 + }

87 + if (fcol) {

88 + *fcol = field->fcol;

89 + }

90 +EOF

91 +cat <<EOF >expected4

92 +1

93 +1

94 +Final

95 +2

96 +2

97 +Final

98 +3

99 +3

100 +Final

101 +4

102 +4

103 +Final

104 +EOF

105 +

106 +

107 +title5="Combine many lines and change case"

108 +cat <<EOF >a5

109 +for(row=0,pBuffer=field->buf;

110 + row<height;

111 + row++,pBuffer+=width )

112 +{

113 + if ((len = (int)( After_End_Of_Data( pBuffer, width ) - pBuffer )) > 0)

114 + {

115 + wmove( win, row, 0 );

116 + waddnstr( win, pBuffer, len );

117 +EOF

118 +cat <<EOF >b5

119 +for (Row = 0, PBuffer = field->buf; Row < Height; Row++, PBuffer += Width) {

120 + if ((Len = (int)(afterEndOfData(PBuffer, Width) - PBuffer)) > 0) {

121 + wmove(win, Row, 0);

122 + waddnstr(win, PBuffer, Len);

123 +EOF

124 +cat <<EOF >expected5

125 +1

126 +5

127 +7

128 +8

129 +EOF

130 +

131 +title6="Rename and combine lines"

132 +cat <<EOF >a6

133 +bool need_visual_update = ((form != (FORM *)0) &&

134 + (form->status & _POSTED) &&

135 + (form->current==field));

136 +

137 +if (need_visual_update)

138 + Synchronize_Buffer(form);

139 +

140 +if (single_line_field)

141 +{

142 + growth = field->cols * amount;

143 + if (field->maxgrow)

144 + growth = Minimum(field->maxgrow - field->dcols,growth);

145 + field->dcols += growth;

146 + if (field->dcols == field->maxgrow)

147 +EOF

148 +cat <<EOF >b6

149 +bool NeedVisualUpdate = ((Form != (FORM *)0) && (Form->status & _POSTED) &&

150 + (Form->current == field));

151 +

152 +if (NeedVisualUpdate) {

153 + synchronizeBuffer(Form);

154 +}

155 +

156 +if (SingleLineField) {

157 + Growth = field->cols * amount;

158 + if (field->maxgrow) {

159 + Growth = Minimum(field->maxgrow - field->dcols, Growth);

160 + }

161 + field->dcols += Growth;

162 + if (field->dcols == field->maxgrow) {

163 +EOF

164 +cat <<EOF >expected6

165 +1

166 +3

167 +4

168 +5

169 +6

170 +Final

171 +7

172 +8

173 +10

174 +11

175 +12

176 +Final

177 +13

178 +14

179 +EOF

180 +

181 +# Both lines match identically so position must be used to tie-break.

182 +title7="Same line twice"

183 +cat <<EOF >a7

184 +abc

185 +abc

186 +EOF

187 +cat <<EOF >b7

188 +abcd

189 +abcd

190 +EOF

191 +cat <<EOF >expected7

192 +1

193 +2

194 +EOF

195 +

196 +title8="Enforce line order"

197 +cat <<EOF >a8

198 +abcdef

199 +ghijkl

200 +ab

201 +EOF

202 +cat <<EOF >b8

203 +ghijk

204 +abcd

205 +EOF

206 +cat <<EOF >expected8

207 +2

208 +3

209 +EOF

210 +

211 +title9="Expand lines and rename variables"

212 +cat <<EOF >a9

213 +int myFunction(int ArgumentOne, Thing *ArgTwo, Blah XuglyBug) {

214 + Squiggle FabulousResult = squargle(ArgumentOne, *ArgTwo,

215 + XuglyBug) + EwwwGlobalWithAReallyLongNameYepTooLong;

216 + return FabulousResult * 42;

217 +}

218 +EOF

219 +cat <<EOF >b9

220 +int myFunction(int argument_one, Thing *arg_asdfgh,

221 + Blah xugly_bug) {

222 + Squiggle fabulous_result = squargle(argument_one,

223 + *arg_asdfgh, xugly_bug)

224 + + g_ewww_global_with_a_really_long_name_yep_too_long;

225 + return fabulous_result * 42;

226 +}

227 +EOF

228 +cat <<EOF >expected9

229 +1

230 +1

231 +2

232 +3

233 +3

234 +4

235 +5

236 +EOF

237 +

238 +title10="Two close matches versus one less close match"

239 +cat <<EOF >a10

240 +abcdef

241 +abcdef

242 +ghijkl

243 +EOF

244 +cat <<EOF >b10

245 +gh

246 +abcdefx

247 +EOF

248 +cat <<EOF >expected10

249 +Final

250 +2

251 +EOF

252 +

253 +# The first line of b matches best with the last line of a, but the overall

254 +# match is better if we match it with the the first line of a.

255 +title11="Piggy in the middle"

256 +cat <<EOF >a11

257 +abcdefg

258 +ijklmn

259 +abcdefgh

260 +EOF

261 +cat <<EOF >b11

262 +abcdefghx

263 +ijklm

264 +EOF

265 +cat <<EOF >expected11

266 +1

267 +2

268 +EOF

269 +

270 +title12="No trailing newline"

271 +printf "abc\ndef" >a12

272 +printf "abx\nstu" >b12

273 +cat <<EOF >expected12

274 +1

275 +Final

276 +EOF

277 +

278 +title13="Reorder includes"

279 +cat <<EOF >a13

280 +#include "c.h"

281 +#include "b.h"

282 +#include "a.h"

283 +#include "e.h"

284 +#include "d.h"

285 +EOF

286 +cat <<EOF >b13

287 +#include "a.h"

288 +#include "b.h"

289 +#include "c.h"

290 +#include "d.h"

291 +#include "e.h"

292 +EOF

293 +cat <<EOF >expected13

294 +3

295 +2

296 +1

297 +5

298 +4

299 +EOF

300 +

301 +last_test=13

302 +

303 +test_expect_success setup '

304 + { for i in $(test_seq 2 $last_test)

305 + do

306 + # Append each line in a separate commit to make it easy to

307 + # check which original line the blame output relates to.

308 +

309 + line_count=0 &&

310 + { while IFS= read line

311 + do

312 + line_count=$((line_count+1)) &&

313 + echo "$line" >>"$i" &&

314 + git add "$i" &&

315 + test_tick &&

316 + GIT_AUTHOR_NAME="$line_count" git commit -m "$line_count"

317 + done } <"a$i"

318 + done } &&

319 +

320 + { for i in $(test_seq 2 $last_test)

321 + do

322 + # Overwrite the files with the final content.

323 + cp b$i $i &&

324 + git add $i

325 + done } &&

326 + test_tick &&

327 +

328 + # Commit the final content all at once so it can all be

329 + # referred to with the same commit ID.

330 + GIT_AUTHOR_NAME=Final git commit -m Final &&

331 +

332 + IGNOREME=$(git rev-parse HEAD)

333 +'

334 +

335 +for i in $(test_seq 2 $last_test); do

336 + eval title="\$title$i"

337 + test_expect_success "$title" \

338 + "git blame -M9 --ignore-rev $IGNOREME $i >output &&

339 + sed -e \"$pick_author\" output >actual &&

340 + test_cmp expected$i actual"

341 +done

342 +

343 +# This invoked a null pointer dereference when the chunk callback was called

344 +# with a zero length parent chunk and there were no more suspects.

345 +test_expect_success 'Diff chunks with no suspects' '

346 + test_write_lines xy1 A B C xy1 >file &&

347 + git add file &&

348 + test_tick &&

349 + GIT_AUTHOR_NAME=1 git commit -m 1 &&

350 +

351 + test_write_lines xy2 A B xy2 C xy2 >file &&

352 + git add file &&

353 + test_tick &&

354 + GIT_AUTHOR_NAME=2 git commit -m 2 &&

355 + REV_2=$(git rev-parse HEAD) &&

356 +

357 + test_write_lines xy3 A >file &&

358 + git add file &&

359 + test_tick &&

360 + GIT_AUTHOR_NAME=3 git commit -m 3 &&

361 + REV_3=$(git rev-parse HEAD) &&

362 +

363 + test_write_lines 1 1 >expected &&

364 +

365 + git blame --ignore-rev $REV_2 --ignore-rev $REV_3 file >output &&

366 + sed -e "$pick_author" output >actual &&

367 +

368 + test_cmp expected actual

369 + '

370 +

371 +test_expect_success 'position matching' '

372 + test_write_lines abc def >file2 &&

373 + git add file2 &&

374 + test_tick &&

375 + GIT_AUTHOR_NAME=1 git commit -m 1 &&

376 +

377 + test_write_lines abc def abc def >file2 &&

378 + git add file2 &&

379 + test_tick &&

380 + GIT_AUTHOR_NAME=2 git commit -m 2 &&

381 +

382 + test_write_lines abcx defx abcx defx >file2 &&

383 + git add file2 &&

384 + test_tick &&

385 + GIT_AUTHOR_NAME=3 git commit -m 3 &&

386 + REV_3=$(git rev-parse HEAD) &&

387 +

388 + test_write_lines abcy defy abcx defx >file2 &&

389 + git add file2 &&

390 + test_tick &&

391 + GIT_AUTHOR_NAME=4 git commit -m 4 &&

392 + REV_4=$(git rev-parse HEAD) &&

393 +

394 + test_write_lines 1 1 2 2 >expected &&

395 +

396 + git blame --ignore-rev $REV_3 --ignore-rev $REV_4 file2 >output &&

397 + sed -e "$pick_author" output >actual &&

398 +

399 + test_cmp expected actual

400 + '

401 +

402 +# This fails if each blame entry is processed independently instead of

403 +# processing each diff change in full.

404 +test_expect_success 'preserve order' '

405 + test_write_lines bcde >file3 &&

406 + git add file3 &&

407 + test_tick &&

408 + GIT_AUTHOR_NAME=1 git commit -m 1 &&

409 +

410 + test_write_lines bcde fghij >file3 &&

411 + git add file3 &&

412 + test_tick &&

413 + GIT_AUTHOR_NAME=2 git commit -m 2 &&

414 +

415 + test_write_lines bcde fghij abcd >file3 &&

416 + git add file3 &&

417 + test_tick &&

418 + GIT_AUTHOR_NAME=3 git commit -m 3 &&

419 +

420 + test_write_lines abcdx fghijx bcdex >file3 &&

421 + git add file3 &&

422 + test_tick &&

423 + GIT_AUTHOR_NAME=4 git commit -m 4 &&

424 + REV_4=$(git rev-parse HEAD) &&

425 +

426 + test_write_lines abcdx fghijy bcdex >file3 &&

427 + git add file3 &&

428 + test_tick &&

429 + GIT_AUTHOR_NAME=5 git commit -m 5 &&

430 + REV_5=$(git rev-parse HEAD) &&

431 +

432 + test_write_lines 1 2 3 >expected &&

433 +

434 + git blame --ignore-rev $REV_4 --ignore-rev $REV_5 file3 >output &&

435 + sed -e "$pick_author" output >actual &&

436 +

437 + test_cmp expected actual

438 + '

439 +

440 +test_done