df5be01669 · matheusbsbs/matheus-bagliano

@@ -466,13 +466,13 @@ The performance of git-filter-branch is glacially slow; its design makes it

466 impossible for a backward-compatible implementation to ever be fast:

467

468 * In editing files, git-filter-branch by design checks out each and

469 -every commit as it existed in the original repo. If your repo has 10\^5

470 -files and 10\^5 commits, but each commit only modifies 5 files, then

471 -git-filter-branch will make you do 10\^10 modifications, despite only

472 -having (at most) 5*10^5 unique blobs.

469 + every commit as it existed in the original repo. If your repo has

470 + 10\^5 files and 10\^5 commits, but each commit only modifies 5

471 + files, then git-filter-branch will make you do 10\^10 modifications,

472 + despite only having (at most) 5*10^5 unique blobs.

473

474 * If you try and cheat and try to make git-filter-branch only work on

475 -files modified in a commit, then two things happen

475 + files modified in a commit, then two things happen

476

477 ** you run into problems with deletions whenever the user is simply

478 trying to rename files (because attempting to delete files that

@@ -481,39 +481,41 @@ files modified in a commit, then two things happen

481 user-provided shell)

482

483 ** even if you succeed at the map-deletes-for-renames chicanery, you

484 - still technically violate backward compatibility because users are

485 - allowed to filter files in ways that depend upon topology of

486 - commits instead of filtering solely based on file contents or names

487 - (though this has not been observed in the wild).

484 + still technically violate backward compatibility because users

485 + are allowed to filter files in ways that depend upon topology of

486 + commits instead of filtering solely based on file contents or

487 + names (though this has not been observed in the wild).

488

489 * Even if you don't need to edit files but only want to e.g. rename or

490 -remove some and thus can avoid checking out each file (i.e. you can use

491 ---index-filter), you still are passing shell snippets for your filters.

492 -This means that for every commit, you have to have a prepared git repo

493 -where those filters can be run. That's a significant setup.

494 -

495 -* Further, several additional files are created or updated per commit by

496 -git-filter-branch. Some of these are for supporting the convenience

497 -functions provided by git-filter-branch (such as map()), while others

498 -are for keeping track of internal state (but could have also been

499 -accessed by user filters; one of git-filter-branch's regression tests

500 -does so). This essentially amounts to using the filesystem as an IPC

501 -mechanism between git-filter-branch and the user-provided filters.

502 -Disks tend to be a slow IPC mechanism, and writing these files also

503 -effectively represents a forced synchronization point between separate

504 -processes that we hit with every commit.

490 + remove some and thus can avoid checking out each file (i.e. you can

491 + use --index-filter), you still are passing shell snippets for your

492 + filters. This means that for every commit, you have to have a

493 + prepared git repo where those filters can be run. That's a

494 + significant setup.

495 +

496 +* Further, several additional files are created or updated per commit

497 + by git-filter-branch. Some of these are for supporting the

498 + convenience functions provided by git-filter-branch (such as map()),

499 + while others are for keeping track of internal state (but could have

500 + also been accessed by user filters; one of git-filter-branch's

501 + regression tests does so). This essentially amounts to using the

502 + filesystem as an IPC mechanism between git-filter-branch and the

503 + user-provided filters. Disks tend to be a slow IPC mechanism, and

504 + writing these files also effectively represents a forced

505 + synchronization point between separate processes that we hit with

506 + every commit.

507

508 * The user-provided shell commands will likely involve a pipeline of

507 -commands, resulting in the creation of many processes per commit.

508 -Creating and running another process takes a widely varying amount of

509 -time between operating systems, but on any platform it is very slow

510 -relative to invoking a function.

509 + commands, resulting in the creation of many processes per commit.

510 + Creating and running another process takes a widely varying amount

511 + of time between operating systems, but on any platform it is very

512 + slow relative to invoking a function.

513

514 * git-filter-branch itself is written in shell, which is kind of slow.

513 -This is the one performance issue that could be backward-compatibly

514 -fixed, but compared to the above problems that are intrinsic to the

515 -design of git-filter-branch, the language of the tool itself is a

516 -relatively minor issue.

515 + This is the one performance issue that could be backward-compatibly

516 + fixed, but compared to the above problems that are intrinsic to the

517 + design of git-filter-branch, the language of the tool itself is a

518 + relatively minor issue.

519

520 ** Side note: Unfortunately, people tend to fixate on the

521 written-in-shell aspect and periodically ask if git-filter-branch

@@ -546,51 +548,55 @@ easily corrupt repos or end up with a mess worse than what you started

548 with:

549

550 * Someone can have a set of "working and tested filters" which they

549 -document or provide to a coworker, who then runs them on a different OS

550 -where the same commands are not working/tested (some examples in the

551 -git-filter-branch manpage are also affected by this). BSD vs. GNU

552 -userland differences can really bite. If lucky, error messages are

553 -spewed. But just as likely, the commands either don't do the filtering

554 -requested, or silently corrupt by making some unwanted change. The

555 -unwanted change may only affect a few commits, so it's not necessarily

556 -obvious either. (The fact that problems won't necessarily be obvious

557 -means they are likely to go unnoticed until the rewritten history is in

558 -use for quite a while, at which point it's really hard to justify

559 -another flag-day for another rewrite.)

551 + document or provide to a coworker, who then runs them on a different

552 + OS where the same commands are not working/tested (some examples in

553 + the git-filter-branch manpage are also affected by this).

554 + BSD vs. GNU userland differences can really bite. If lucky, error

555 + messages are spewed. But just as likely, the commands either don't

556 + do the filtering requested, or silently corrupt by making some

557 + unwanted change. The unwanted change may only affect a few commits,

558 + so it's not necessarily obvious either. (The fact that problems

559 + won't necessarily be obvious means they are likely to go unnoticed

560 + until the rewritten history is in use for quite a while, at which

561 + point it's really hard to justify another flag-day for another

562 + rewrite.)

563

564 * Filenames with spaces are often mishandled by shell snippets since

562 -they cause problems for shell pipelines. Not everyone is familiar with

563 -find -print0, xargs -0, git-ls-files -z, etc. Even people who are

564 -familiar with these may assume such flags are not relevant because

565 -someone else renamed any such files in their repo back before the person

566 -doing the filtering joined the project. And often, even those familiar

567 -with handling arguments with spaces may not do so just because they

568 -aren't in the mindset of thinking about everything that could possibly

569 -go wrong.

570 -

571 -* Non-ascii filenames can be silently removed despite being in a desired

572 -directory. Keeping only wanted paths is often done using pipelines like

573 -`git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. ls-files will

574 -only quote filenames if needed, so folks may not notice that one of the

575 -files didn't match the regex (at least not until it's much too late).

576 -Yes, someone who knows about core.quotePath can avoid this (unless they

577 -have other special characters like \t, \n, or "), and people who use

578 -ls-files -z with something other than grep can avoid this, but that

579 -doesn't mean they will.

580 -

581 -* Similarly, when moving files around, one can find that filenames with

582 -non-ascii or special characters end up in a different directory, one

583 -that includes a double quote character. (This is technically the same

584 -issue as above with quoting, but perhaps an interesting different way

585 -that it can and has manifested as a problem.)

565 + they cause problems for shell pipelines. Not everyone is familiar

566 + with find -print0, xargs -0, git-ls-files -z, etc. Even people who

567 + are familiar with these may assume such flags are not relevant

568 + because someone else renamed any such files in their repo back

569 + before the person doing the filtering joined the project. And

570 + often, even those familiar with handling arguments with spaces may

571 + not do so just because they aren't in the mindset of thinking about

572 + everything that could possibly go wrong.

573 +

574 +* Non-ascii filenames can be silently removed despite being in a

575 + desired directory. Keeping only wanted paths is often done using

576 + pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.

577 + ls-files will only quote filenames if needed, so folks may not

578 + notice that one of the files didn't match the regex (at least not

579 + until it's much too late). Yes, someone who knows about

580 + core.quotePath can avoid this (unless they have other special

581 + characters like \t, \n, or "), and people who use ls-files -z with

582 + something other than grep can avoid this, but that doesn't mean they

583 + will.

584 +

585 +* Similarly, when moving files around, one can find that filenames

586 + with non-ascii or special characters end up in a different

587 + directory, one that includes a double quote character. (This is

588 + technically the same issue as above with quoting, but perhaps an

589 + interesting different way that it can and has manifested as a

590 + problem.)

591

592 * It's far too easy to accidentally mix up old and new history. It's

588 -still possible with any tool, but git-filter-branch almost invites it.

589 -If lucky, the only downside is users getting frustrated that they don't

590 -know how to shrink their repo and remove the old stuff. If unlucky,

591 -they merge old and new history and end up with multiple "copies" of each

592 -commit, some of which have unwanted or sensitive files and others which

593 -don't. This comes about in multiple different ways:

593 + still possible with any tool, but git-filter-branch almost

594 + invites it. If lucky, the only downside is users getting frustrated

595 + that they don't know how to shrink their repo and remove the old

596 + stuff. If unlucky, they merge old and new history and end up with

597 + multiple "copies" of each commit, some of which have unwanted or

598 + sensitive files and others which don't. This comes about in

599 + multiple different ways:

600

601 ** the default to only doing a partial history rewrite ('--all' is not

602 the default and few examples show it)

@@ -609,8 +615,8 @@ don't. This comes about in multiple different ways:

615 "DISCUSSION" section of the git filter-repo manual page for more

616 details.

617

612 -* Annotated tags can be accidentally converted to lightweight tags, due

613 -to either of two issues:

618 +* Annotated tags can be accidentally converted to lightweight tags,

619 + due to either of two issues:

620

621 ** Someone can do a history rewrite, realize they messed up, restore

622 from the backups in refs/original/, and then redo their

@@ -623,71 +629,74 @@ to either of two issues:

629 restored from refs/original/ in a previously botched rewrite).

630

631 * Any commit messages that specify an encoding will become corrupted

626 -by the rewrite; git-filter-branch ignores the encoding, takes the original

627 -bytes, and feeds it to commit-tree without telling it the proper

628 -encoding. (This happens whether or not --msg-filter is used.)

632 + by the rewrite; git-filter-branch ignores the encoding, takes the

633 + original bytes, and feeds it to commit-tree without telling it the

634 + proper encoding. (This happens whether or not --msg-filter is

635 + used.)

636

637 * Commit messages (even if they are all UTF-8) by default become

631 -corrupted due to not being updated -- any references to other commit

632 -hashes in commit messages will now refer to no-longer-extant commits.

633 -

634 -* There are no facilities for helping users find what unwanted crud they

635 -should delete, which means they are much more likely to have incomplete

636 -or partial cleanups that sometimes result in confusion and people

637 -wasting time trying to understand. (For example, folks tend to just

638 -look for big files to delete instead of big directories or extensions,

639 -and once they do so, then sometime later folks using the new repository

640 -who are going through history will notice a build artifact directory

641 -that has some files but not others, or a cache of dependencies

642 -(node_modules or similar) which couldn't have ever been functional since

643 -it's missing some files.)

638 + corrupted due to not being updated -- any references to other commit

639 + hashes in commit messages will now refer to no-longer-extant

640 + commits.

641 +

642 +* There are no facilities for helping users find what unwanted crud

643 + they should delete, which means they are much more likely to have

644 + incomplete or partial cleanups that sometimes result in confusion

645 + and people wasting time trying to understand. (For example, folks

646 + tend to just look for big files to delete instead of big directories

647 + or extensions, and once they do so, then sometime later folks using

648 + the new repository who are going through history will notice a build

649 + artifact directory that has some files but not others, or a cache of

650 + dependencies (node_modules or similar) which couldn't have ever been

651 + functional since it's missing some files.)

652

653 * If --prune-empty isn't specified, then the filtering process can

646 -create hoards of confusing empty commits

654 + create hoards of confusing empty commits

655

656 * If --prune-empty is specified, then intentionally placed empty

649 -commits from before the filtering operation are also pruned instead of

650 -just pruning commits that became empty due to filtering rules.

657 + commits from before the filtering operation are also pruned instead

658 + of just pruning commits that became empty due to filtering rules.

659

660 * If --prune empty is specified, sometimes empty commits are missed

653 -and left around anyway (a somewhat rare bug, but it happens...)

661 + and left around anyway (a somewhat rare bug, but it happens...)

662

663 * A minor issue, but users who have a goal to update all names and

656 -emails in a repository may be led to --env-filter which will only update

657 -authors and committers, missing taggers.

664 + emails in a repository may be led to --env-filter which will only

665 + update authors and committers, missing taggers.

666

667 * If the user provides a --tag-name-filter that maps multiple tags to

660 -the same name, no warning or error is provided; git-filter-branch simply

661 -overwrites each tag in some undocumented pre-defined order resulting in

662 -only one tag at the end. (A git-filter-branch regression test requires

663 -this surprising behavior.)

668 + the same name, no warning or error is provided; git-filter-branch

669 + simply overwrites each tag in some undocumented pre-defined order

670 + resulting in only one tag at the end. (A git-filter-branch

671 + regression test requires this surprising behavior.)

672

673 Also, the poor performance of git-filter-branch often leads to safety

674 issues:

675

668 -* Coming up with the correct shell snippet to do the filtering you want

669 -is sometimes difficult unless you're just doing a trivial modification

670 -such as deleting a couple files. Unfortunately, people often learn if

671 -the snippet is right or wrong by trying it out, but the rightness or

672 -wrongness can vary depending on special circumstances (spaces in

673 -filenames, non-ascii filenames, funny author names or emails, invalid

674 -timezones, presence of grafts or replace objects, etc.), meaning they

675 -may have to wait a long time, hit an error, then restart. The

676 -performance of git-filter-branch is so bad that this cycle is painful,

677 -reducing the time available to carefully re-check (to say nothing about

678 -what it does to the patience of the person doing the rewrite even if

679 -they do technically have more time available). This problem is extra

680 -compounded because errors from broken filters may not be shown for a

681 -long time and/or get lost in a sea of output. Even worse, broken

682 -filters often just result in silent incorrect rewrites.

683 -

684 -* To top it all off, even when users finally find working commands, they

685 -naturally want to share them. But they may be unaware that their repo

686 -didn't have some special cases that someone else's does. So, when

687 -someone else with a different repository runs the same commands, they

688 -get hit by the problems above. Or, the user just runs commands that

689 -really were vetted for special cases, but they run it on a different OS

690 -where it doesn't work, as noted above.

676 +* Coming up with the correct shell snippet to do the filtering you

677 + want is sometimes difficult unless you're just doing a trivial

678 + modification such as deleting a couple files. Unfortunately, people

679 + often learn if the snippet is right or wrong by trying it out, but

680 + the rightness or wrongness can vary depending on special

681 + circumstances (spaces in filenames, non-ascii filenames, funny

682 + author names or emails, invalid timezones, presence of grafts or

683 + replace objects, etc.), meaning they may have to wait a long time,

684 + hit an error, then restart. The performance of git-filter-branch is

685 + so bad that this cycle is painful, reducing the time available to

686 + carefully re-check (to say nothing about what it does to the

687 + patience of the person doing the rewrite even if they do technically

688 + have more time available). This problem is extra compounded because

689 + errors from broken filters may not be shown for a long time and/or

690 + get lost in a sea of output. Even worse, broken filters often just

691 + result in silent incorrect rewrites.

692 +

693 +* To top it all off, even when users finally find working commands,

694 + they naturally want to share them. But they may be unaware that

695 + their repo didn't have some special cases that someone else's does.

696 + So, when someone else with a different repository runs the same

697 + commands, they get hit by the problems above. Or, the user just

698 + runs commands that really were vetted for special cases, but they

699 + run it on a different OS where it doesn't work, as noted above.

700

701 GIT

702 ---

doc: indent multi-line items in list