| 1 | gitdiffcore(7) |
| 2 | ============== |
| 3 | |
| 4 | NAME |
| 5 | ---- |
| 6 | gitdiffcore - Tweaking diff output |
| 7 | |
| 8 | SYNOPSIS |
| 9 | -------- |
| 10 | [verse] |
| 11 | 'git diff' * |
| 12 | |
| 13 | DESCRIPTION |
| 14 | ----------- |
| 15 | |
| 16 | The diff commands 'git diff-index', 'git diff-files', and 'git diff-tree' |
| 17 | can be told to manipulate differences they find in |
| 18 | unconventional ways before showing 'diff' output. The manipulation |
| 19 | is collectively called "diffcore transformation". This short note |
| 20 | describes what they are and how to use them to produce 'diff' output |
| 21 | that is easier to understand than the conventional kind. |
| 22 | |
| 23 | |
| 24 | The chain of operation |
| 25 | ---------------------- |
| 26 | |
| 27 | The 'git diff-{asterisk}' family works by first comparing two sets of |
| 28 | files: |
| 29 | |
| 30 | - 'git diff-index' compares contents of a "tree" object and the |
| 31 | working directory (when `--cached` flag is not used) or a |
| 32 | "tree" object and the index file (when `--cached` flag is |
| 33 | used); |
| 34 | |
| 35 | - 'git diff-files' compares contents of the index file and the |
| 36 | working directory; |
| 37 | |
| 38 | - 'git diff-tree' compares contents of two "tree" objects; |
| 39 | |
| 40 | In all of these cases, the commands themselves first optionally limit |
| 41 | the two sets of files by any pathspecs given on their command-lines, |
| 42 | and compare corresponding paths in the two resulting sets of files. |
| 43 | |
| 44 | The pathspecs are used to limit the world diff operates in. They remove |
| 45 | the filepairs outside the specified sets of pathnames. E.g. If the |
| 46 | input set of filepairs included: |
| 47 | |
| 48 | ------------------------------------------------ |
| 49 | :100644 100644 bcd1234... 0123456... M junkfile |
| 50 | ------------------------------------------------ |
| 51 | |
| 52 | but the command invocation was `git diff-files myfile`, then the |
| 53 | junkfile entry would be removed from the list because only "myfile" |
| 54 | is under consideration. |
| 55 | |
| 56 | The result of comparison is passed from these commands to what is |
| 57 | internally called "diffcore", in a format similar to what is output |
| 58 | when the -p option is not used. E.g. |
| 59 | |
| 60 | ------------------------------------------------ |
| 61 | in-place edit :100644 100644 bcd1234... 0123456... M file0 |
| 62 | create :000000 100644 0000000... 1234567... A file4 |
| 63 | delete :100644 000000 1234567... 0000000... D file5 |
| 64 | unmerged :000000 000000 0000000... 0000000... U file6 |
| 65 | ------------------------------------------------ |
| 66 | |
| 67 | The diffcore mechanism is fed a list of such comparison results |
| 68 | (each of which is called "filepair", although at this point each |
| 69 | of them talks about a single file), and transforms such a list |
| 70 | into another list. There are currently 5 such transformations: |
| 71 | |
| 72 | - diffcore-break |
| 73 | - diffcore-rename |
| 74 | - diffcore-merge-broken |
| 75 | - diffcore-pickaxe |
| 76 | - diffcore-order |
| 77 | - diffcore-rotate |
| 78 | |
| 79 | These are applied in sequence. The set of filepairs 'git diff-{asterisk}' |
| 80 | commands find are used as the input to diffcore-break, and |
| 81 | the output from diffcore-break is used as the input to the |
| 82 | next transformation. The final result is then passed to the |
| 83 | output routine and generates either diff-raw format (see Output |
| 84 | format sections of the manual for 'git diff-{asterisk}' commands) or |
| 85 | diff-patch format. |
| 86 | |
| 87 | |
| 88 | diffcore-break: For Splitting Up Complete Rewrites |
| 89 | -------------------------------------------------- |
| 90 | |
| 91 | The second transformation in the chain is diffcore-break, and is |
| 92 | controlled by the -B option to the 'git diff-{asterisk}' commands. This is |
| 93 | used to detect a filepair that represents "complete rewrite" and |
| 94 | break such filepair into two filepairs that represent delete and |
| 95 | create. E.g. If the input contained this filepair: |
| 96 | |
| 97 | ------------------------------------------------ |
| 98 | :100644 100644 bcd1234... 0123456... M file0 |
| 99 | ------------------------------------------------ |
| 100 | |
| 101 | and if it detects that the file "file0" is completely rewritten, |
| 102 | it changes it to: |
| 103 | |
| 104 | ------------------------------------------------ |
| 105 | :100644 000000 bcd1234... 0000000... D file0 |
| 106 | :000000 100644 0000000... 0123456... A file0 |
| 107 | ------------------------------------------------ |
| 108 | |
| 109 | For the purpose of breaking a filepair, diffcore-break examines |
| 110 | the extent of changes between the contents of the files before |
| 111 | and after modification (i.e. the contents that have "bcd1234..." |
| 112 | and "0123456..." as their SHA-1 content ID, in the above |
| 113 | example). The amount of deletion of original contents and |
| 114 | insertion of new material are added together, and if it exceeds |
| 115 | the "break score", the filepair is broken into two. The break |
| 116 | score defaults to 50% of the size of the smaller of the original |
| 117 | and the result (i.e. if the edit shrinks the file, the size of |
| 118 | the result is used; if the edit lengthens the file, the size of |
| 119 | the original is used), and can be customized by giving a number |
| 120 | after "-B" option (e.g. "-B75" to tell it to use 75%). |
| 121 | |
| 122 | |
| 123 | diffcore-rename: For Detecting Renames and Copies |
| 124 | ------------------------------------------------- |
| 125 | |
| 126 | This transformation is used to detect renames and copies, and is |
| 127 | controlled by the -M option (to detect renames) and the -C option |
| 128 | (to detect copies as well) to the 'git diff-{asterisk}' commands. If the |
| 129 | input contained these filepairs: |
| 130 | |
| 131 | ------------------------------------------------ |
| 132 | :100644 000000 0123456... 0000000... D fileX |
| 133 | :000000 100644 0000000... 0123456... A file0 |
| 134 | ------------------------------------------------ |
| 135 | |
| 136 | and the contents of the deleted file fileX is similar enough to |
| 137 | the contents of the created file file0, then rename detection |
| 138 | merges these filepairs and creates: |
| 139 | |
| 140 | ------------------------------------------------ |
| 141 | :100644 100644 0123456... 0123456... R100 fileX file0 |
| 142 | ------------------------------------------------ |
| 143 | |
| 144 | When the "-C" option is used, the original contents of modified files, |
| 145 | and deleted files (and also unmodified files, if the |
| 146 | "--find-copies-harder" option is used) are considered as candidates |
| 147 | of the source files in rename/copy operation. If the input were like |
| 148 | these filepairs, that talk about a modified file fileY and a newly |
| 149 | created file file0: |
| 150 | |
| 151 | ------------------------------------------------ |
| 152 | :100644 100644 0123456... 1234567... M fileY |
| 153 | :000000 100644 0000000... bcd3456... A file0 |
| 154 | ------------------------------------------------ |
| 155 | |
| 156 | the original contents of fileY and the resulting contents of |
| 157 | file0 are compared, and if they are similar enough, they are |
| 158 | changed to: |
| 159 | |
| 160 | ------------------------------------------------ |
| 161 | :100644 100644 0123456... 1234567... M fileY |
| 162 | :100644 100644 0123456... bcd3456... C100 fileY file0 |
| 163 | ------------------------------------------------ |
| 164 | |
| 165 | In both rename and copy detection, the same "extent of changes" |
| 166 | algorithm used in diffcore-break is used to determine if two |
| 167 | files are "similar enough", and can be customized to use |
| 168 | a similarity score different from the default of 50% by giving a |
| 169 | number after the "-M" or "-C" option (e.g. "-M8" to tell it to use |
| 170 | 8/10 = 80%). |
| 171 | |
| 172 | Note that when rename detection is on but both copy and break |
| 173 | detection are off, rename detection adds a preliminary step that first |
| 174 | checks if files are moved across directories while keeping their |
| 175 | filename the same. If there is a file added to a directory whose |
| 176 | contents are sufficiently similar to a file with the same name that got |
| 177 | deleted from a different directory, it will mark them as renames and |
| 178 | exclude them from the later quadratic step (the one that pairwise |
| 179 | compares all unmatched files to find the "best" matches, determined by |
| 180 | the highest content similarity). So, for example, if a deleted |
| 181 | docs/ext.txt and an added docs/config/ext.txt are similar enough, they |
| 182 | will be marked as a rename and prevent an added docs/ext.md that may |
| 183 | be even more similar to the deleted docs/ext.txt from being considered |
| 184 | as the rename destination in the later step. For this reason, the |
| 185 | preliminary "match same filename" step uses a bit higher threshold to |
| 186 | mark a file pair as a rename and stop considering other candidates for |
| 187 | better matches. At most, one comparison is done per file in this |
| 188 | preliminary pass; so if there are several remaining ext.txt files |
| 189 | throughout the directory hierarchy after exact rename detection, this |
| 190 | preliminary step may be skipped for those files. |
| 191 | |
| 192 | Note. When the "-C" option is used with `--find-copies-harder` |
| 193 | option, 'git diff-{asterisk}' commands feed unmodified filepairs to |
| 194 | diffcore mechanism as well as modified ones. This lets the copy |
| 195 | detector consider unmodified files as copy source candidates at |
| 196 | the expense of making it slower. Without `--find-copies-harder`, |
| 197 | 'git diff-{asterisk}' commands can detect copies only if the file that was |
| 198 | copied happened to have been modified in the same changeset. |
| 199 | |
| 200 | |
| 201 | diffcore-merge-broken: For Putting Complete Rewrites Back Together |
| 202 | ------------------------------------------------------------------ |
| 203 | |
| 204 | This transformation is used to merge filepairs broken by |
| 205 | diffcore-break, and not transformed into rename/copy by |
| 206 | diffcore-rename, back into a single modification. This always |
| 207 | runs when diffcore-break is used. |
| 208 | |
| 209 | For the purpose of merging broken filepairs back, it uses a |
| 210 | different "extent of changes" computation from the ones used by |
| 211 | diffcore-break and diffcore-rename. It counts only the deletion |
| 212 | from the original, and does not count insertion. If you removed |
| 213 | only 10 lines from a 100-line document, even if you added 910 |
| 214 | new lines to make a new 1000-line document, you did not do a |
| 215 | complete rewrite. diffcore-break breaks such a case in order to |
| 216 | help diffcore-rename to consider such filepairs as a candidate of |
| 217 | rename/copy detection, but if filepairs broken that way were not |
| 218 | matched with other filepairs to create rename/copy, then this |
| 219 | transformation merges them back into the original |
| 220 | "modification". |
| 221 | |
| 222 | The "extent of changes" parameter can be tweaked from the |
| 223 | default 80% (that is, unless more than 80% of the original |
| 224 | material is deleted, the broken pairs are merged back into a |
| 225 | single modification) by giving a second number to -B option, |
| 226 | like these: |
| 227 | |
| 228 | * -B50/60 (give 50% "break score" to diffcore-break, use 60% |
| 229 | for diffcore-merge-broken). |
| 230 | |
| 231 | * -B/60 (the same as above, since diffcore-break defaults to 50%). |
| 232 | |
| 233 | Note that earlier implementation left a broken pair as separate |
| 234 | creation and deletion patches. This was an unnecessary hack, and |
| 235 | the latest implementation always merges all the broken pairs |
| 236 | back into modifications, but the resulting patch output is |
| 237 | formatted differently for easier review in case of such |
| 238 | a complete rewrite by showing the entire contents of the old version |
| 239 | prefixed with '-', followed by the entire contents of the new |
| 240 | version prefixed with '+'. |
| 241 | |
| 242 | |
| 243 | diffcore-pickaxe: For Detecting Addition/Deletion of Specified String |
| 244 | --------------------------------------------------------------------- |
| 245 | |
| 246 | This transformation limits the set of filepairs to those that change |
| 247 | specified strings between the preimage and the postimage in a certain |
| 248 | way. -S<block-of-text> and -G<regular-expression> options are used to |
| 249 | specify different ways these strings are sought. |
| 250 | |
| 251 | "-S<block-of-text>" detects filepairs whose preimage and postimage |
| 252 | have different number of occurrences of the specified block of text. |
| 253 | By definition, it will not detect in-file moves. Also, when a |
| 254 | changeset moves a file wholesale without affecting the interesting |
| 255 | string, diffcore-rename kicks in as usual, and `-S` omits the filepair |
| 256 | (since the number of occurrences of that string didn't change in that |
| 257 | rename-detected filepair). When used with `--pickaxe-regex`, treat |
| 258 | the <block-of-text> as an extended POSIX regular expression to match, |
| 259 | instead of a literal string. |
| 260 | |
| 261 | "-G<regular-expression>" (mnemonic: grep) detects filepairs whose |
| 262 | textual diff has an added or a deleted line that matches the given |
| 263 | regular expression. This means that it will detect in-file (or what |
| 264 | rename-detection considers the same file) moves, which is noise. The |
| 265 | implementation runs diff twice and greps, and this can be quite |
| 266 | expensive. To speed things up, binary files without textconv filters |
| 267 | will be ignored. |
| 268 | |
| 269 | When `-S` or `-G` are used without `--pickaxe-all`, only filepairs |
| 270 | that match their respective criterion are kept in the output. When |
| 271 | `--pickaxe-all` is used, if even one filepair matches their respective |
| 272 | criterion in a changeset, the entire changeset is kept. This behavior |
| 273 | is designed to make reviewing changes in the context of the whole |
| 274 | changeset easier. |
| 275 | |
| 276 | diffcore-order: For Sorting the Output Based on Filenames |
| 277 | --------------------------------------------------------- |
| 278 | |
| 279 | This is used to reorder the filepairs according to the user's |
| 280 | (or project's) taste, and is controlled by the -O option to the |
| 281 | 'git diff-{asterisk}' commands. |
| 282 | |
| 283 | This takes a text file each of whose lines is a shell glob |
| 284 | pattern. Filepairs that match a glob pattern on an earlier line |
| 285 | in the file are output before ones that match a later line, and |
| 286 | filepairs that do not match any glob pattern are output last. |
| 287 | |
| 288 | As an example, a typical orderfile for the core Git probably |
| 289 | would look like this: |
| 290 | |
| 291 | ------------------------------------------------ |
| 292 | README |
| 293 | Makefile |
| 294 | Documentation |
| 295 | *.h |
| 296 | *.c |
| 297 | t |
| 298 | ------------------------------------------------ |
| 299 | |
| 300 | diffcore-rotate: For Changing At Which Path Output Starts |
| 301 | --------------------------------------------------------- |
| 302 | |
| 303 | This transformation takes one pathname, and rotates the set of |
| 304 | filepairs so that the filepair for the given pathname comes first, |
| 305 | optionally discarding the paths that come before it. This is used |
| 306 | to implement the `--skip-to` and the `--rotate-to` options. It is |
| 307 | an error when the specified pathname is not in the set of filepairs, |
| 308 | but it is not useful to error out when used with "git log" family of |
| 309 | commands, because it is unreasonable to expect that a given path |
| 310 | would be modified by each and every commit shown by the "git log" |
| 311 | command. For this reason, when used with "git log", the filepair |
| 312 | that sorts the same as, or the first one that sorts after, the given |
| 313 | pathname is where the output starts. |
| 314 | |
| 315 | Use of this transformation combined with diffcore-order will produce |
| 316 | unexpected results, as the input to this transformation is likely |
| 317 | not sorted when diffcore-order is in effect. |
| 318 | |
| 319 | |
| 320 | SEE ALSO |
| 321 | -------- |
| 322 | linkgit:git-diff[1], |
| 323 | linkgit:git-diff-files[1], |
| 324 | linkgit:git-diff-index[1], |
| 325 | linkgit:git-diff-tree[1], |
| 326 | linkgit:git-format-patch[1], |
| 327 | linkgit:git-log[1], |
| 328 | linkgit:gitglossary[7], |
| 329 | link:user-manual.html[The Git User's Manual] |
| 330 | |
| 331 | GIT |
| 332 | --- |
| 333 | Part of the linkgit:git[1] suite |