@matheusbsbs / matheus-bagliano / commits / 520cd3eca5
Fork 0

Documentation/technical/racy-git.txt

Signed-off-by: Junio C Hamano <junkio@cox.net>

Junio C Hamano committed Aug 15, 2006 at 13:09 UTC 520cd3eca5743bebd217423e1fd0721f32613bb1
1 file changed +193
Documentation/technical/racy-git.txt new
+193
@@ -0,0 +1,193 @@
1 +Use of index and Racy git problem
2 +=================================
3 +
4 +Background
5 +----------
6 +
7 +The index is one of the most important data structure in git.
8 +It represents a virtual working tree state by recording list of
9 +paths and their object names and serves as a staging area to
10 +write out the next tree object to be committed. The state is
11 +"virtual" in the sense that it does not necessarily have to, and
12 +often does not, match the files in the working tree.
13 +
14 +There are cases git needs to examine the differences between the
15 +virtual working tree state in the index and the files in the
16 +working tree. The most obvious case is when the user asks `git
17 +diff` (or its low level implementation, `git diff-files`) or
18 +`git-ls-files --modified`. In addition, git internally checks
19 +if the files in the working tree is different from what are
20 +recorded in the index to avoid stomping on local changes in them
21 +during patch application, switching branches, and merging.
22 +
23 +In order to speed up this comparison between the files in the
24 +working tree and the index entries, the index entries record the
25 +information obtained from the filesystem via `lstat(2)` system
26 +call when they were last updated. When checking if they differ,
27 +git first runs `lstat(2)` on the files and compare the result
28 +with this information (this is what was originally done by the
29 +`ce_match_stat()` function, which the current code does in
30 +`ce_match_stat_basic()` function). If some of these "cached
31 +stat information" fields do not match, git can tell that the
32 +files are modified without even looking at their contents.
33 +
34 +Note: not all members in `struct stat` obtained via `lstat(2)`
35 +are used for this comparison. For example, `st_atime` obviously
36 +is not useful. Currently, git compares the file type (regular
37 +files vs symbolic links) and executable bits (only for regular
38 +files) from `st_mode` member, `st_mtime` and `st_ctime`
39 +timestamps, `st_uid`, `st_gid`, `st_ino`, and `st_size` members.
40 +With a `USE_STDEV` compile-time option, `st_dev` is also
41 +compared, but this is not enabled by default because this member
42 +is not stable on network filesystems. With `USE_NSEC`
43 +compile-time option, `st_mtim.tv_nsec` and `st_ctim.tv_nsec`
44 +members are also compared, but this is not enabled by default
45 +because the value of this member becomes meaningless once the
46 +inode is evicted from the inode cache on filesystems that do not
47 +store it on disk.
48 +
49 +
50 +Racy git
51 +--------
52 +
53 +There is one slight problem with the optimization based on the
54 +cached stat information. Consider this sequence:
55 +
56 + $ git update-index 'foo'
57 + : modify 'foo' in-place without changing its size
58 +
59 +The first `update-index` computes the object name of the
60 +contents of file `foo` and updates the index entry for `foo`
61 +along with the `struct stat` information. If the modification
62 +that follows it happens very fast so that the file's `st_mtime`
63 +timestamp does not change, after this sequence, the cached stat
64 +information the index entry records still exactly match what you
65 +can obtain from the filesystem, but the file `foo` is modified.
66 +This way, git can incorrectly think files in the working tree
67 +are unmodified even though they actually are. This is called
68 +the "racy git" problem (discovered by Pasky), and the entries
69 +that appear clean when they may not be because of this problem
70 +are called "racily clean".
71 +
72 +To avoid this problem, git does two things:
73 +
74 +. When the cached stat information says the file has not been
75 + modified, and the `st_mtime` is the same as (or newer than)
76 + the timestamp of the index file itself (which is the time `git
77 + update-index foo` finished running in the above example), it
78 + also compares the contents with the object registered in the
79 + index entry to make sure they match.
80 +
81 +. When the index file is updated that contains racily clean
82 + entries, cached `st_size` information is truncated to zero
83 + before writing a new version of the index file.
84 +
85 +Because the index file itself is written after collecting all
86 +the stat information from updated paths, `st_mtime` timestamp of
87 +it is usually the same as or newer than any of the paths the
88 +index contains. And no matter how quick the modification that
89 +follows `git update-index foo` finishes, the resulting
90 +`st_mtime` timestamp on `foo` cannot get the timestamp earlier
91 +than the index file. Therefore, index entries that can be
92 +racily clean are limited to the ones that have the same
93 +timestamp as the index file itself.
94 +
95 +The callers that want to check if an index entry matches the
96 +corresponding file in the working tree continue to call
97 +`ce_match_stat()`, but with this change, `ce_match_stat()` uses
98 +`ce_modified_check_fs()` to see if racily clean ones are
99 +actually clean after comparing the cached stat information using
100 +`ce_match_stat_basic()`.
101 +
102 +The problem the latter solves is this sequence:
103 +
104 + $ git update-index 'foo'
105 + : modify 'foo' in-place without changing its size
106 + : wait for enough time
107 + $ git update-index 'bar'
108 +
109 +Without the latter, the timestamp of the index file gets a newer
110 +value, and falsely clean entry `foo` would not be caught by the
111 +timestamp comparison check done with the former logic anymore.
112 +The latter makes sure that the cached stat information for `foo`
113 +would never match with the file in the working tree, so later
114 +checks by `ce_match_stat_basic()` would report the index entry
115 +does not match the file and git does not have to fall back on more
116 +expensive `ce_modified_check_fs()`.
117 +
118 +
119 +Runtime penalty
120 +---------------
121 +
122 +The runtime penalty of falling back to `ce_modified_check_fs()`
123 +from `ce_match_stat()` can be very expensive when there are many
124 +racily clean entries. An obvious way to artificially create
125 +this situation is to give the same timestamp to all the files in
126 +the working tree in a large project, run `git update-index` on
127 +them, and give the same timestamp to the index file:
128 +
129 + $ date >.datestamp
130 + $ git ls-files | xargs touch -r .datestamp
131 + $ git ls-files | git update-index --stdin
132 + $ touch -r .datestamp .git/index
133 +
134 +This will make all index entries racily clean. The linux-2.6
135 +project, for example, there are over 20,000 files in the working
136 +tree. On my Athron 64X2 3800+, after the above:
137 +
138 + $ /usr/bin/time git diff-files
139 + 1.68user 0.54system 0:02.22elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
140 + 0inputs+0outputs (0major+67111minor)pagefaults 0swaps
141 + $ git update-index MAINTAINERS
142 + $ /usr/bin/time git diff-files
143 + 0.02user 0.12system 0:00.14elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
144 + 0inputs+0outputs (0major+935minor)pagefaults 0swaps
145 +
146 +Running `git update-index` in the middle checked the racily
147 +clean entries, and left the cached `st_mtime` for all the paths
148 +intact because they were actually clean (so this step took about
149 +the same amount of time as the first `git diff-files`). After
150 +that, they are not racily clean anymore but are truly clean, so
151 +the second invocation of `git diff-files` fully took advantage
152 +of the cached stat information.
153 +
154 +
155 +Avoiding runtime penalty
156 +------------------------
157 +
158 +In order to avoid the above runtime penalty, the recent "master"
159 +branch (post 1.4.2) has a code that makes sure the index file
160 +gets timestamp newer than the youngest files in the index when
161 +there are many young files with the same timestamp as the
162 +resulting index file would otherwise would have by waiting
163 +before finishing writing the index file out.
164 +
165 +I suspect that in practice the situation where many paths in the
166 +index are all racily clean is quite rare. The only code paths
167 +that can record recent timestamp for large number of paths I
168 +know of are:
169 +
170 +. Initial `git add .` of a large project.
171 +
172 +. `git checkout` of a large project from an empty index into an
173 + unpopulated working tree.
174 +
175 +Note: switching branches with `git checkout` keeps the cached
176 +stat information of existing working tree files that are the
177 +same between the current branch and the new branch, which are
178 +all older than the resulting index file, and they will not
179 +become racily clean. Only the files that are actually checked
180 +out can become racily clean.
181 +
182 +In a large project where raciness avoidance cost really matters,
183 +however, the initial computation of all object names in the
184 +index takes more than one second, and the index file is written
185 +out after all that happens. Therefore the timestamp of the
186 +index file will be more than one seconds later than the the
187 +youngest file in the working tree. This means that in these
188 +cases there actually will not be any racily clean entry in
189 +the resulting index.
190 +
191 +So in summary I think we should not worry about avoiding the
192 +runtime penalty and get rid of the "wait before finishing
193 +writing" code out.