Added Packing Heursitics IRC writeup.
Signed-off-by: Jon Loeliger <jdl@jdl.com> Signed-off-by: Junio C Hamano <junkio@cox.net>
Jon Loeliger committed
Mar 2, 2006 at 19:19 UTC
b116b297a80b54632256eb89dd22ea2b140de622
1 file changed
+466
Documentation/technical/pack-heuristics.txt
new
+466
@@ -0,0 +1,466 @@
1
+ Concerning Git's Packing Heuristics
2
+ ===================================
3
+
4
+ Oh, here's a really stupid question:
5
+
6
+ Where do I go
7
+ to learn the details
8
+ of git's packing heuristics?
9
+
10
+Be careful what you ask!
11
+
12
+Followers of the git, please open the git IRC Log and turn to
13
+February 10, 2006.
14
+
15
+It's a rare occasion, and we are joined by the King Git Himself,
16
+Linus Torvalds (linus). Nathaniel Smith, (njs`), has the floor
17
+and seeks enlightenment. Others are present, but silent.
18
+
19
+Let's listen in!
20
+
21
+ <njs`> Oh, here's a really stupid question -- where do I go to
22
+ learn the details of git's packing heuristics? google avails
23
+ me not, reading the source didn't help a lot, and wading
24
+ through the whole mailing list seems less efficient than any
25
+ of that.
26
+
27
+It is a bold start! A plea for help combined with a simultaneous
28
+tri-part attack on some of the tried and true mainstays in the quest
29
+for enlightenment. Brash accusations of google being useless. Hubris!
30
+Maligning the source. Heresy! Disdain for the mailing list archives.
31
+Woe.
32
+
33
+ <pasky> yes, the packing-related delta stuff is somewhat
34
+ mysterious even for me ;)
35
+
36
+Ah! Modesty after all.
37
+
38
+ <linus> njs, I don't think the docs exist. That's something where
39
+ I don't think anybody else than me even really got involved.
40
+ Most of the rest of git others have been busy with (especially
41
+ Junio), but packing nobody touched after I did it.
42
+
43
+It's cryptic, yet vague. Linus in style for sure. Wise men
44
+interpret this as an apology. A few argue it is merely a
45
+statement of fact.
46
+
47
+ <njs`> I guess the next step is "read the source again", but I
48
+ have to build up a certain level of gumption first :-)
49
+
50
+Indeed! On both points.
51
+
52
+ <linus> The packing heuristic is actually really really simple.
53
+
54
+Bait...
55
+
56
+ <linus> But strange.
57
+
58
+And switch. That ought to do it!
59
+
60
+ <linus> Remember: git really doesn't follow files. So what it does is
61
+ - generate a list of all objects
62
+ - sort the list according to magic heuristics
63
+ - walk the list, using a sliding window, seeing if an object
64
+ can be diffed against another object in the window
65
+ - write out the list in recency order
66
+
67
+The traditional understatement:
68
+
69
+ <njs`> I suspect that what I'm missing is the precise definition of
70
+ the word "magic"
71
+
72
+The traditional insight:
73
+
74
+ <pasky> yes
75
+
76
+And Bable-like confusion flowed.
77
+
78
+ <njs`> oh, hmm, and I'm not sure what this sliding window means either
79
+
80
+ <pasky> iirc, it appeared to me to be just the sha1 of the object
81
+ when reading the code casually ...
82
+
83
+ ... which simply doesn't sound as a very good heuristics, though ;)
84
+
85
+ <njs`> .....and recency order. okay, I think it's clear I didn't
86
+ even realize how much I wasn't realizing :-)
87
+
88
+Ah, grasshopper! And thus the enlightenment begins anew.
89
+
90
+ <linus> The "magic" is actually in theory totally arbitrary.
91
+ ANY order will give you a working pack, but no, it's not
92
+ ordered by SHA1.
93
+
94
+ Before talking about the ordering for the sliding delta
95
+ window, let's talk about the recency order. That's more
96
+ important in one way.
97
+
98
+ <njs`> Right, but if all you want is a working way to pack things
99
+ together, you could just use cat and save yourself some
100
+ trouble...
101
+
102
+Waaait for it....
103
+
104
+ <linus> The recency ordering (which is basically: put objects
105
+ _physically_ into the pack in the order that they are
106
+ "reachable" from the head) is important.
107
+
108
+ <njs`> okay
109
+
110
+ <linus> It's important because that's the thing that gives packs
111
+ good locality. It keeps the objects close to the head (whether
112
+ they are old or new, but they are _reachable_ from the head)
113
+ at the head of the pack. So packs actually have absolutely
114
+ _wonderful_ IO patterns.
115
+
116
+Read that again, because it is important.
117
+
118
+ <linus> But recency ordering is totally useless for deciding how
119
+ to actually generate the deltas, so the delta ordering is
120
+ something else.
121
+
122
+ The delta ordering is (wait for it):
123
+ - first sort by the "basename" of the object, as defined by
124
+ the name the object was _first_ reached through when
125
+ generating the object list
126
+ - within the same basename, sort by size of the object
127
+ - but always sort different types separately (commits first).
128
+
129
+ That's not exactly it, but it's very close.
130
+
131
+ <njs`> The "_first_ reached" thing is not too important, just you
132
+ need some way to break ties since the same objects may be
133
+ reachable many ways, yes?
134
+
135
+And as if to clarify:
136
+
137
+ <linus> The point is that it's all really just any random
138
+ heuristic, and the ordering is totally unimportant for
139
+ correctness, but it helps a lot if the heuristic gives
140
+ "clumping" for things that are likely to delta well against
141
+ each other.
142
+
143
+It is an important point, so secretly, I did my own research and have
144
+included my results below. To be fair, it has changed some over time.
145
+And through the magic of Revisionistic History, I draw upon this entry
146
+from The Git IRC Logs on my father's birthday, March 1:
147
+
148
+ <gitster> The quote from the above linus should be rewritten a
149
+ bit (wait for it):
150
+ - first sort by type. Different objects never delta with
151
+ each other.
152
+ - then sort by filename/dirname. hash of the basename
153
+ occupies the top BITS_PER_INT-DIR_BITS bits, and bottom
154
+ DIR_BITS are for the hash of leading path elements.
155
+ - then if we are doing "thin" pack, the objects we are _not_
156
+ going to pack but we know about are sorted earlier than
157
+ other objects.
158
+ - and finally sort by size, larger to smaller.
159
+
160
+In one swell-foop, clarification and obscurification! Nonetheless,
161
+authoritative. Cryptic, yet concise. It even solicits notions of
162
+quotes from The Source Code. Clearly, more study is needed.
163
+
164
+ <gitster> That's the sort order. What this means is:
165
+ - we do not delta different object types.
166
+ - we prefer to delta the objects with the same full path, but
167
+ allow files with the same name from different directories.
168
+ - we always prefer to delta against objects we are not going
169
+ to send, if there are some.
170
+ - we prefer to delta against larger objects, so that we have
171
+ lots of removals.
172
+
173
+ The penultimate rule is for "thin" packs. It is used when
174
+ the other side is known to have such objects.
175
+
176
+There it is again. "Thin" packs. I'm thinking to myself, "What
177
+is a 'thin' pack?" So I ask:
178
+
179
+ <jdl> What is a "thin" pack?
180
+
181
+ <gitster> Use of --objects-edge to rev-list as the upstream of
182
+ pack-objects. The pack transfer protocol negotiates that.
183
+
184
+Woo hoo! Cleared that _right_ up!
185
+
186
+ <gitster> There are two directions - push and fetch.
187
+
188
+There! Did you see it? It is not '"push" and "pull"'! How often the
189
+confusion has started here. So casually mentioned, too!
190
+
191
+ <gitster> For push, git-send-pack invokes git-receive-pack on the
192
+ other end. The receive-pack says "I have up to these commits".
193
+ send-pack looks at them, and computes what are missing from
194
+ the other end. So "thin" could be the default there.
195
+
196
+ In the other direction, fetch, git-fetch-pack and
197
+ git-clone-pack invokes git-upload-pack on the other end
198
+ (via ssh or by talking to the daemon).
199
+
200
+ There are two cases: fetch-pack with -k and clone-pack is one,
201
+ fetch-pack without -k is the other. clone-pack and fetch-pack
202
+ with -k will keep the downloaded packfile without expanded, so
203
+ we do not use thin pack transfer. Otherwise, the generated
204
+ pack will have delta without base object in the same pack.
205
+
206
+ But fetch-pack without -k will explode the received pack into
207
+ individual objects, so we automatically ask upload-pack to
208
+ give us a thin pack if upload-pack supports it.
209
+
210
+OK then.
211
+
212
+Uh.
213
+
214
+Let's return to the previous conversation still in progress.
215
+
216
+ <njs`> and "basename" means something like "the tail of end of
217
+ path of file objects and dir objects, as per basename(3), and
218
+ we just declare all commit and tag objects to have the same
219
+ basename" or something?
220
+
221
+Luckily, that too is a point that gitster clarified for us!
222
+
223
+If I might add, the trick is to make files that _might_ be similar be
224
+located close to each other in the hash buckets based on their file
225
+names. It used to be that "foo/Makefile", "bar/baz/quux/Makefile" and
226
+"Makefile" all landed in the same bucket due to their common basename,
227
+"Makefile". However, now they land in "close" buckets.
228
+
229
+The algorithm allows not just for the _same_ bucket, but for _close_
230
+buckets to be considered delta candidates. The rationale is
231
+essentially that files, like Makefiles, often have very similar
232
+content no matter what directory they live in.
233
+
234
+ <linus> I played around with different delta algorithms, and with
235
+ making the "delta window" bigger, but having too big of a
236
+ sliding window makes it very expensive to generate the pack:
237
+ you need to compare every object with a _ton_ of other objects.
238
+
239
+ There are a number of other trivial heuristics too, which
240
+ basically boil down to "don't bother even trying to delta this
241
+ pair" if we can tell before-hand that the delta isn't worth it
242
+ (due to size differences, where we can take a previous delta
243
+ result into account to decide that "ok, no point in trying
244
+ that one, it will be worse").
245
+
246
+ End result: packing is actually very size efficient. It's
247
+ somewhat CPU-wasteful, but on the other hand, since you're
248
+ really only supposed to do it maybe once a month (and you can
249
+ do it during the night), nobody really seems to care.
250
+
251
+Nice Engineering Touch, there. Find when it doesn't matter, and
252
+proclaim it a non-issue. Good style too!
253
+
254
+ <njs`> So, just to repeat to see if I'm following, we start by
255
+ getting a list of the objects we want to pack, we sort it by
256
+ this heuristic (basically lexicographically on the tuple
257
+ (type, basename, size)).
258
+
259
+ Then we walk through this list, and calculate a delta of
260
+ each object against the last n (tunable paramater) objects,
261
+ and pick the smallest of these deltas.
262
+
263
+Vastly simplified, but the essence is there!
264
+
265
+ <linus> Correct.
266
+
267
+ <njs`> And then once we have picked a delta or fulltext to
268
+ represent each object, we re-sort by recency, and write them
269
+ out in that order.
270
+
271
+ <linus> Yup. Some other small details:
272
+
273
+And of course there is the "Other Shoe" Factor too.
274
+
275
+ <linus> - We limit the delta depth to another magic value (right
276
+ now both the window and delta depth magic values are just "10")
277
+
278
+ <njs`> Hrm, my intuition is that you'd end up with really _bad_ IO
279
+ patterns, because the things you want are near by, but to
280
+ actually reconstruct them you may have to jump all over in
281
+ random ways.
282
+
283
+ <linus> - When we write out a delta, and we haven't yet written
284
+ out the object it is a delta against, we write out the base
285
+ object first. And no, when we reconstruct them, we actually
286
+ get nice IO patterns, because:
287
+ - larger objects tend to be "more recent" (Linus' law: files grow)
288
+ - we actively try to generate deltas from a larger object to a
289
+ smaller one
290
+ - this means that the top-of-tree very seldom has deltas
291
+ (ie deltas in _practice_ are "backwards deltas")
292
+
293
+Again, we should reread that whole paragraph. Not just because
294
+Linus has slipped Linus's Law in there on us, but because it is
295
+important. Let's make sure we clarify some of the points here:
296
+
297
+ <njs`> So the point is just that in practice, delta order and
298
+ recency order match each other quite well.
299
+
300
+ <linus> Yes. There's another nice side to this (and yes, it was
301
+ designed that way ;):
302
+ - the reason we generate deltas against the larger object is
303
+ actually a big space saver too!
304
+
305
+ <njs`> Hmm, but your last comment (if "we haven't yet written out
306
+ the object it is a delta against, we write out the base object
307
+ first"), seems like it would make these facts mostly
308
+ irrelevant because even if in practice you would not have to
309
+ wander around much, in fact you just brute-force say that in
310
+ the cases where you might have to wander, don't do that :-)
311
+
312
+ <linus> Yes and no. Notice the rule: we only write out the base
313
+ object first if the delta against it was more recent. That
314
+ means that you can actually have deltas that refer to a base
315
+ object that is _not_ close to the delta object, but that only
316
+ happens when the delta is needed to generate an _old_ object.
317
+
318
+ <linus> See?
319
+
320
+Yeah, no. I missed that on the first two or three readings myself.
321
+
322
+ <linus> This keeps the front of the pack dense. The front of the
323
+ pack never contains data that isn't relevant to a "recent"
324
+ object. The size optimization comes from our use of xdelta
325
+ (but is true for many other delta algorithms): removing data
326
+ is cheaper (in size) than adding data.
327
+
328
+ When you remove data, you only need to say "copy bytes n--m".
329
+ In contrast, in a delta that _adds_ data, you have to say "add
330
+ these bytes: 'actual data goes here'"
331
+
332
+ *** njs` has quit: Read error: 104 (Connection reset by peer)
333
+
334
+ <linus> Uhhuh. I hope I didn't blow njs` mind.
335
+
336
+ *** njs` has joined channel #git
337
+
338
+ <pasky> :)
339
+
340
+The silent observers are amused. Of course.
341
+
342
+And as if njs` was expected to be omniscient:
343
+
344
+ <linus> njs - did you miss anything?
345
+
346
+OK, I'll spell it out. That's Geek Humor. If njs` was not actually
347
+connected for a little bit there, how would he know if missed anything
348
+while he was disconnected? He's a benevolent dictator with a sense of
349
+humor! Well noted!
350
+
351
+ <njs`> Stupid router. Or gremlins, or whatever.
352
+
353
+It's a cheap shot at Cisco. Take 'em when you can.
354
+
355
+ <njs`> Yes and no. Notice the rule: we only write out the base
356
+ object first if the delta against it was more recent.
357
+
358
+ I'm getting lost in all these orders, let me re-read :-)
359
+ So the write-out order is from most recent to least recent?
360
+ (Conceivably it could be the opposite way too, I'm not sure if
361
+ we've said) though my connection back at home is logging, so I
362
+ can just read what you said there :-)
363
+
364
+And for those of you paying attention, the Omniscient Trick has just
365
+been detailed!
366
+
367
+ <linus> Yes, we always write out most recent first
368
+
369
+For the other record:
370
+
371
+ <pasky> njs`: http://pastebin.com/547965
372
+
373
+The 'net never forgets, so that should be good until the end of time.
374
+
375
+ <njs`> And, yeah, I got the part about deeper-in-history stuff
376
+ having worse IO characteristics, one sort of doesn't care.
377
+
378
+ <linus> With the caveat that if the "most recent" needs an older
379
+ object to delta against (hey, shrinking sometimes does
380
+ happen), we write out the old object with the delta.
381
+
382
+ <njs`> (if only it happened more...)
383
+
384
+ <linus> Anyway, the pack-file could easily be denser still, but
385
+ because it's used both for streaming (the git protocol) and
386
+ for on-disk, it has a few pessimizations.
387
+
388
+Actually, it is a made-up word. But it is a made-up word being
389
+used as setup for a later optimization, which is a real word:
390
+
391
+ <linus> In particular, while the pack-file is then compressed,
392
+ it's compressed just one object at a time, so the actual
393
+ compression factor is less than it could be in theory. But it
394
+ means that it's all nice random-access with a simple index to
395
+ do "object name->location in packfile" translation.
396
+
397
+ <njs`> I'm assuming the real win for delta-ing large->small is
398
+ more homogenous statistics for gzip to run over?
399
+
400
+ (You have to put the bytes in one place or another, but
401
+ putting them in a larger blob wins on compression)
402
+
403
+ Actually, what is the compression strategy -- each delta
404
+ individually gzipped, the whole file gzipped, somewhere in
405
+ between, no compression at all, ....?
406
+
407
+ Right.
408
+
409
+Reality IRC sets in. For example:
410
+
411
+ <pasky> I'll read the rest in the morning, I really have to go
412
+ sleep or there's no hope whatsoever for me at the today's
413
+ exam... g'nite all.
414
+
415
+Heh.
416
+
417
+ <linus> pasky: g'nite
418
+
419
+ <njs`> pasky: 'luck
420
+
421
+ <linus> Right: large->small matters exactly because of compression
422
+ behaviour. If it was non-compressed, it probably wouldn't make
423
+ any difference.
424
+
425
+ <njs`> yeah
426
+
427
+ <linus> Anyway: I'm not even trying to claim that the pack-files
428
+ are perfect, but they do tend to have a nice balance of
429
+ density vs ease-of use.
430
+
431
+Gasp! OK, saved. That's a fair Engineering trade off. Close call!
432
+In fact, Linus reflects on some Basic Engineering Fundamentals,
433
+design options, etc.
434
+
435
+ <linus> More importantly, they allow git to still _conceptually_
436
+ never deal with deltas at all, and be a "whole object" store.
437
+
438
+ Which has some problems (we discussed bad huge-file
439
+ behaviour on the git lists the other day), but it does mean
440
+ that the basic git concepts are really really simple and
441
+ straightforward.
442
+
443
+ It's all been quite stable.
444
+
445
+ Which I think is very much a result of having very simple
446
+ basic ideas, so that there's never any confusion about what's
447
+ going on.
448
+
449
+ Bugs happen, but they are "simple" bugs. And bugs that
450
+ actually get some object store detail wrong are almost always
451
+ so obious that they never go anywhere.
452
+
453
+ <njs`> Yeah.
454
+
455
+Nuff said.
456
+
457
+ <linus> Anyway. I'm off for bed. It's not 6AM here, but I've got
458
+ three kids, and have to get up early in the morning to send
459
+ them off. I need my beauty sleep.
460
+
461
+ <njs`> :-)
462
+
463
+ <njs`> appreciate the infodump, I really was failing to find the
464
+ details on git packs :-)
465
+
466
+And now you know the rest of the story.