Documentation/technical: describe multi-pack reverse indexes
As a prerequisite to implementing multi-pack bitmaps, motivate and describe the format and ordering of the multi-pack reverse index. The subsequent patch will implement reading this format, and the patch after that will implement writing it while producing a multi-pack index. Co-authored-by: Jeff King <peff@peff.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Taylor Blau committed
Mar 30, 2021 at 11:04 UTC
b25fd24c00a6670a940d998287736c5abe4ce09d
1 file changed
+83
Documentation/technical/pack-format.txt
+83
@@ -379,3 +379,86 @@ CHUNK DATA:
379
TRAILER:
380
381
Index checksum of the above contents.
382
+
383
+== multi-pack-index reverse indexes
384
+
385
+Similar to the pack-based reverse index, the multi-pack index can also
386
+be used to generate a reverse index.
387
+
388
+Instead of mapping between offset, pack-, and index position, this
389
+reverse index maps between an object's position within the MIDX, and
390
+that object's position within a pseudo-pack that the MIDX describes
391
+(i.e., the ith entry of the multi-pack reverse index holds the MIDX
392
+position of ith object in pseudo-pack order).
393
+
394
+To clarify the difference between these orderings, consider a multi-pack
395
+reachability bitmap (which does not yet exist, but is what we are
396
+building towards here). Each bit needs to correspond to an object in the
397
+MIDX, and so we need an efficient mapping from bit position to MIDX
398
+position.
399
+
400
+One solution is to let bits occupy the same position in the oid-sorted
401
+index stored by the MIDX. But because oids are effectively random, their
402
+resulting reachability bitmaps would have no locality, and thus compress
403
+poorly. (This is the reason that single-pack bitmaps use the pack
404
+ordering, and not the .idx ordering, for the same purpose.)
405
+
406
+So we'd like to define an ordering for the whole MIDX based around
407
+pack ordering, which has far better locality (and thus compresses more
408
+efficiently). We can think of a pseudo-pack created by the concatenation
409
+of all of the packs in the MIDX. E.g., if we had a MIDX with three packs
410
+(a, b, c), with 10, 15, and 20 objects respectively, we can imagine an
411
+ordering of the objects like:
412
+
413
+ |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|
414
+
415
+where the ordering of the packs is defined by the MIDX's pack list,
416
+and then the ordering of objects within each pack is the same as the
417
+order in the actual packfile.
418
+
419
+Given the list of packs and their counts of objects, you can
420
+naïvely reconstruct that pseudo-pack ordering (e.g., the object at
421
+position 27 must be (c,1) because packs "a" and "b" consumed 25 of the
422
+slots). But there's a catch. Objects may be duplicated between packs, in
423
+which case the MIDX only stores one pointer to the object (and thus we'd
424
+want only one slot in the bitmap).
425
+
426
+Callers could handle duplicates themselves by reading objects in order
427
+of their bit-position, but that's linear in the number of objects, and
428
+much too expensive for ordinary bitmap lookups. Building a reverse index
429
+solves this, since it is the logical inverse of the index, and that
430
+index has already removed duplicates. But, building a reverse index on
431
+the fly can be expensive. Since we already have an on-disk format for
432
+pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,
433
+too.
434
+
435
+Objects from the MIDX are ordered as follows to string together the
436
+pseudo-pack. Let `pack(o)` return the pack from which `o` was selected
437
+by the MIDX, and define an ordering of packs based on their numeric ID
438
+(as stored by the MIDX). Let `offset(o)` return the object offset of `o`
439
+within `pack(o)`. Then, compare `o1` and `o2` as follows:
440
+
441
+ - If one of `pack(o1)` and `pack(o2)` is preferred and the other
442
+ is not, then the preferred one sorts first.
443
++
444
+(This is a detail that allows the MIDX bitmap to determine which
445
+pack should be used by the pack-reuse mechanism, since it can ask
446
+the MIDX for the pack containing the object at bit position 0).
447
+
448
+ - If `pack(o1) ≠ pack(o2)`, then sort the two objects in descending
449
+ order based on the pack ID.
450
+
451
+ - Otherwise, `pack(o1) = pack(o2)`, and the objects are sorted in
452
+ pack-order (i.e., `o1` sorts ahead of `o2` exactly when `offset(o1)
453
+ < offset(o2)`).
454
+
455
+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of
456
+objects in packs stored by the MIDX, laid out in pack order, and the
457
+packs arranged in MIDX order (with the preferred pack coming first).
458
+
459
+Finally, note that the MIDX's reverse index is not stored as a chunk in
460
+the multi-pack-index itself. This is done because the reverse index
461
+includes the checksum of the pack or MIDX to which it belongs, which
462
+makes it impossible to write in the MIDX. To avoid races when rewriting
463
+the MIDX, a MIDX reverse index includes the MIDX's checksum in its
464
+filename (e.g., `multi-pack-index-xyz.rev`).