@matheusbsbs / matheus-bagliano / commits / 917a54c017

Documentation: describe MIDX-based bitmaps

Update the technical documentation to describe the multi-pack bitmap format. This patch merely introduces the new format, and describes its high-level ideas. Git does not yet know how to read nor write these multi-pack variants, and so the subsequent patches will: - Introduce code to interpret multi-pack bitmaps, according to this document. - Then, introduce code to write multi-pack bitmaps from the 'git multi-pack-index write' sub-command. Finally, the implementation will gain tests in subsequent patches (as opposed to inline with the patch teaching Git how to write multi-pack bitmaps) to avoid a cyclic dependency. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Taylor Blau committed Aug 24, 2021 at 12:15 UTC 917a54c01755c3a0de7abea4b3afbc125aaf1d76
2 files changed +60 -21
Documentation/technical/bitmap-format.txt
+57 -14
@@ -1,6 +1,44 @@
1 GIT bitmap v1 format
2 ====================
3
4 +== Pack and multi-pack bitmaps
5 +
6 +Bitmaps store reachability information about the set of objects in a packfile,
7 +or a multi-pack index (MIDX). The former is defined obviously, and the latter is
8 +defined as the union of objects in packs contained in the MIDX.
9 +
10 +A bitmap may belong to either one pack, or the repository's multi-pack index (if
11 +it exists). A repository may have at most one bitmap.
12 +
13 +An object is uniquely described by its bit position within a bitmap:
14 +
15 + - If the bitmap belongs to a packfile, the __n__th bit corresponds to
16 + the __n__th object in pack order. For a function `offset` which maps
17 + objects to their byte offset within a pack, pack order is defined as
18 + follows:
19 +
20 + o1 <= o2 <==> offset(o1) <= offset(o2)
21 +
22 + - If the bitmap belongs to a MIDX, the __n__th bit corresponds to the
23 + __n__th object in MIDX order. With an additional function `pack` which
24 + maps objects to the pack they were selected from by the MIDX, MIDX order
25 + is defined as follows:
26 +
27 + o1 <= o2 <==> pack(o1) <= pack(o2) /\ offset(o1) <= offset(o2)
28 +
29 + The ordering between packs is done according to the MIDX's .rev file.
30 + Notably, the preferred pack sorts ahead of all other packs.
31 +
32 +The on-disk representation (described below) of a bitmap is the same regardless
33 +of whether or not that bitmap belongs to a packfile or a MIDX. The only
34 +difference is the interpretation of the bits, which is described above.
35 +
36 +Certain bitmap extensions are supported (see: Appendix B). No extensions are
37 +required for bitmaps corresponding to packfiles. For bitmaps that correspond to
38 +MIDXs, both the bit-cache and rev-cache extensions are required.
39 +
40 +== On-disk format
41 +
42 - A header appears at the beginning:
43
44 4-byte signature: {'B', 'I', 'T', 'M'}
@@ -14,17 +52,19 @@ GIT bitmap v1 format
52 The following flags are supported:
53
54 - BITMAP_OPT_FULL_DAG (0x1) REQUIRED
17 - This flag must always be present. It implies that the bitmap
18 - index has been generated for a packfile with full closure
19 - (i.e. where every single object in the packfile can find
20 - its parent links inside the same packfile). This is a
21 - requirement for the bitmap index format, also present in JGit,
22 - that greatly reduces the complexity of the implementation.
55 + This flag must always be present. It implies that the
56 + bitmap index has been generated for a packfile or
57 + multi-pack index (MIDX) with full closure (i.e. where
58 + every single object in the packfile/MIDX can find its
59 + parent links inside the same packfile/MIDX). This is a
60 + requirement for the bitmap index format, also present in
61 + JGit, that greatly reduces the complexity of the
62 + implementation.
63
64 - BITMAP_OPT_HASH_CACHE (0x4)
65 If present, the end of the bitmap file contains
66 `N` 32-bit name-hash values, one per object in the
27 - pack. The format and meaning of the name-hash is
67 + pack/MIDX. The format and meaning of the name-hash is
68 described below.
69
70 4-byte entry count (network byte order)
@@ -33,7 +73,8 @@ GIT bitmap v1 format
73
74 20-byte checksum
75
36 - The SHA1 checksum of the pack this bitmap index belongs to.
76 + The SHA1 checksum of the pack/MIDX this bitmap index
77 + belongs to.
78
79 - 4 EWAH bitmaps that act as type indexes
80
@@ -50,7 +91,7 @@ GIT bitmap v1 format
91 - Tags
92
93 In each bitmap, the `n`th bit is set to true if the `n`th object
53 - in the packfile is of that type.
94 + in the packfile or multi-pack index is of that type.
95
96 The obvious consequence is that the OR of all 4 bitmaps will result
97 in a full set (all bits set), and the AND of all 4 bitmaps will
@@ -62,8 +103,9 @@ GIT bitmap v1 format
103 Each entry contains the following:
104
105 - 4-byte object position (network byte order)
65 - The position **in the index for the packfile** where the
66 - bitmap for this commit is found.
106 + The position **in the index for the packfile or
107 + multi-pack index** where the bitmap for this commit is
108 + found.
109
110 - 1-byte XOR-offset
111 The xor offset used to compress this bitmap. For an entry
@@ -146,10 +188,11 @@ Name-hash cache
188 ---------------
189
190 If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains
149 -a cache of 32-bit values, one per object in the pack. The value at
191 +a cache of 32-bit values, one per object in the pack/MIDX. The value at
192 position `i` is the hash of the pathname at which the `i`th object
151 -(counting in index order) in the pack can be found. This can be fed
152 -into the delta heuristics to compare objects with similar pathnames.
193 +(counting in index or multi-pack index order) in the pack/MIDX can be found.
194 +This can be fed into the delta heuristics to compare objects with similar
195 +pathnames.
196
197 The hash algorithm used is:
198
Documentation/technical/multi-pack-index.txt
+3 -7
@@ -71,14 +71,10 @@ Future Work
71 still reducing the number of binary searches required for object
72 lookups.
73
74 -- The reachability bitmap is currently paired directly with a single
75 - packfile, using the pack-order as the object order to hopefully
76 - compress the bitmaps well using run-length encoding. This could be
77 - extended to pair a reachability bitmap with a multi-pack-index. If
78 - the multi-pack-index is extended to store a "stable object order"
74 +- If the multi-pack-index is extended to store a "stable object order"
75 (a function Order(hash) = integer that is constant for a given hash,
80 - even as the multi-pack-index is updated) then a reachability bitmap
81 - could point to a multi-pack-index and be updated independently.
76 + even as the multi-pack-index is updated) then MIDX bitmaps could be
77 + updated independently of the MIDX.
78
79 - Packfiles can be marked as "special" using empty files that share
80 the initial name but replace ".pack" with ".keep" or ".promisor".