@matheusbsbs / matheus-bagliano / commits / e9977b12fd

Documentation/technical: describe bitmap lookup table extension

When reading bitmap file, Git loads each and every bitmap one by one even if all the bitmaps are not required. A "bitmap lookup table" extension to the bitmap format can reduce the overhead of loading bitmaps which stores a list of bitmapped commit id pos (in the midx or pack, along with their offset and xor offset. This way Git can load only the necessary bitmaps without loading the previous bitmaps. Older versions of Git ignore the lookup table extension and don't throw any kind of warning or error while parsing the bitmap file. Add some information for the new "bitmap lookup table" extension in the bitmap-format documentation. Mentored-by: Taylor Blau <me@ttaylorr.com> Co-Mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com> Co-Authored-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com> Reviewed-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Abhradeep Chakraborty committed Aug 14, 2022 at 16:55 UTC e9977b12fdd64a9c5d7d0aff10709e043181ca7b
1 file changed +39
Documentation/technical/bitmap-format.txt
+39
@@ -72,6 +72,17 @@ MIDXs, both the bit-cache and rev-cache extensions are required.
72 pack/MIDX. The format and meaning of the name-hash is
73 described below.
74
75 + ** {empty}
76 + BITMAP_OPT_LOOKUP_TABLE (0x10): :::
77 + If present, the end of the bitmap file contains a table
78 + containing a list of `N` <commit_pos, offset, xor_row>
79 + triplets. The format and meaning of the table is described
80 + below.
81 ++
82 +NOTE: Unlike the xor_offset used to compress an individual bitmap,
83 +`xor_row` stores an *absolute* index into the lookup table, not a location
84 +relative to the current entry.
85 +
86 4-byte entry count (network byte order): ::
87 The total count of entries (bitmapped commits) in this bitmap index.
88
@@ -216,3 +227,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag.
227 If implementations want to choose a different hashing scheme, they are
228 free to do so, but MUST allocate a new header flag (because comparing
229 hashes made under two different schemes would be pointless).
230 +
231 +Commit lookup table
232 +-------------------
233 +
234 +If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)`
235 +bytes (preceding the name-hash cache and trailing hash) of the `.bitmap`
236 +file contains a lookup table specifying the information needed to get
237 +the desired bitmap from the entries without parsing previous unnecessary
238 +bitmaps.
239 +
240 +For a `.bitmap` containing `nr_entries` reachability bitmaps, the table
241 +contains a list of `nr_entries` <commit_pos, offset, xor_row> triplets
242 +(sorted in the ascending order of `commit_pos`). The content of i'th
243 +triplet is -
244 +
245 + * {empty}
246 + commit_pos (4 byte integer, network byte order): ::
247 + It stores the object position of a commit (in the midx or pack
248 + index).
249 +
250 + * {empty}
251 + offset (8 byte integer, network byte order): ::
252 + The offset from which that commit's bitmap can be read.
253 +
254 + * {empty}
255 + xor_row (4 byte integer, network byte order): ::
256 + The position of the triplet whose bitmap is used to compress
257 + this one, or `0xffffffff` if no such bitmap exists.