Documentation: describe pack idx v2
Lifted from the log message of c553ca25bd60dc9fd50b8bc7bd329601b81cee66 (pack-objects: learn about pack index version 2). Acked-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
linux@horizon.com committed
Dec 14, 2007 at 06:28 UTC
71362bd55280bb0f4aa67f89435308a4b26213b5
1 file changed
+61
-33
Documentation/technical/pack-format.txt
+61
-33
@@ -1,9 +1,9 @@
1
GIT pack format
2
===============
3
4
-= pack-*.pack file has the following format:
4
+= pack-*.pack files have the following format:
5
6
- - The header appears at the beginning and consists of the following:
6
+ - A header appears at the beginning and consists of the following:
7
8
4-byte signature:
9
The signature is: {'P', 'A', 'C', 'K'}
@@ -34,18 +34,14 @@ GIT pack format
34
35
- The trailer records 20-byte SHA1 checksum of all of the above.
36
37
-= pack-*.idx file has the following format:
37
+= Original (version 1) pack-*.idx files have the following format:
38
39
- The header consists of 256 4-byte network byte order
40
integers. N-th entry of this table records the number of
41
objects in the corresponding pack, the first byte of whose
42
- object name are smaller than N. This is called the
42
+ object name is less than or equal to N. This is called the
43
'first-level fan-out' table.
44
45
- Observation: we would need to extend this to an array of
46
- 8-byte integers to go beyond 4G objects per pack, but it is
47
- not strictly necessary.
48
-
45
- The header is followed by sorted 24-byte entries, one entry
46
per object in the pack. Each entry is:
47
@@ -55,10 +51,6 @@ GIT pack format
51
52
20-byte object name.
53
58
- Observation: we would definitely need to extend this to
59
- 8-byte integer plus 20-byte object name to handle a packfile
60
- that is larger than 4GB.
61
-
54
- The file is concluded with a trailer:
55
56
A copy of the 20-byte SHA1 checksum at the end of
@@ -68,31 +60,30 @@ GIT pack format
60
61
Pack Idx file:
62
71
- idx
72
- +--------------------------------+
73
- | fanout[0] = 2 |-.
74
- +--------------------------------+ |
63
+ -- +--------------------------------+
64
+fanout | fanout[0] = 2 (for example) |-.
65
+table +--------------------------------+ |
66
| fanout[1] | |
67
+--------------------------------+ |
68
| fanout[2] | |
69
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
79
- | fanout[255] | |
80
- +--------------------------------+ |
81
-main | offset | |
82
-index | object name 00XXXXXXXXXXXXXXXX | |
83
-table +--------------------------------+ |
84
- | offset | |
85
- | object name 00XXXXXXXXXXXXXXXX | |
86
- +--------------------------------+ |
87
- .-| offset |<+
88
- | | object name 01XXXXXXXXXXXXXXXX |
89
- | +--------------------------------+
90
- | | offset |
91
- | | object name 01XXXXXXXXXXXXXXXX |
92
- | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
93
- | | offset |
94
- | | object name FFXXXXXXXXXXXXXXXX |
95
- | +--------------------------------+
70
+ | fanout[255] = total objects |---.
71
+ -- +--------------------------------+ | |
72
+main | offset | | |
73
+index | object name 00XXXXXXXXXXXXXXXX | | |
74
+table +--------------------------------+ | |
75
+ | offset | | |
76
+ | object name 00XXXXXXXXXXXXXXXX | | |
77
+ +--------------------------------+<+ |
78
+ .-| offset | |
79
+ | | object name 01XXXXXXXXXXXXXXXX | |
80
+ | +--------------------------------+ |
81
+ | | offset | |
82
+ | | object name 01XXXXXXXXXXXXXXXX | |
83
+ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
84
+ | | offset | |
85
+ | | object name FFXXXXXXXXXXXXXXXX | |
86
+ --| +--------------------------------+<--+
87
trailer | | packfile checksum |
88
| +--------------------------------+
89
| | idxfile checksum |
@@ -116,3 +107,40 @@ Pack file entry: <+
107
20-byte base object name SHA1 (the size above is the
108
size of the delta data that follows).
109
delta data, deflated.
110
+
111
+
112
+= Version 2 pack-*.idx files support packs larger than 4 GiB, and
113
+ have some other reorganizations. They have the format:
114
+
115
+ - A 4-byte magic number '\377tOc' which is an unreasonable
116
+ fanout[0] value.
117
+
118
+ - A 4-byte version number (= 2)
119
+
120
+ - A 256-entry fan-out table just like v1.
121
+
122
+ - A table of sorted 20-byte SHA1 object names. These are
123
+ packed together without offset values to reduce the cache
124
+ footprint of the binary search for a specific object name.
125
+
126
+ - A table of 4-byte CRC32 values of the packed object data.
127
+ This is new in v2 so compressed data can be copied directly
128
+ from pack to pack during repacking withough undetected
129
+ data corruption.
130
+
131
+ - A table of 4-byte offset values (in network byte order).
132
+ These are usually 31-bit pack file offsets, but large
133
+ offsets are encoded as an index into the next table with
134
+ the msbit set.
135
+
136
+ - A table of 8-byte offset entries (empty for pack files less
137
+ than 2 GiB). Pack files are organized with heavily used
138
+ objects toward the front, so most object references should
139
+ not need to refer to this table.
140
+
141
+ - The same trailer as a v1 pack file:
142
+
143
+ A copy of the 20-byte SHA1 checksum at the end of
144
+ corresponding packfile.
145
+
146
+ 20-byte SHA1-checksum of all of the above.