Raw
1 git-pack-objects(1)
2 ===================
3
4 NAME
5 ----
6 git-pack-objects - Create a packed archive of objects
7
8
9 SYNOPSIS
10 --------
11 [verse]
12 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]
13 [--no-reuse-delta] [--delta-base-offset] [--non-empty]
14 [--local] [--incremental] [--window=<n>] [--depth=<n>]
15 [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
16 [--cruft] [--cruft-expiration=<time>]
17 [--stdout [--filter=<filter-spec>] | <base-name>]
18 [--shallow] [--keep-true-parents] [--[no-]sparse]
19 [--name-hash-version=<n>] [--path-walk] < <object-list>
20
21
22 DESCRIPTION
23 -----------
24 Reads list of objects from the standard input, and writes either one or
25 more packed archives with the specified base-name to disk, or a packed
26 archive to the standard output.
27
28 A packed archive is an efficient way to transfer a set of objects
29 between two repositories as well as an access efficient archival
30 format. In a packed archive, an object is either stored as a
31 compressed whole or as a difference from some other object.
32 The latter is often called a delta.
33
34 The packed archive format (.pack) is designed to be self-contained
35 so that it can be unpacked without any further information. Therefore,
36 each object that a delta depends upon must be present within the pack.
37
38 A pack index file (.idx) is generated for fast, random access to the
39 objects in the pack. Placing both the index file (.idx) and the packed
40 archive (.pack) in the pack/ subdirectory of $GIT_OBJECT_DIRECTORY (or
41 any of the directories on $GIT_ALTERNATE_OBJECT_DIRECTORIES)
42 enables Git to read from the pack archive.
43
44 The 'git unpack-objects' command can read the packed archive and
45 expand the objects contained in the pack into "one-file
46 one-object" format; this is typically done by the smart-pull
47 commands when a pack is created on-the-fly for efficient network
48 transport by their peers.
49
50
51 OPTIONS
52 -------
53 base-name::
54 Write into pairs of files (.pack and .idx), using
55 <base-name> to determine the name of the created file.
56 When this option is used, the two files in a pair are written in
57 <base-name>-<SHA-1>.{pack,idx} files. <SHA-1> is a hash
58 based on the pack content and is written to the standard
59 output of the command.
60
61 --stdout::
62 Write the pack contents (what would have been written to
63 .pack file) out to the standard output.
64
65 --revs::
66 Read the revision arguments from the standard input, instead of
67 individual object names. The revision arguments are processed
68 the same way as 'git rev-list' with the `--objects` flag
69 uses its `commit` arguments to build the list of objects it
70 outputs. The objects on the resulting list are packed.
71 Besides revisions, `--not` or `--shallow <SHA-1>` lines are
72 also accepted.
73
74 --unpacked::
75 This implies `--revs`. When processing the list of
76 revision arguments read from the standard input, limit
77 the objects packed to those that are not already packed.
78
79 --all::
80 This implies `--revs`. In addition to the list of
81 revision arguments read from the standard input, pretend
82 as if all refs under `refs/` are specified to be
83 included.
84
85 --include-tag::
86 Include unasked-for annotated tags if the object they
87 reference was included in the resulting packfile. This
88 can be useful to send new tags to native Git clients.
89
90 --stdin-packs[=<mode>]::
91 Read the basenames of packfiles (e.g., `pack-1234abcd.pack`)
92 from the standard input, instead of object names or revision
93 arguments. The resulting pack contains all objects listed in the
94 included packs (those not beginning with `^`), excluding any
95 objects listed in the excluded packs (beginning with `^`).
96 +
97 When `mode` is "follow" packs may additionally be prefixed with `!`,
98 indicating that they are excluded but not necessarily closed under
99 reachability. In addition to objects in included packs, the resulting
100 pack may include additional objects based on the following:
101 +
102 --
103 * If any packs are marked with `!`, then objects reachable from such
104 packs or included ones via objects outside of excluded-closed packs
105 will be included. In this case, all `^` packs are treated as closed
106 under reachability.
107 * Otherwise (if there are no `!` packs), objects within unlisted packs
108 will be included if those objects are (1) reachable from the
109 included packs, and (2) not found in any excluded packs.
110 --
111 +
112 This mode is useful, for example, to resurrect once-unreachable
113 objects found in cruft packs to generate packs which are closed under
114 reachability up to the boundary set by the excluded packs.
115 +
116 Incompatible with `--revs`, or options that imply `--revs` (such as
117 `--all`), with the exception of `--unpacked`, which is compatible.
118
119 --cruft::
120 Packs unreachable objects into a separate "cruft" pack, denoted
121 by the existence of a `.mtimes` file. Typically used by `git
122 repack --cruft`. Callers provide a list of pack names and
123 indicate which packs will remain in the repository, along with
124 which packs will be deleted (indicated by the `-` prefix). The
125 contents of the cruft pack are all objects not contained in the
126 surviving packs which have not exceeded the grace period (see
127 `--cruft-expiration` below), or which have exceeded the grace
128 period, but are reachable from an other object which hasn't.
129 +
130 When the input lists a pack containing all reachable objects (and lists
131 all other packs as pending deletion), the corresponding cruft pack will
132 contain all unreachable objects (with mtime newer than the
133 `--cruft-expiration`) along with any unreachable objects whose mtime is
134 older than the `--cruft-expiration`, but are reachable from an
135 unreachable object whose mtime is newer than the `--cruft-expiration`).
136 +
137 Incompatible with `--unpack-unreachable`, `--keep-unreachable`,
138 `--pack-loose-unreachable`, `--stdin-packs`, as well as any other
139 options which imply `--revs`.
140
141 --cruft-expiration=<approxidate>::
142 If specified, objects are eliminated from the cruft pack if they
143 have an mtime older than `<approxidate>`. If unspecified (and
144 given `--cruft`), then no objects are eliminated.
145
146 --window=<n>::
147 --depth=<n>::
148 These two options affect how the objects contained in
149 the pack are stored using delta compression. The
150 objects are first internally sorted by type, size and
151 optionally names and compared against the other objects
152 within --window to see if using delta compression saves
153 space. --depth limits the maximum delta depth; making
154 it too deep affects the performance on the unpacker
155 side, because delta data needs to be applied that many
156 times to get to the necessary object.
157 +
158 The default value for --window is 10 and --depth is 50. The maximum
159 depth is 4095.
160
161 --window-memory=<n>::
162 This option provides an additional limit on top of `--window`;
163 the window size will dynamically scale down so as to not take
164 up more than '<n>' bytes in memory. This is useful in
165 repositories with a mix of large and small objects to not run
166 out of memory with a large window, but still be able to take
167 advantage of the large window for the smaller objects. The
168 size can be suffixed with "k", "m", or "g".
169 `--window-memory=0` makes memory usage unlimited. The default
170 is taken from the `pack.windowMemory` configuration variable.
171
172 --max-pack-size=<n>::
173 In unusual scenarios, you may not be able to create files
174 larger than a certain size on your filesystem, and this option
175 can be used to tell the command to split the output packfile
176 into multiple independent packfiles, each not larger than the
177 given size. The size can be suffixed with
178 "k", "m", or "g". The minimum size allowed is limited to 1 MiB.
179 The default is unlimited, unless the config variable
180 `pack.packSizeLimit` is set. Note that this option may result in
181 a larger and slower repository; see the discussion in
182 `pack.packSizeLimit`.
183
184 --honor-pack-keep::
185 This flag causes an object already in a local pack that
186 has a .keep file to be ignored, even if it would have
187 otherwise been packed.
188
189 --keep-pack=<pack-name>::
190 This flag causes an object already in the given pack to be
191 ignored, even if it would have otherwise been
192 packed. `<pack-name>` is the pack file name without
193 leading directory (e.g. `pack-123.pack`). The option could be
194 specified multiple times to keep multiple packs.
195
196 --incremental::
197 This flag causes an object already in a pack to be ignored
198 even if it would have otherwise been packed.
199
200 --local::
201 This flag causes an object that is borrowed from an alternate
202 object store to be ignored even if it would have otherwise been
203 packed.
204
205 --non-empty::
206 Only create a packed archive if it would contain at
207 least one object.
208
209 --progress::
210 Progress status is reported on the standard error stream
211 by default when it is attached to a terminal, unless -q
212 is specified. This flag forces progress status even if
213 the standard error stream is not directed to a terminal.
214
215 --all-progress::
216 When --stdout is specified then progress report is
217 displayed during the object count and compression phases
218 but inhibited during the write-out phase. The reason is
219 that in some cases the output stream is directly linked
220 to another command which may wish to display progress
221 status of its own as it processes incoming pack data.
222 This flag is like --progress except that it forces progress
223 report for the write-out phase as well even if --stdout is
224 used.
225
226 --all-progress-implied::
227 This is used to imply --all-progress whenever progress display
228 is activated. Unlike --all-progress this flag doesn't actually
229 force any progress display by itself.
230
231 -q::
232 This flag makes the command not to report its progress
233 on the standard error stream.
234
235 --no-reuse-delta::
236 When creating a packed archive in a repository that
237 has existing packs, the command reuses existing deltas.
238 This sometimes results in a slightly suboptimal pack.
239 This flag tells the command not to reuse existing deltas
240 but compute them from scratch.
241
242 --no-reuse-object::
243 This flag tells the command not to reuse existing object data at all,
244 including non deltified object, forcing recompression of everything.
245 This implies --no-reuse-delta. Useful only in the obscure case where
246 wholesale enforcement of a different compression level on the
247 packed data is desired.
248
249 --compression=<n>::
250 Specifies compression level for newly-compressed data in the
251 generated pack. If not specified, pack compression level is
252 determined first by pack.compression, then by core.compression,
253 and defaults to -1, the zlib default, if neither is set.
254 Add --no-reuse-object if you want to force a uniform compression
255 level on all data no matter the source.
256
257 --sparse::
258 --no-sparse::
259 Toggle the "sparse" algorithm to determine which objects to include in
260 the pack, when combined with the "--revs" option. This algorithm
261 only walks trees that appear in paths that introduce new objects.
262 This can have significant performance benefits when computing
263 a pack to send a small change. However, it is possible that extra
264 objects are added to the pack-file if the included commits contain
265 certain types of direct renames. If this option is not included,
266 it defaults to the value of `pack.useSparse`, which is true unless
267 otherwise specified.
268
269 --thin::
270 Create a "thin" pack by omitting the common objects between a
271 sender and a receiver in order to reduce network transfer. This
272 option only makes sense in conjunction with --stdout.
273 +
274 Note: A thin pack violates the packed archive format by omitting
275 required objects and is thus unusable by Git without making it
276 self-contained. Use `git index-pack --fix-thin`
277 (see linkgit:git-index-pack[1]) to restore the self-contained property.
278
279 --shallow::
280 Optimize a pack that will be provided to a client with a shallow
281 repository. This option, combined with --thin, can result in a
282 smaller pack at the cost of speed.
283
284 --delta-base-offset::
285 A packed archive can express the base object of a delta as
286 either a 20-byte object name or as an offset in the
287 stream, but ancient versions of Git don't understand the
288 latter. By default, 'git pack-objects' only uses the
289 former format for better compatibility. This option
290 allows the command to use the latter format for
291 compactness. Depending on the average delta chain
292 length, this option typically shrinks the resulting
293 packfile by 3-5 per-cent.
294 +
295 Note: Porcelain commands such as `git gc` (see linkgit:git-gc[1]),
296 `git repack` (see linkgit:git-repack[1]) pass this option by default
297 in modern Git when they put objects in your repository into pack files.
298 So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.
299
300 --threads=<n>::
301 Specifies the number of threads to spawn when searching for best
302 delta matches. This requires that pack-objects be compiled with
303 pthreads otherwise this option is ignored with a warning.
304 This is meant to reduce packing time on multiprocessor machines.
305 The required amount of memory for the delta search window is
306 however multiplied by the number of threads.
307 Specifying 0 will cause Git to auto-detect the number of CPU's
308 and set the number of threads accordingly.
309
310 --index-version=<version>[,<offset>]::
311 This is intended to be used by the test suite only. It allows
312 to force the version for the generated pack index, and to force
313 64-bit index entries on objects located above the given offset.
314
315 --keep-true-parents::
316 With this option, parents that are hidden by grafts are packed
317 nevertheless.
318
319 --filter=<filter-spec>::
320 Omits certain objects (usually blobs) from the resulting
321 packfile. See linkgit:git-rev-list[1] for valid
322 `<filter-spec>` forms.
323
324 --no-filter::
325 Turns off any previous `--filter=` argument.
326
327 --missing=<missing-action>::
328 A debug option to help with future "partial clone" development.
329 This option specifies how missing objects are handled.
330 +
331 The form '--missing=error' requests that pack-objects stop with an error if
332 a missing object is encountered. If the repository is a partial clone, an
333 attempt to fetch missing objects will be made before declaring them missing.
334 This is the default action.
335 +
336 The form '--missing=allow-any' will allow object traversal to continue
337 if a missing object is encountered. No fetch of a missing object will occur.
338 Missing objects will silently be omitted from the results.
339 +
340 The form '--missing=allow-promisor' is like 'allow-any', but will only
341 allow object traversal to continue for EXPECTED promisor missing objects.
342 No fetch of a missing object will occur. An unexpected missing object will
343 raise an error.
344
345 --exclude-promisor-objects::
346 Omit objects that are known to be in the promisor remote. (This
347 option has the purpose of operating only on locally created objects,
348 so that when we repack, we still maintain a distinction between
349 locally created objects [without .promisor] and objects from the
350 promisor remote [with .promisor].) This is used with partial clone.
351
352 --keep-unreachable::
353 Objects unreachable from the refs in packs named with
354 --unpacked= option are added to the resulting pack, in
355 addition to the reachable objects that are not in packs marked
356 with *.keep files. This implies `--revs`.
357
358 --pack-loose-unreachable::
359 Pack unreachable loose objects (and their loose counterparts
360 removed). This implies `--revs`.
361
362 --unpack-unreachable::
363 Keep unreachable objects in loose form. This implies `--revs`.
364
365 --delta-islands::
366 Restrict delta matches based on "islands". See DELTA ISLANDS
367 below.
368
369 --name-hash-version=<n>::
370 While performing delta compression, Git groups objects that may be
371 similar based on heuristics using the path to that object. While
372 grouping objects by an exact path match is good for paths with
373 many versions, there are benefits for finding delta pairs across
374 different full paths. Git collects objects by type and then by a
375 "name hash" of the path and then by size, hoping to group objects
376 that will compress well together.
377 +
378 The default name hash version is `1`, which prioritizes hash locality by
379 considering the final bytes of the path as providing the maximum magnitude
380 to the hash function. This version excels at distinguishing short paths
381 and finding renames across directories. However, the hash function depends
382 primarily on the final 16 bytes of the path. If there are many paths in
383 the repo that have the same final 16 bytes and differ only by parent
384 directory, then this name-hash may lead to too many collisions and cause
385 poor results. At the moment, this version is required when writing
386 reachability bitmap files with `--write-bitmap-index`.
387 +
388 The name hash version `2` has similar locality features as version `1`,
389 except it considers each path component separately and overlays the hashes
390 with a shift. This still prioritizes the final bytes of the path, but also
391 "salts" the lower bits of the hash using the parent directory names. This
392 method allows for some of the locality benefits of version `1` while
393 breaking most of the collisions from a similarly-named file appearing in
394 many different directories. At the moment, this version is not allowed
395 when writing reachability bitmap files with `--write-bitmap-index` and it
396 will be automatically changed to version `1`.
397
398 --path-walk::
399 Perform compression by first organizing objects by path, then a
400 second pass that compresses across paths as normal. This has the
401 potential to improve delta compression especially in the presence
402 of filenames that cause collisions in Git's default name-hash
403 algorithm.
404 +
405 When `--use-bitmap-index` is specified with `--path-walk`, a successful
406 bitmap traversal is used for object enumeration, with path-walk
407 remaining as the fallback traversal when the bitmap cannot satisfy the
408 request. The `--path-walk` option supports the `--filter=<spec>` forms
409 `blob:none`, `blob:limit=<n>`, `tree:0`, `object:type=<type>`, and
410 `sparse:<oid>`. These supported filter types can be combined with the
411 `combine:<spec>+<spec>` form.
412
413
414 DELTA ISLANDS
415 -------------
416
417 When possible, `pack-objects` tries to reuse existing on-disk deltas to
418 avoid having to search for new ones on the fly. This is an important
419 optimization for serving fetches, because it means the server can avoid
420 inflating most objects at all and just send the bytes directly from
421 disk. This optimization can't work when an object is stored as a delta
422 against a base which the receiver does not have (and which we are not
423 already sending). In that case the server "breaks" the delta and has to
424 find a new one, which has a high CPU cost. Therefore it's important for
425 performance that the set of objects in on-disk delta relationships match
426 what a client would fetch.
427
428 In a normal repository, this tends to work automatically. The objects
429 are mostly reachable from the branches and tags, and that's what clients
430 fetch. Any deltas we find on the server are likely to be between objects
431 the client has or will have.
432
433 But in some repository setups, you may have several related but separate
434 groups of ref tips, with clients tending to fetch those groups
435 independently. For example, imagine that you are hosting several "forks"
436 of a repository in a single shared object store, and letting clients
437 view them as separate repositories through `GIT_NAMESPACE` or separate
438 repos using the alternates mechanism. A naive repack may find that the
439 optimal delta for an object is against a base that is only found in
440 another fork. But when a client fetches, they will not have the base
441 object, and we'll have to find a new delta on the fly.
442
443 A similar situation may exist if you have many refs outside of
444 `refs/heads/` and `refs/tags/` that point to related objects (e.g.,
445 `refs/pull` or `refs/changes` used by some hosting providers). By
446 default, clients fetch only heads and tags, and deltas against objects
447 found only in those other groups cannot be sent as-is.
448
449 Delta islands solve this problem by allowing you to group your refs into
450 distinct "islands". Pack-objects computes which objects are reachable
451 from which islands, and refuses to make a delta from an object `A`
452 against a base which is not present in all of `A`'s islands. This
453 results in slightly larger packs (because we miss some delta
454 opportunities), but guarantees that a fetch of one island will not have
455 to recompute deltas on the fly due to crossing island boundaries.
456
457 When repacking with delta islands the delta window tends to get
458 clogged with candidates that are forbidden by the config. Repacking
459 with a big --window helps (and doesn't take as long as it otherwise
460 might because we can reject some object pairs based on islands before
461 doing any computation on the content).
462
463 Islands are configured via the `pack.island` option, which can be
464 specified multiple times. Each value is a left-anchored regular
465 expressions matching refnames. For example:
466
467 -------------------------------------------
468 [pack]
469 island = refs/heads/
470 island = refs/tags/
471 -------------------------------------------
472
473 puts heads and tags into an island (whose name is the empty string; see
474 below for more on naming). Any refs which do not match those regular
475 expressions (e.g., `refs/pull/123`) is not in any island. Any object
476 which is reachable only from `refs/pull/` (but not heads or tags) is
477 therefore not a candidate to be used as a base for `refs/heads/`.
478
479 Refs are grouped into islands based on their "names", and two regexes
480 that produce the same name are considered to be in the same
481 island. The names are computed from the regexes by concatenating any
482 capture groups from the regex, with a '-' dash in between. (And if
483 there are no capture groups, then the name is the empty string, as in
484 the above example.) This allows you to create arbitrary numbers of
485 islands. Only up to 14 such capture groups are supported though.
486
487 For example, imagine you store the refs for each fork in
488 `refs/virtual/ID`, where `ID` is a numeric identifier. You might then
489 configure:
490
491 -------------------------------------------
492 [pack]
493 island = refs/virtual/([0-9]+)/heads/
494 island = refs/virtual/([0-9]+)/tags/
495 island = refs/virtual/([0-9]+)/(pull)/
496 -------------------------------------------
497
498 That puts the heads and tags for each fork in their own island (named
499 "1234" or similar), and the pull refs for each go into their own
500 "1234-pull".
501
502 Note that we pick a single island for each regex to go into, using "last
503 one wins" ordering (which allows repo-specific config to take precedence
504 over user-wide config, and so forth).
505
506
507 CONFIGURATION
508 -------------
509
510 Various configuration variables affect packing, see
511 linkgit:git-config[1] (search for "pack" and "delta").
512
513 Notably, delta compression is not used on objects larger than the
514 `core.bigFileThreshold` configuration variable and on files with the
515 attribute `delta` set to false.
516
517 SEE ALSO
518 --------
519 linkgit:git-rev-list[1]
520 linkgit:git-repack[1]
521 linkgit:git-prune-packed[1]
522
523 GIT
524 ---
525 Part of the linkgit:git[1] suite