Raw
1 git-pack-objects(1)
2 ===================
3
4 NAME
5 ----
6 git-pack-objects - Create a packed archive of objects
7
8
9 SYNOPSIS
10 --------
11 [verse]
12 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]
13 [--no-reuse-delta] [--delta-base-offset] [--no-ref-delta]
14 [--non-empty]
15 [--local] [--incremental] [--window=<n>] [--depth=<n>]
16 [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
17 [--cruft] [--cruft-expiration=<time>]
18 [--stdout [--filter=<filter-spec>] | <base-name>]
19 [--shallow] [--keep-true-parents] [--[no-]sparse]
20 [--name-hash-version=<n>] [--path-walk] < <object-list>
21
22
23 DESCRIPTION
24 -----------
25 Reads list of objects from the standard input, and writes either one or
26 more packed archives with the specified base-name to disk, or a packed
27 archive to the standard output.
28
29 A packed archive is an efficient way to transfer a set of objects
30 between two repositories as well as an access efficient archival
31 format. In a packed archive, an object is either stored as a
32 compressed whole or as a difference from some other object.
33 The latter is often called a delta.
34
35 The packed archive format (.pack) is designed to be self-contained
36 so that it can be unpacked without any further information. Therefore,
37 each object that a delta depends upon must be present within the pack.
38
39 A pack index file (.idx) is generated for fast, random access to the
40 objects in the pack. Placing both the index file (.idx) and the packed
41 archive (.pack) in the pack/ subdirectory of $GIT_OBJECT_DIRECTORY (or
42 any of the directories on $GIT_ALTERNATE_OBJECT_DIRECTORIES)
43 enables Git to read from the pack archive.
44
45 The 'git unpack-objects' command can read the packed archive and
46 expand the objects contained in the pack into "one-file
47 one-object" format; this is typically done by the smart-pull
48 commands when a pack is created on-the-fly for efficient network
49 transport by their peers.
50
51
52 OPTIONS
53 -------
54 base-name::
55 Write into pairs of files (.pack and .idx), using
56 <base-name> to determine the name of the created file.
57 When this option is used, the two files in a pair are written in
58 <base-name>-<SHA-1>.{pack,idx} files. <SHA-1> is a hash
59 based on the pack content and is written to the standard
60 output of the command.
61
62 --stdout::
63 Write the pack contents (what would have been written to
64 .pack file) out to the standard output.
65
66 --revs::
67 Read the revision arguments from the standard input, instead of
68 individual object names. The revision arguments are processed
69 the same way as 'git rev-list' with the `--objects` flag
70 uses its `commit` arguments to build the list of objects it
71 outputs. The objects on the resulting list are packed.
72 Besides revisions, `--not` or `--shallow <SHA-1>` lines are
73 also accepted.
74
75 --unpacked::
76 This implies `--revs`. When processing the list of
77 revision arguments read from the standard input, limit
78 the objects packed to those that are not already packed.
79
80 --all::
81 This implies `--revs`. In addition to the list of
82 revision arguments read from the standard input, pretend
83 as if all refs under `refs/` are specified to be
84 included.
85
86 --include-tag::
87 Include unasked-for annotated tags if the object they
88 reference was included in the resulting packfile. This
89 can be useful to send new tags to native Git clients.
90
91 --stdin-packs[=<mode>]::
92 Read the basenames of packfiles (e.g., `pack-1234abcd.pack`)
93 from the standard input, instead of object names or revision
94 arguments. The resulting pack contains all objects listed in the
95 included packs (those not beginning with `^`), excluding any
96 objects listed in the excluded packs (beginning with `^`).
97 +
98 When `mode` is "follow" packs may additionally be prefixed with `!`,
99 indicating that they are excluded but not necessarily closed under
100 reachability. In addition to objects in included packs, the resulting
101 pack may include additional objects based on the following:
102 +
103 --
104 * If any packs are marked with `!`, then objects reachable from such
105 packs or included ones via objects outside of excluded-closed packs
106 will be included. In this case, all `^` packs are treated as closed
107 under reachability.
108 * Otherwise (if there are no `!` packs), objects within unlisted packs
109 will be included if those objects are (1) reachable from the
110 included packs, and (2) not found in any excluded packs.
111 --
112 +
113 This mode is useful, for example, to resurrect once-unreachable
114 objects found in cruft packs to generate packs which are closed under
115 reachability up to the boundary set by the excluded packs.
116 +
117 When `mode` is "follow-reachable", the same pack prefixes are recognized
118 as in "follow" (`!` for excluded-open, `^` for excluded-closed). However,
119 instead of including all objects from included packs, only objects that
120 are reachable from reference tips AND belong to an included pack (or are
121 reachable from a commit in one) are included. Objects in excluded-open
122 packs are traversed but not included; objects in excluded-closed packs
123 halt the traversal.
124 +
125 This mode is designed for geometric repacking with cruft packs, where
126 the output pack should contain only reachable objects so that unreachable
127 ones can be collected separately.
128 +
129 When `--unpacked` is given alongside `--stdin-packs=follow-reachable`,
130 reachable loose objects are also included in the output pack, while
131 unreachable loose objects are left alone. This includes both loose
132 commits and annotated tag objects.
133 +
134 Incompatible with `--revs`, or options that imply `--revs` (such as
135 `--all`), with the exception of `--unpacked`, which is compatible.
136
137 --refs-snapshot=<path>::
138 When used with `--stdin-packs=follow-reachable`, read reference
139 tips from `<path>` instead of iterating live references. The file
140 format is one hex object ID per line, with an optional `+` prefix
141 (for preferred bitmap commits). This ensures a consistent view of
142 references when the same snapshot is shared with other tools (e.g.,
143 the MIDX bitmap writer).
144
145 --cruft::
146 Packs unreachable objects into a separate "cruft" pack, denoted
147 by the existence of a `.mtimes` file. Typically used by `git
148 repack --cruft`. Callers provide a list of pack names and
149 indicate which packs will remain in the repository, along with
150 which packs will be deleted (indicated by the `-` prefix). The
151 contents of the cruft pack are all objects not contained in the
152 surviving packs which have not exceeded the grace period (see
153 `--cruft-expiration` below), or which have exceeded the grace
154 period, but are reachable from an other object which hasn't.
155 +
156 When the input lists a pack containing all reachable objects (and lists
157 all other packs as pending deletion), the corresponding cruft pack will
158 contain all unreachable objects (with mtime newer than the
159 `--cruft-expiration`) along with any unreachable objects whose mtime is
160 older than the `--cruft-expiration`, but are reachable from an
161 unreachable object whose mtime is newer than the `--cruft-expiration`).
162 +
163 Incompatible with `--unpack-unreachable`, `--keep-unreachable`,
164 `--pack-loose-unreachable`, `--stdin-packs`, as well as any other
165 options which imply `--revs`.
166
167 --cruft-expiration=<approxidate>::
168 If specified, objects are eliminated from the cruft pack if they
169 have an mtime older than `<approxidate>`. If unspecified (and
170 given `--cruft`), then no objects are eliminated.
171
172 --window=<n>::
173 --depth=<n>::
174 These two options affect how the objects contained in
175 the pack are stored using delta compression. The
176 objects are first internally sorted by type, size and
177 optionally names and compared against the other objects
178 within --window to see if using delta compression saves
179 space. --depth limits the maximum delta depth; making
180 it too deep affects the performance on the unpacker
181 side, because delta data needs to be applied that many
182 times to get to the necessary object.
183 +
184 The default value for --window is 10 and --depth is 50. The maximum
185 depth is 4095.
186
187 --window-memory=<n>::
188 This option provides an additional limit on top of `--window`;
189 the window size will dynamically scale down so as to not take
190 up more than '<n>' bytes in memory. This is useful in
191 repositories with a mix of large and small objects to not run
192 out of memory with a large window, but still be able to take
193 advantage of the large window for the smaller objects. The
194 size can be suffixed with "k", "m", or "g".
195 `--window-memory=0` makes memory usage unlimited. The default
196 is taken from the `pack.windowMemory` configuration variable.
197
198 --max-pack-size=<n>::
199 In unusual scenarios, you may not be able to create files
200 larger than a certain size on your filesystem, and this option
201 can be used to tell the command to split the output packfile
202 into multiple independent packfiles, each not larger than the
203 given size. The size can be suffixed with
204 "k", "m", or "g". The minimum size allowed is limited to 1 MiB.
205 The default is unlimited, unless the config variable
206 `pack.packSizeLimit` is set. Note that this option may result in
207 a larger and slower repository; see the discussion in
208 `pack.packSizeLimit`.
209
210 --honor-pack-keep::
211 This flag causes an object already in a local pack that
212 has a .keep file to be ignored, even if it would have
213 otherwise been packed.
214
215 --keep-pack=<pack-name>::
216 This flag causes an object already in the given pack to be
217 ignored, even if it would have otherwise been
218 packed. `<pack-name>` is the pack file name without
219 leading directory (e.g. `pack-123.pack`). The option could be
220 specified multiple times to keep multiple packs.
221
222 --incremental::
223 This flag causes an object already in a pack to be ignored
224 even if it would have otherwise been packed.
225
226 --local::
227 This flag causes an object that is borrowed from an alternate
228 object store to be ignored even if it would have otherwise been
229 packed.
230
231 --non-empty::
232 Only create a packed archive if it would contain at
233 least one object.
234
235 --progress::
236 Progress status is reported on the standard error stream
237 by default when it is attached to a terminal, unless -q
238 is specified. This flag forces progress status even if
239 the standard error stream is not directed to a terminal.
240
241 --all-progress::
242 When --stdout is specified then progress report is
243 displayed during the object count and compression phases
244 but inhibited during the write-out phase. The reason is
245 that in some cases the output stream is directly linked
246 to another command which may wish to display progress
247 status of its own as it processes incoming pack data.
248 This flag is like --progress except that it forces progress
249 report for the write-out phase as well even if --stdout is
250 used.
251
252 --all-progress-implied::
253 This is used to imply --all-progress whenever progress display
254 is activated. Unlike --all-progress this flag doesn't actually
255 force any progress display by itself.
256
257 -q::
258 This flag makes the command not to report its progress
259 on the standard error stream.
260
261 --no-reuse-delta::
262 When creating a packed archive in a repository that
263 has existing packs, the command reuses existing deltas.
264 This sometimes results in a slightly suboptimal pack.
265 This flag tells the command not to reuse existing deltas
266 but compute them from scratch.
267
268 --no-reuse-object::
269 This flag tells the command not to reuse existing object data at all,
270 including non deltified object, forcing recompression of everything.
271 This implies --no-reuse-delta. Useful only in the obscure case where
272 wholesale enforcement of a different compression level on the
273 packed data is desired.
274
275 --compression=<n>::
276 Specifies compression level for newly-compressed data in the
277 generated pack. If not specified, pack compression level is
278 determined first by pack.compression, then by core.compression,
279 and defaults to -1, the zlib default, if neither is set.
280 Add --no-reuse-object if you want to force a uniform compression
281 level on all data no matter the source.
282
283 --sparse::
284 --no-sparse::
285 Toggle the "sparse" algorithm to determine which objects to include in
286 the pack, when combined with the "--revs" option. This algorithm
287 only walks trees that appear in paths that introduce new objects.
288 This can have significant performance benefits when computing
289 a pack to send a small change. However, it is possible that extra
290 objects are added to the pack-file if the included commits contain
291 certain types of direct renames. If this option is not included,
292 it defaults to the value of `pack.useSparse`, which is true unless
293 otherwise specified.
294
295 --thin::
296 Create a "thin" pack by omitting the common objects between a
297 sender and a receiver in order to reduce network transfer. This
298 option only makes sense in conjunction with --stdout.
299 +
300 Note: A thin pack violates the packed archive format by omitting
301 required objects and is thus unusable by Git without making it
302 self-contained. Use `git index-pack --fix-thin`
303 (see linkgit:git-index-pack[1]) to restore the self-contained property.
304
305 --shallow::
306 Optimize a pack that will be provided to a client with a shallow
307 repository. This option, combined with --thin, can result in a
308 smaller pack at the cost of speed.
309
310 --delta-base-offset::
311 A packed archive can express the base object of a delta as
312 either a 20-byte object name or as an offset in the
313 stream, but ancient versions of Git don't understand the
314 latter. By default, 'git pack-objects' only uses the
315 former format for better compatibility. This option
316 allows the command to use the latter format for
317 compactness. Depending on the average delta chain
318 length, this option typically shrinks the resulting
319 packfile by 3-5 per-cent.
320 +
321 Note: Porcelain commands such as `git gc` (see linkgit:git-gc[1]),
322 `git repack` (see linkgit:git-repack[1]) pass this option by default
323 in modern Git when they put objects in your repository into pack files.
324 So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.
325
326 --no-ref-delta::
327 Do not emit deltas which represent their base by their literal
328 object ID. This is independent of `--delta-base-offset`;
329 without that option, no deltas are emitted.
330
331 --threads=<n>::
332 Specifies the number of threads to spawn when searching for best
333 delta matches. This requires that pack-objects be compiled with
334 pthreads otherwise this option is ignored with a warning.
335 This is meant to reduce packing time on multiprocessor machines.
336 The required amount of memory for the delta search window is
337 however multiplied by the number of threads.
338 Specifying 0 will cause Git to auto-detect the number of CPU's
339 and set the number of threads accordingly.
340
341 --index-version=<version>[,<offset>]::
342 This is intended to be used by the test suite only. It allows
343 to force the version for the generated pack index, and to force
344 64-bit index entries on objects located above the given offset.
345
346 --keep-true-parents::
347 With this option, parents that are hidden by grafts are packed
348 nevertheless.
349
350 --filter=<filter-spec>::
351 Omits certain objects (usually blobs) from the resulting
352 packfile. See linkgit:git-rev-list[1] for valid
353 `<filter-spec>` forms.
354
355 --no-filter::
356 Turns off any previous `--filter=` argument.
357
358 --missing=<missing-action>::
359 A debug option to help with future "partial clone" development.
360 This option specifies how missing objects are handled.
361 +
362 The form '--missing=error' requests that pack-objects stop with an error if
363 a missing object is encountered. If the repository is a partial clone, an
364 attempt to fetch missing objects will be made before declaring them missing.
365 This is the default action.
366 +
367 The form '--missing=allow-any' will allow object traversal to continue
368 if a missing object is encountered. No fetch of a missing object will occur.
369 Missing objects will silently be omitted from the results.
370 +
371 The form '--missing=allow-promisor' is like 'allow-any', but will only
372 allow object traversal to continue for EXPECTED promisor missing objects.
373 No fetch of a missing object will occur. An unexpected missing object will
374 raise an error.
375
376 --exclude-promisor-objects::
377 Omit objects that are known to be in the promisor remote. (This
378 option has the purpose of operating only on locally created objects,
379 so that when we repack, we still maintain a distinction between
380 locally created objects [without .promisor] and objects from the
381 promisor remote [with .promisor].) This is used with partial clone.
382
383 --keep-unreachable::
384 Objects unreachable from the refs in packs named with
385 --unpacked= option are added to the resulting pack, in
386 addition to the reachable objects that are not in packs marked
387 with *.keep files. This implies `--revs`.
388
389 --pack-loose-unreachable::
390 Pack unreachable loose objects (and their loose counterparts
391 removed). This implies `--revs`.
392
393 --unpack-unreachable::
394 Keep unreachable objects in loose form. This implies `--revs`.
395
396 --delta-islands::
397 Restrict delta matches based on "islands". See DELTA ISLANDS
398 below.
399
400 --name-hash-version=<n>::
401 While performing delta compression, Git groups objects that may be
402 similar based on heuristics using the path to that object. While
403 grouping objects by an exact path match is good for paths with
404 many versions, there are benefits for finding delta pairs across
405 different full paths. Git collects objects by type and then by a
406 "name hash" of the path and then by size, hoping to group objects
407 that will compress well together.
408 +
409 The default name hash version is `1`, which prioritizes hash locality by
410 considering the final bytes of the path as providing the maximum magnitude
411 to the hash function. This version excels at distinguishing short paths
412 and finding renames across directories. However, the hash function depends
413 primarily on the final 16 bytes of the path. If there are many paths in
414 the repo that have the same final 16 bytes and differ only by parent
415 directory, then this name-hash may lead to too many collisions and cause
416 poor results. At the moment, this version is required when writing
417 reachability bitmap files with `--write-bitmap-index`.
418 +
419 The name hash version `2` has similar locality features as version `1`,
420 except it considers each path component separately and overlays the hashes
421 with a shift. This still prioritizes the final bytes of the path, but also
422 "salts" the lower bits of the hash using the parent directory names. This
423 method allows for some of the locality benefits of version `1` while
424 breaking most of the collisions from a similarly-named file appearing in
425 many different directories. At the moment, this version is not allowed
426 when writing reachability bitmap files with `--write-bitmap-index` and it
427 will be automatically changed to version `1`.
428
429 --path-walk::
430 Perform compression by first organizing objects by path, then a
431 second pass that compresses across paths as normal. This has the
432 potential to improve delta compression especially in the presence
433 of filenames that cause collisions in Git's default name-hash
434 algorithm.
435 +
436 When `--use-bitmap-index` is specified with `--path-walk`, a successful
437 bitmap traversal is used for object enumeration, with path-walk
438 remaining as the fallback traversal when the bitmap cannot satisfy the
439 request. The `--path-walk` option supports the `--filter=<spec>` forms
440 `blob:none`, `blob:limit=<n>`, `tree:0`, `object:type=<type>`, and
441 `sparse:<oid>`. These supported filter types can be combined with the
442 `combine:<spec>+<spec>` form.
443
444
445 DELTA ISLANDS
446 -------------
447
448 When possible, `pack-objects` tries to reuse existing on-disk deltas to
449 avoid having to search for new ones on the fly. This is an important
450 optimization for serving fetches, because it means the server can avoid
451 inflating most objects at all and just send the bytes directly from
452 disk. This optimization can't work when an object is stored as a delta
453 against a base which the receiver does not have (and which we are not
454 already sending). In that case the server "breaks" the delta and has to
455 find a new one, which has a high CPU cost. Therefore it's important for
456 performance that the set of objects in on-disk delta relationships match
457 what a client would fetch.
458
459 In a normal repository, this tends to work automatically. The objects
460 are mostly reachable from the branches and tags, and that's what clients
461 fetch. Any deltas we find on the server are likely to be between objects
462 the client has or will have.
463
464 But in some repository setups, you may have several related but separate
465 groups of ref tips, with clients tending to fetch those groups
466 independently. For example, imagine that you are hosting several "forks"
467 of a repository in a single shared object store, and letting clients
468 view them as separate repositories through `GIT_NAMESPACE` or separate
469 repos using the alternates mechanism. A naive repack may find that the
470 optimal delta for an object is against a base that is only found in
471 another fork. But when a client fetches, they will not have the base
472 object, and we'll have to find a new delta on the fly.
473
474 A similar situation may exist if you have many refs outside of
475 `refs/heads/` and `refs/tags/` that point to related objects (e.g.,
476 `refs/pull` or `refs/changes` used by some hosting providers). By
477 default, clients fetch only heads and tags, and deltas against objects
478 found only in those other groups cannot be sent as-is.
479
480 Delta islands solve this problem by allowing you to group your refs into
481 distinct "islands". Pack-objects computes which objects are reachable
482 from which islands, and refuses to make a delta from an object `A`
483 against a base which is not present in all of `A`'s islands. This
484 results in slightly larger packs (because we miss some delta
485 opportunities), but guarantees that a fetch of one island will not have
486 to recompute deltas on the fly due to crossing island boundaries.
487
488 When repacking with delta islands the delta window tends to get
489 clogged with candidates that are forbidden by the config. Repacking
490 with a big --window helps (and doesn't take as long as it otherwise
491 might because we can reject some object pairs based on islands before
492 doing any computation on the content).
493
494 Islands are configured via the `pack.island` option, which can be
495 specified multiple times. Each value is a left-anchored regular
496 expressions matching refnames. For example:
497
498 -------------------------------------------
499 [pack]
500 island = refs/heads/
501 island = refs/tags/
502 -------------------------------------------
503
504 puts heads and tags into an island (whose name is the empty string; see
505 below for more on naming). Any refs which do not match those regular
506 expressions (e.g., `refs/pull/123`) is not in any island. Any object
507 which is reachable only from `refs/pull/` (but not heads or tags) is
508 therefore not a candidate to be used as a base for `refs/heads/`.
509
510 Refs are grouped into islands based on their "names", and two regexes
511 that produce the same name are considered to be in the same
512 island. The names are computed from the regexes by concatenating any
513 capture groups from the regex, with a '-' dash in between. (And if
514 there are no capture groups, then the name is the empty string, as in
515 the above example.) This allows you to create arbitrary numbers of
516 islands. Only up to 14 such capture groups are supported though.
517
518 For example, imagine you store the refs for each fork in
519 `refs/virtual/ID`, where `ID` is a numeric identifier. You might then
520 configure:
521
522 -------------------------------------------
523 [pack]
524 island = refs/virtual/([0-9]+)/heads/
525 island = refs/virtual/([0-9]+)/tags/
526 island = refs/virtual/([0-9]+)/(pull)/
527 -------------------------------------------
528
529 That puts the heads and tags for each fork in their own island (named
530 "1234" or similar), and the pull refs for each go into their own
531 "1234-pull".
532
533 Note that we pick a single island for each regex to go into, using "last
534 one wins" ordering (which allows repo-specific config to take precedence
535 over user-wide config, and so forth).
536
537
538 CONFIGURATION
539 -------------
540
541 Various configuration variables affect packing, see
542 linkgit:git-config[1] (search for "pack" and "delta").
543
544 Notably, delta compression is not used on objects larger than the
545 `core.bigFileThreshold` configuration variable and on files with the
546 attribute `delta` set to false.
547
548 SEE ALSO
549 --------
550 linkgit:git-rev-list[1]
551 linkgit:git-repack[1]
552 linkgit:git-prune-packed[1]
553
554 GIT
555 ---
556 Part of the linkgit:git[1] suite