Documentation: add Packfile URIs design doc
Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jonathan Tan committed
Jun 10, 2020 at 13:57 UTC
cd8402e0fd8cfc0ec9fb10e22ffb6aabd992eae1
2 files changed
+109
-1
Documentation/technical/packfile-uri.txt
new
+78
@@ -0,0 +1,78 @@
1
+Packfile URIs
2
+=============
3
+
4
+This feature allows servers to serve part of their packfile response as URIs.
5
+This allows server designs that improve scalability in bandwidth and CPU usage
6
+(for example, by serving some data through a CDN), and (in the future) provides
7
+some measure of resumability to clients.
8
+
9
+This feature is available only in protocol version 2.
10
+
11
+Protocol
12
+--------
13
+
14
+The server advertises the `packfile-uris` capability.
15
+
16
+If the client then communicates which protocols (HTTPS, etc.) it supports with
17
+a `packfile-uris` argument, the server MAY send a `packfile-uris` section
18
+directly before the `packfile` section (right after `wanted-refs` if it is
19
+sent) containing URIs of any of the given protocols. The URIs point to
20
+packfiles that use only features that the client has declared that it supports
21
+(e.g. ofs-delta and thin-pack). See protocol-v2.txt for the documentation of
22
+this section.
23
+
24
+Clients should then download and index all the given URIs (in addition to
25
+downloading and indexing the packfile given in the `packfile` section of the
26
+response) before performing the connectivity check.
27
+
28
+Server design
29
+-------------
30
+
31
+The server can be trivially made compatible with the proposed protocol by
32
+having it advertise `packfile-uris`, tolerating the client sending
33
+`packfile-uris`, and never sending any `packfile-uris` section. But we should
34
+include some sort of non-trivial implementation in the Minimum Viable Product,
35
+at least so that we can test the client.
36
+
37
+This is the implementation: a feature, marked experimental, that allows the
38
+server to be configured by one or more `uploadpack.blobPackfileUri=<sha1>
39
+<uri>` entries. Whenever the list of objects to be sent is assembled, all such
40
+blobs are excluded, replaced with URIs. The client will download those URIs,
41
+expecting them to each point to packfiles containing single blobs.
42
+
43
+Client design
44
+-------------
45
+
46
+The client has a config variable `fetch.uriprotocols` that determines which
47
+protocols the end user is willing to use. By default, this is empty.
48
+
49
+When the client downloads the given URIs, it should store them with "keep"
50
+files, just like it does with the packfile in the `packfile` section. These
51
+additional "keep" files can only be removed after the refs have been updated -
52
+just like the "keep" file for the packfile in the `packfile` section.
53
+
54
+The division of work (initial fetch + additional URIs) introduces convenient
55
+points for resumption of an interrupted clone - such resumption can be done
56
+after the Minimum Viable Product (see "Future work").
57
+
58
+Future work
59
+-----------
60
+
61
+The protocol design allows some evolution of the server and client without any
62
+need for protocol changes, so only a small-scoped design is included here to
63
+form the MVP. For example, the following can be done:
64
+
65
+ * On the server, more sophisticated means of excluding objects (e.g. by
66
+ specifying a commit to represent that commit and all objects that it
67
+ references).
68
+ * On the client, resumption of clone. If a clone is interrupted, information
69
+ could be recorded in the repository's config and a "clone-resume" command
70
+ can resume the clone in progress. (Resumption of subsequent fetches is more
71
+ difficult because that must deal with the user wanting to use the repository
72
+ even after the fetch was interrupted.)
73
+
74
+There are some possible features that will require a change in protocol:
75
+
76
+ * Additional HTTP headers (e.g. authentication)
77
+ * Byte range support
78
+ * Different file formats referenced by URIs (e.g. raw object)
Documentation/technical/protocol-v2.txt
+31
-1
@@ -323,13 +323,26 @@ included in the client's request:
323
indicating its sideband (1, 2, or 3), and the server may send "0005\2"
324
(a PKT-LINE of sideband 2 with no payload) as a keepalive packet.
325
326
+If the 'packfile-uris' feature is advertised, the following argument
327
+can be included in the client's request as well as the potential
328
+addition of the 'packfile-uris' section in the server's response as
329
+explained below.
330
+
331
+ packfile-uris <comma-separated list of protocols>
332
+ Indicates to the server that the client is willing to receive
333
+ URIs of any of the given protocols in place of objects in the
334
+ sent packfile. Before performing the connectivity check, the
335
+ client should download from all given URIs. Currently, the
336
+ protocols supported are "http" and "https".
337
+
338
The response of `fetch` is broken into a number of sections separated by
339
delimiter packets (0001), with each section beginning with its section
340
header. Most sections are sent only when the packfile is sent.
341
342
output = acknowledgements flush-pkt |
343
[acknowledgments delim-pkt] [shallow-info delim-pkt]
332
- [wanted-refs delim-pkt] packfile flush-pkt
344
+ [wanted-refs delim-pkt] [packfile-uris delim-pkt]
345
+ packfile flush-pkt
346
347
acknowledgments = PKT-LINE("acknowledgments" LF)
348
(nak | *ack)
@@ -347,6 +360,9 @@ header. Most sections are sent only when the packfile is sent.
360
*PKT-LINE(wanted-ref LF)
361
wanted-ref = obj-id SP refname
362
363
+ packfile-uris = PKT-LINE("packfile-uris" LF) *packfile-uri
364
+ packfile-uri = PKT-LINE(40*(HEXDIGIT) SP *%x20-ff LF)
365
+
366
packfile = PKT-LINE("packfile" LF)
367
*PKT-LINE(%x01-03 *%x00-ff)
368
@@ -418,6 +434,20 @@ header. Most sections are sent only when the packfile is sent.
434
* The server MUST NOT send any refs which were not requested
435
using 'want-ref' lines.
436
437
+ packfile-uris section
438
+ * This section is only included if the client sent
439
+ 'packfile-uris' and the server has at least one such URI to
440
+ send.
441
+
442
+ * Always begins with the section header "packfile-uris".
443
+
444
+ * For each URI the server sends, it sends a hash of the pack's
445
+ contents (as output by git index-pack) followed by the URI.
446
+
447
+ * The hashes are 40 hex characters long. When Git upgrades to a new
448
+ hash algorithm, this might need to be updated. (It should match
449
+ whatever index-pack outputs after "pack\t" or "keep\t".
450
+
451
packfile section
452
* This section is only included if the client has sent 'want'
453
lines in its request and either requested that no more