@matheusbsbs / matheus-bagliano / commits / b31222cfb7
Fork 0

Update packfile transfer protocol documentation

The current technical documentation for the packfile protocol is both sparse and incorrect. This documents the fetch-pack/upload-pack and send-pack/ receive-pack protocols much more fully. Add documentation from Shawn's upcoming http-protocol docs that is shared by the packfile protocol. protocol-common.txt describes ABNF notation amendments, refname rules and the packet line format. Add documentation on the various capabilities supported by the upload-pack and receive-pack protocols. protocol-capabilities.txt describes multi-ack, thin-pack, side-band[-64k], shallow, no-progress, include-tag, ofs-delta, delete-refs and report-status. Signed-off-by: Scott Chacon <schacon@gmail.com> Signed-off-by: Nanako Shiraishi <nanako3@lavabit.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Scott Chacon committed Nov 3, 2009 at 21:58 UTC b31222cfb7f2fcdc5b3d76ce5fcebf6ff1929826
3 files changed +777 -41
Documentation/technical/pack-protocol.txt
+494 -41
@@ -1,41 +1,494 @@
1 -Pack transfer protocols
2 -=======================
3 -
4 -There are two Pack push-pull protocols.
5 -
6 -upload-pack (S) | fetch/clone-pack (C) protocol:
7 -
8 - # Tell the puller what commits we have and what their names are
9 - S: SHA1 name
10 - S: ...
11 - S: SHA1 name
12 - S: # flush -- it's your turn
13 - # Tell the pusher what commits we want, and what we have
14 - C: want name
15 - C: ..
16 - C: want name
17 - C: have SHA1
18 - C: have SHA1
19 - C: ...
20 - C: # flush -- occasionally ask "had enough?"
21 - S: NAK
22 - C: have SHA1
23 - C: ...
24 - C: have SHA1
25 - S: ACK
26 - C: done
27 - S: XXXXXXX -- packfile contents.
28 -
29 -send-pack | receive-pack protocol.
30 -
31 - # Tell the pusher what commits we have and what their names are
32 - C: SHA1 name
33 - C: ...
34 - C: SHA1 name
35 - C: # flush -- it's your turn
36 - # Tell the puller what the pusher has
37 - S: old-SHA1 new-SHA1 name
38 - S: old-SHA1 new-SHA1 name
39 - S: ...
40 - S: # flush -- done with the list
41 - S: XXXXXXX --- packfile contents.
1 +Packfile transfer protocols
2 +===========================
3 +
4 +Git supports transferring data in packfiles over the ssh://, git:// and
5 +file:// transports. There exist two sets of protocols, one for pushing
6 +data from a client to a server and another for fetching data from a
7 +server to a client. All three transports (ssh, git, file) use the same
8 +protocol to transfer data.
9 +
10 +The processes invoked in the canonical Git implementation are 'upload-pack'
11 +on the server side and 'fetch-pack' on the client side for fetching data;
12 +then 'receive-pack' on the server and 'send-pack' on the client for pushing
13 +data. The protocol functions to have a server tell a client what is
14 +currently on the server, then for the two to negotiate the smallest amount
15 +of data to send in order to fully update one or the other.
16 +
17 +Transports
18 +----------
19 +There are three transports over which the packfile protocol is
20 +initiated. The Git transport is a simple, unauthenticated server that
21 +takes the command (almost always 'upload-pack', though Git
22 +servers can be configured to be globally writable, in which 'receive-
23 +pack' initiation is also allowed) with which the client wishes to
24 +communicate and executes it and connects it to the requesting
25 +process.
26 +
27 +In the SSH transport, the client just runs the 'upload-pack'
28 +or 'receive-pack' process on the server over the SSH protocol and then
29 +communicates with that invoked process over the SSH connection.
30 +
31 +The file:// transport runs the 'upload-pack' or 'receive-pack'
32 +process locally and communicates with it over a pipe.
33 +
34 +Git Transport
35 +-------------
36 +
37 +The Git transport starts off by sending the command and repository
38 +on the wire using the pkt-line format, followed by a NUL byte and a
39 +hostname paramater, terminated by a NUL byte.
40 +
41 + 0032git-upload-pack /project.git\0host=myserver.com\0
42 +
43 +--
44 + git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]
45 + request-command = "git-upload-pack" / "git-receive-pack" /
46 + "git-upload-archive" ; case sensitive
47 + pathname = *( %x01-ff ) ; exclude NUL
48 + host-parameter = "host=" hostname [ ":" port ]
49 +--
50 +
51 +Only host-parameter is allowed in the git-proto-request. Clients
52 +MUST NOT attempt to send additional parameters. It is used for the
53 +git-daemon name based virtual hosting. See --interpolated-path
54 +option to git daemon, with the %H/%CH format characters.
55 +
56 +Basically what the Git client is doing to connect to an 'upload-pack'
57 +process on the server side over the Git protocol is this:
58 +
59 + $ echo -e -n \
60 + "0039git-upload-pack /schacon/gitbook.git\0host=example.com\0" |
61 + nc -v example.com 9418
62 +
63 +
64 +SSH Transport
65 +-------------
66 +
67 +Initiating the upload-pack or receive-pack processes over SSH is
68 +executing the binary on the server via SSH remote execution.
69 +It is basically equivalent to running this:
70 +
71 + $ ssh git.example.com "git-upload-pack '/project.git'"
72 +
73 +For a server to support Git pushing and pulling for a given user over
74 +SSH, that user needs to be able to execute one or both of those
75 +commands via the SSH shell that they are provided on login. On some
76 +systems, that shell access is limited to only being able to run those
77 +two commands, or even just one of them.
78 +
79 +In an ssh:// format URI, it's absolute in the URI, so the '/' after
80 +the host name (or port number) is sent as an argument, which is then
81 +read by the remote git-upload-pack exactly as is, so it's effectively
82 +an absolute path in the remote filesystem.
83 +
84 + git clone ssh://user@example.com/project.git
85 + |
86 + v
87 + ssh user@example.com "git-upload-pack '/project.git'"
88 +
89 +In a "user@host:path" format URI, its relative to the user's home
90 +directory, because the Git client will run:
91 +
92 + git clone user@example.com:project.git
93 + |
94 + v
95 + ssh user@example.com "git-upload-pack 'project.git'"
96 +
97 +The exception is if a '~' is used, in which case
98 +we execute it without the leading '/'.
99 +
100 + ssh://user@example.com/~alice/project.git,
101 + |
102 + v
103 + ssh user@example.com "git-upload-pack '~alice/project.git'"
104 +
105 +A few things to remember here:
106 +
107 +- The "command name" is spelled with dash (e.g. git-upload-pack), but
108 + this can be overridden by the client;
109 +
110 +- The repository path is always quoted with single quotes.
111 +
112 +Fetching Data From a Server
113 +===========================
114 +
115 +When one Git repository wants to get data that a second repository
116 +has, the first can 'fetch' from the second. This operation determines
117 +what data the server has that the client does not then streams that
118 +data down to the client in packfile format.
119 +
120 +
121 +Reference Discovery
122 +-------------------
123 +
124 +When the client initially connects the server will immediately respond
125 +with a listing of each reference it has (all branches and tags) along
126 +with the object name that each reference currently points to.
127 +
128 + $ echo -e -n "0039git-upload-pack /schacon/gitbook.git\0host=example.com\0" |
129 + nc -v example.com 9418
130 + 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag
131 + 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration
132 + 003f7217a7c7e582c46cec22a130adf4b9d7d950fba0 refs/heads/master
133 + 003cb88d2441cac0977faf98efc80305012112238d9d refs/tags/v0.9
134 + 003c525128480b96c89e6418b1e40909bf6c5b2d580f refs/tags/v1.0
135 + 003fe92df48743b7bc7d26bcaabfddde0a1e20cae47c refs/tags/v1.0^{}
136 + 0000
137 +
138 +Server SHOULD terminate each non-flush line using LF ("\n") terminator;
139 +client MUST NOT complain if there is no terminator.
140 +
141 +The returned response is a pkt-line stream describing each ref and
142 +its current value. The stream MUST be sorted by name according to
143 +the C locale ordering.
144 +
145 +If HEAD is a valid ref, HEAD MUST appear as the first advertised
146 +ref. If HEAD is not a valid ref, HEAD MUST NOT appear in the
147 +advertisement list at all, but other refs may still appear.
148 +
149 +The stream MUST include capability declarations behind a NUL on the
150 +first ref. The peeled value of a ref (that is "ref^{}") MUST be
151 +immediately after the ref itself, if presented. A conforming server
152 +MUST peel the ref if its an annotated tag.
153 +
154 +----
155 + advertised-refs = (no-refs / list-of-refs)
156 + flush-pkt
157 +
158 + no-refs = PKT-LINE(zero-id SP "capabilities^{}"
159 + NUL capability-list LF)
160 +
161 + list-of-refs = first-ref *other-ref
162 + first-ref = PKT-LINE(obj-id SP refname
163 + NUL capability-list LF)
164 +
165 + other-ref = PKT-LINE(other-tip / other-peeled)
166 + other-tip = obj-id SP refname LF
167 + other-peeled = obj-id SP refname "^{}" LF
168 +
169 + capability-list = capability *(SP capability)
170 + capability = 1*(LC_ALPHA / DIGIT / "-" / "_")
171 + LC_ALPHA = %x61-7A
172 +----
173 +
174 +Server and client MUST use lowercase for obj-id, both MUST treat obj-id
175 +as case-insensitive.
176 +
177 +See protocol-capabilities.txt for a list of allowed server capabilities
178 +and descriptions.
179 +
180 +Packfile Negotiation
181 +--------------------
182 +After reference and capabilities discovery, the client can decide
183 +to terminate the connection by sending a flush-pkt, telling the
184 +server it can now gracefully terminate (as happens with the ls-remote
185 +command) or it can enter the negotiation phase, where the client and
186 +server determine what the minimal packfile necessary for transport is.
187 +
188 +Once the client has the initial list of references that the server
189 +has, as well as the list of capabilities, it will begin telling the
190 +server what objects it wants and what objects it has, so the server
191 +can make a packfile that only contains the objects that the client needs.
192 +The client will also send a list of the capabilities it wants to be in
193 +effect, out of what the server said it could do with the first 'want' line.
194 +
195 +----
196 + upload-request = want-list
197 + have-list
198 + compute-end
199 +
200 + want-list = first-want
201 + *additional-want
202 + flush-pkt
203 +
204 + first-want = PKT-LINE("want" SP obj-id SP capability-list LF)
205 + additional-want = PKT-LINE("want" SP obj-id LF)
206 +
207 + have-list = *have-line
208 + have-line = PKT-LINE("have" SP obj-id LF)
209 + compute-end = flush-pkt / PKT-LINE("done")
210 +----
211 +
212 +Clients MUST send all the obj-ids it wants from the reference
213 +discovery phase as 'want' lines. Clients MUST send at least one
214 +'want' command in the request body. Clients MUST NOT mention an
215 +obj-id in a 'want' command which did not appear in the response
216 +obtained through ref discovery.
217 +
218 +If client is requesting a shallow clone, it will now send a 'deepen'
219 +line with the depth it is requesting.
220 +
221 +Once all the "want"s (and optional 'deepen') are transferred,
222 +clients MUST send a flush-pkt. If the client has all the references
223 +on the server, client flushes and disconnects.
224 +
225 +TODO: shallow/unshallow response and document the deepen command in the ABNF.
226 +
227 +Now the client will send a list of the obj-ids it has using 'have'
228 +lines. In multi_ack mode, the canonical implementation will send up
229 +to 32 of these at a time, then will send a flush-pkt. The canonical
230 +implementation will skip ahead and send the next 32 immediately,
231 +so that there is always a block of 32 "in-flight on the wire" at a
232 +time.
233 +
234 +If the server reads 'have' lines, it then will respond by ACKing any
235 +of the obj-ids the client said it had that the server also has. The
236 +server will ACK obj-ids differently depending on which ack mode is
237 +chosen by the client.
238 +
239 +In multi_ack mode:
240 +
241 + * the server will respond with 'ACK obj-id continue' for any common
242 + commits.
243 +
244 + * once the server has found an acceptable common base commit and is
245 + ready to make a packfile, it will blindly ACK all 'have' obj-ids
246 + back to the client.
247 +
248 + * the server will then send a 'NACK' and then wait for another response
249 + from the client - either a 'done' or another list of 'have' lines.
250 +
251 +In multi_ack_detailed mode:
252 +
253 + * the server will differentiate the ACKs where it is signaling
254 + that it is ready to send data with 'ACK obj-id ready' lines, and
255 + signals the identified common commits with 'ACK obj-id common' lines.
256 +
257 +Without either multi_ack or multi_ack_detailed:
258 +
259 + * upload-pack sends "ACK obj-id" on the first common object it finds.
260 + After that it says nothing until the client gives it a "done".
261 +
262 + * upload-pack sends "NAK" on a flush-pkt if no common object
263 + has been found yet. If one has been found, and thus an ACK
264 + was already sent, its silent on the flush-pkt.
265 +
266 +After the client has gotten enough ACK responses that it can determine
267 +that the server has enough information to send an efficient packfile
268 +(in the canonical implementation, this is determined when it has received
269 +enough ACKs that it can color everything left in the --date-order queue
270 +as common with the server, or the --date-order queue is empty), or the
271 +client determines that it wants to give up (in the canonical implementation,
272 +this is determined when the client sends 256 'have' lines without getting
273 +any of them ACKed by the server - meaning there is nothing in common and
274 +the server should just send all it's objects), then the client will send
275 +a 'done' command. The 'done' command signals to the server that the client
276 +is ready to receive it's packfile data.
277 +
278 +However, the 256 limit *only* turns on in the canonical client
279 +implementation if we have received at least one "ACK %s continue"
280 +during a prior round. This helps to ensure that at least one common
281 +ancestor is found before we give up entirely.
282 +
283 +Once the 'done' line is read from the client, the server will either
284 +send a final 'ACK obj-id' or it will send a 'NAK'. The server only sends
285 +ACK after 'done' if there is at least one common base and multi_ack or
286 +multi_ack_detailed is enabled. The server always sends NAK after 'done'
287 +if there is no common base found.
288 +
289 +Then the server will start sending it's packfile data.
290 +
291 +----
292 + server-response = *ack_multi ack / nak
293 + ack_multi = PKT-LINE("ACK" SP obj-id ack_status LF)
294 + ack_status = "continue" / "common" / "ready"
295 + ack = PKT-LINE("ACK SP obj-id LF)
296 + nak = PKT-LINE("NAK" LF)
297 +----
298 +
299 +A simple clone may look like this (with no 'have' lines):
300 +
301 +----
302 + C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d\0multi_ack \
303 + side-band-64k ofs-delta\n
304 + C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n
305 + C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n
306 + C: 0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n
307 + C: 0032want 74730d410fcb6603ace96f1dc55ea6196122532d\n
308 + C: 0000
309 + C: 0009done\n
310 +
311 + S: 0008NAK\n
312 + S: [PACKFILE]
313 +----
314 +
315 +An incremental update (fetch) response might look like this:
316 +
317 +----
318 + C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d\0multi_ack \
319 + side-band-64k ofs-delta\n
320 + C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n
321 + C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n
322 + C: 0000
323 + C: 0032have 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n
324 + C: [30 more have lines]
325 + C: 0032have 74730d410fcb6603ace96f1dc55ea6196122532d\n
326 + C: 0000
327 +
328 + S: 003aACK 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01 continue\n
329 + S: 003aACK 74730d410fcb6603ace96f1dc55ea6196122532d continue\n
330 + S: 0008NAK\n
331 +
332 + C: 0009done\n
333 +
334 + S: 003aACK 74730d410fcb6603ace96f1dc55ea6196122532d\n
335 + S: [PACKFILE]
336 +----
337 +
338 +
339 +Packfile Data
340 +-------------
341 +
342 +Now that the client and server have finished negotiation about what
343 +the minimal amount of data that needs to be sent to the client is, the server
344 +will construct and send the required data in packfile format.
345 +
346 +See pack-format.txt for what the packfile itself actually looks like.
347 +
348 +If 'side-band' or 'side-band-64k' capabilities have been specified by
349 +the client, the server will send the packfile data multiplexed.
350 +
351 +Each packet starting with the packet-line length of the amount of data
352 +that follows, followed by a single byte specifying the sideband the
353 +following data is coming in on.
354 +
355 +In 'side-band' mode, it will send up to 999 data bytes plus 1 control
356 +code, for a total of up to 1000 bytes in a pkt-line. In 'side-band-64k'
357 +mode it will send up to 65519 data bytes plus 1 control code, for a
358 +total of up to 65520 bytes in a pkt-line.
359 +
360 +The sideband byte will be a '1', '2' or a '3'. Sideband '1' will contain
361 +packfile data, sideband '2' will be used for progress information that the
362 +client will generally print to stderr and sideband '3' is used for error
363 +information.
364 +
365 +If no 'side-band' capability was specified, the server will stream the
366 +entire packfile without multiplexing.
367 +
368 +
369 +Pushing Data To a Server
370 +========================
371 +
372 +Pushing data to a server will invoke the 'receive-pack' process on the
373 +server, which will allow the client to tell it which references it should
374 +update and then send all the data the server will need for those new
375 +references to be complete. Once all the data is received and validated,
376 +the server will then update its references to what the client specified.
377 +
378 +Authentication
379 +--------------
380 +
381 +The protocol itself contains no authentication mechanisms. That is to be
382 +handled by the transport, such as SSH, before the 'receive-pack' process is
383 +invoked. If 'receive-pack' is configured over the Git transport, those
384 +repositories will be writable by anyone who can access that port (9418) as
385 +that transport is unauthenticated.
386 +
387 +Reference Discovery
388 +-------------------
389 +
390 +The reference discovery phase is done nearly the same way as it is in the
391 +fetching protocol. Each reference obj-id and name on the server is sent
392 +in packet-line format to the client, followed by a flush-pkt. The only
393 +real difference is that the capability listing is different - the only
394 +possible values are 'report-status', 'delete-refs' and 'ofs-delta'.
395 +
396 +Reference Update Request and Packfile Transfer
397 +----------------------------------------------
398 +
399 +Once the client knows what references the server is at, it can send a
400 +list of reference update requests. For each reference on the server
401 +that it wants to update, it sends a line listing the obj-id currently on
402 +the server, the obj-id the client would like to update it to and the name
403 +of the reference.
404 +
405 +This list is followed by a flush-pkt and then the packfile that should
406 +contain all the objects that the server will need to complete the new
407 +references.
408 +
409 +----
410 + update-request = command-list [pack-file]
411 +
412 + command-list = PKT-LINE(command NUL capability-list LF)
413 + *PKT-LINE(command LF)
414 + flush-pkt
415 +
416 + command = create / delete / update
417 + create = zero-id SP new-id SP name
418 + delete = old-id SP zero-id SP name
419 + update = old-id SP new-id SP name
420 +
421 + old-id = obj-id
422 + new-id = obj-id
423 +
424 + pack-file = "PACK" 28*(OCTET)
425 +----
426 +
427 +If the receiving end does not support delete-refs, the sending end MUST
428 +NOT ask for delete command.
429 +
430 +The pack-file MUST NOT be sent if the only command used is 'delete'.
431 +
432 +A pack-file MUST be sent if either create or update command is used,
433 +even if the server already has all the necessary objects. In this
434 +case the client MUST send an empty pack-file. The only time this
435 +is likely to happen is if the client is creating
436 +a new branch or a tag that points to an existing obj-id.
437 +
438 +The server will receive the packfile, unpack it, then validate each
439 +reference that is being updated that it hasn't changed while the request
440 +was being processed (the obj-id is still the same as the old-id), and
441 +it will run any update hooks to make sure that the update is acceptable.
442 +If all of that is fine, the server will then update the references.
443 +
444 +Report Status
445 +-------------
446 +
447 +After receiving the pack data from the sender, the receiver sends a
448 +report if 'report-status' capability is in effect.
449 +It is a short listing of what happened in that update. It will first
450 +list the status of the packfile unpacking as either 'unpack ok' or
451 +'unpack [error]'. Then it will list the status for each of the references
452 +that it tried to update. Each line is either 'ok [refname]' if the
453 +update was successful, or 'ng [refname] [error]' if the update was not.
454 +
455 +----
456 + report-status = unpack-status
457 + 1*(command-status)
458 + flush-pkt
459 +
460 + unpack-status = PKT-LINE("unpack" SP unpack-result LF)
461 + unpack-result = "ok" / error-msg
462 +
463 + command-status = command-ok / command-fail
464 + command-ok = PKT-LINE("ok" SP refname LF)
465 + command-fail = PKT-LINE("ng" SP refname SP error-msg LF)
466 +
467 + error-msg = 1*(OCTECT) ; where not "ok"
468 +----
469 +
470 +Updates can be unsuccessful for a number of reasons. The reference can have
471 +changed since the reference discovery phase was originally sent, meaning
472 +someone pushed in the meantime. The reference being pushed could be a
473 +non-fast-forward reference and the update hooks or configuration could be
474 +set to not allow that, etc. Also, some references can be updated while others
475 +can be rejected.
476 +
477 +An example client/server communication might look like this:
478 +
479 +----
480 + S: 007c74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/local\0report-status delete-refs ofs-delta\n
481 + S: 003e7d1665144a3a975c05f1f43902ddaf084e784dbe refs/heads/debug\n
482 + S: 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/master\n
483 + S: 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/team\n
484 + S: 0000
485 +
486 + C: 003e7d1665144a3a975c05f1f43902ddaf084e784dbe 74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/debug\n
487 + C: 003e74730d410fcb6603ace96f1dc55ea6196122532d 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/master\n
488 + C: 0000
489 + C: [PACKDATA]
490 +
491 + S: 000aunpack ok\n
492 + S: 0014ok refs/heads/debug\n
493 + S: 0026ng refs/heads/master non-fast-forward\n
494 +----
Documentation/technical/protocol-capabilities.txt new
+187
@@ -0,0 +1,187 @@
1 +Git Protocol Capabilities
2 +=========================
3 +
4 +Servers SHOULD support all capabilities defined in this document.
5 +
6 +On the very first line of the initial server response of either
7 +receive-pack and upload-pack the first reference is followed by
8 +a NUL byte and then a list of space delimited server capabilities.
9 +These allow the server to declare what it can and cannot support
10 +to the client.
11 +
12 +Client will then send a space separated list of capabilities it wants
13 +to be in effect. The client MUST NOT ask for capabilities the server
14 +did not say it supports.
15 +
16 +Server MUST diagnose and abort if capabilities it does not understand
17 +was sent. Server MUST NOT ignore capabilities that client requested
18 +and server advertised. As a consequence of these rules, server MUST
19 +NOT advertise capabilities it does not understand.
20 +
21 +The 'report-status' and 'delete-refs' capabilities are sent and
22 +recognized by the receive-pack (push to server) process.
23 +
24 +The 'ofs-delta' capability is sent and recognized by both upload-pack
25 +and receive-pack protocols.
26 +
27 +All other capabilities are only recognized by the upload-pack (fetch
28 +from server) process.
29 +
30 +multi_ack
31 +---------
32 +
33 +The 'multi_ack' capability allows the server to return "ACK obj-id
34 +continue" as soon as it finds a commit that it can use as a common
35 +base, between the client's wants and the client's have set.
36 +
37 +By sending this early, the server can potentially head off the client
38 +from walking any further down that particular branch of the client's
39 +repository history. The client may still need to walk down other
40 +branches, sending have lines for those, until the server has a
41 +complete cut across the DAG, or the client has said "done".
42 +
43 +Without multi_ack, a client sends have lines in --date-order until
44 +the server has found a common base. That means the client will send
45 +have lines that are already known by the server to be common, because
46 +they overlap in time with another branch that the server hasn't found
47 +a common base on yet.
48 +
49 +For example suppose the client has commits in caps that the server
50 +doesn't and the server has commits in lower case that the client
51 +doesn't, as in the following diagram:
52 +
53 + +---- u ---------------------- x
54 + / +----- y
55 + / /
56 + a -- b -- c -- d -- E -- F
57 + \
58 + +--- Q -- R -- S
59 +
60 +If the client wants x,y and starts out by saying have F,S, the server
61 +doesn't know what F,S is. Eventually the client says "have d" and
62 +the server sends "ACK d continue" to let the client know to stop
63 +walking down that line (so don't send c-b-a), but its not done yet,
64 +it needs a base for x. The client keeps going with S-R-Q, until a
65 +gets reached, at which point the server has a clear base and it all
66 +ends.
67 +
68 +Without multi_ack the client would have sent that c-b-a chain anyway,
69 +interleaved with S-R-Q.
70 +
71 +thin-pack
72 +---------
73 +
74 +This capability means that the server can send a 'thin' pack, a pack
75 +which does not contain base objects; if those base objects are available
76 +on client side. Client requests 'thin-pack' capability when it
77 +understands how to "thicken" it by adding required delta bases making
78 +it self-contained.
79 +
80 +Client MUST NOT request 'thin-pack' capability if it cannot turn a thin
81 +pack into a self-contained pack.
82 +
83 +
84 +side-band, side-band-64k
85 +------------------------
86 +
87 +This capability means that server can send, and client understand multiplexed
88 +progress reports and error info interleaved with the packfile itself.
89 +
90 +These two options are mutually exclusive. A modern client always
91 +favors 'side-band-64k'.
92 +
93 +Either mode indicates that the packfile data will be streamed broken
94 +up into packets of up to either 1000 bytes in the case of 'side_band',
95 +or 65520 bytes in the case of 'side_band_64k'. Each packet is made up
96 +of a leading 4-byte pkt-line length of how much data is in the packet,
97 +followed by a 1-byte stream code, followed by the actual data.
98 +
99 +The stream code can be one of:
100 +
101 + 1 - pack data
102 + 2 - progress messages
103 + 3 - fatal error message just before stream aborts
104 +
105 +The "side-band-64k" capability came about as a way for newer clients
106 +that can handle much larger packets to request packets that are
107 +actually crammed nearly full, while maintaining backward compatibility
108 +for the older clients.
109 +
110 +Further, with side-band and its up to 1000-byte messages, it's actually
111 +999 bytes of payload and 1 byte for the stream code. With side-band-64k,
112 +same deal, you have up to 65519 bytes of data and 1 byte for the stream
113 +code.
114 +
115 +The client MUST send only maximum of one of "side-band" and "side-
116 +band-64k". Server MUST diagnose it as an error if client requests
117 +both.
118 +
119 +ofs-delta
120 +---------
121 +
122 +Server can send, and client understand PACKv2 with delta refering to
123 +its base by position in pack rather than by an obj-id. That is, they can
124 +send/read OBJ_OFS_DELTA (aka type 6) in a packfile.
125 +
126 +shallow
127 +-------
128 +
129 +This capability adds "deepen", "shallow" and "unshallow" commands to
130 +the fetch-pack/upload-pack protocol so clients can request shallow
131 +clones.
132 +
133 +no-progress
134 +-----------
135 +
136 +The client was started with "git clone -q" or something, and doesn't
137 +want that side band 2. Basically the client just says "I do not
138 +wish to receive stream 2 on sideband, so do not send it to me, and if
139 +you did, I will drop it on the floor anyway". However, the sideband
140 +channel 3 is still used for error responses.
141 +
142 +include-tag
143 +-----------
144 +
145 +The 'include-tag' capability is about sending annotated tags if we are
146 +sending objects they point to. If we pack an object to the client, and
147 +a tag object points exactly at that object, we pack the tag object too.
148 +In general this allows a client to get all new annotated tags when it
149 +fetches a branch, in a single network connection.
150 +
151 +Clients MAY always send include-tag, hardcoding it into a request when
152 +the server advertises this capability. The decision for a client to
153 +request include-tag only has to do with the client's desires for tag
154 +data, whether or not a server had advertised objects in the
155 +refs/tags/* namespace.
156 +
157 +Servers MUST pack the tags if their referrant is packed and the client
158 +has requested include-tags.
159 +
160 +Clients MUST be prepared for the case where a server has ignored
161 +include-tag and has not actually sent tags in the pack. In such
162 +cases the client SHOULD issue a subsequent fetch to acquire the tags
163 +that include-tag would have otherwise given the client.
164 +
165 +The server SHOULD send include-tag, if it supports it, regardless
166 +of whether or not there are tags available.
167 +
168 +report-status
169 +-------------
170 +
171 +The upload-pack process can receive a 'report-status' capability,
172 +which tells it that the client wants a report of what happened after
173 +a packfile upload and reference update. If the pushing client requests
174 +this capability, after unpacking and updating references the server
175 +will respond with whether the packfile unpacked successfully and if
176 +each reference was updated successfully. If any of those were not
177 +successful, it will send back an error message. See pack-protocol.txt
178 +for example messages.
179 +
180 +delete-refs
181 +-----------
182 +
183 +If the server sends back the 'delete-refs' capability, it means that
184 +it is capable of accepting an zero-id value as the target
185 +value of a reference update. It is not sent back by the client, it
186 +simply informs the client that it can be sent zero-id values
187 +to delete references.
Documentation/technical/protocol-common.txt new
+96
@@ -0,0 +1,96 @@
1 +Documentation Common to Pack and Http Protocols
2 +===============================================
3 +
4 +ABNF Notation
5 +-------------
6 +
7 +ABNF notation as described by RFC 5234 is used within the protocol documents,
8 +except the following replacement core rules are used:
9 +----
10 + HEXDIG = DIGIT / "a" / "b" / "c" / "d" / "e" / "f"
11 +----
12 +
13 +We also define the following common rules:
14 +----
15 + NUL = %x00
16 + zero-id = 40*"0"
17 + obj-id = 40*(HEXDIGIT)
18 +
19 + refname = "HEAD"
20 + refname /= "refs/" <see discussion below>
21 +----
22 +
23 +A refname is a hierarchical octet string beginning with "refs/" and
24 +not violating the 'git-check-ref-format' command's validation rules.
25 +More specifically, they:
26 +
27 +. They can include slash `/` for hierarchical (directory)
28 + grouping, but no slash-separated component can begin with a
29 + dot `.`.
30 +
31 +. They must contain at least one `/`. This enforces the presence of a
32 + category like `heads/`, `tags/` etc. but the actual names are not
33 + restricted.
34 +
35 +. They cannot have two consecutive dots `..` anywhere.
36 +
37 +. They cannot have ASCII control characters (i.e. bytes whose
38 + values are lower than \040, or \177 `DEL`), space, tilde `~`,
39 + caret `{caret}`, colon `:`, question-mark `?`, asterisk `*`,
40 + or open bracket `[` anywhere.
41 +
42 +. They cannot end with a slash `/` nor a dot `.`.
43 +
44 +. They cannot end with the sequence `.lock`.
45 +
46 +. They cannot contain a sequence `@{`.
47 +
48 +. They cannot contain a `\\`.
49 +
50 +
51 +pkt-line Format
52 +---------------
53 +
54 +Much (but not all) of the payload is described around pkt-lines.
55 +
56 +A pkt-line is a variable length binary string. The first four bytes
57 +of the line, the pkt-len, indicates the total length of the line,
58 +in hexadecimal. The pkt-len includes the 4 bytes used to contain
59 +the length's hexadecimal representation.
60 +
61 +A pkt-line MAY contain binary data, so implementors MUST ensure
62 +pkt-line parsing/formatting routines are 8-bit clean.
63 +
64 +A non-binary line SHOULD BE terminated by an LF, which if present
65 +MUST be included in the total length.
66 +
67 +The maximum length of a pkt-line's data component is 65520 bytes.
68 +Implementations MUST NOT send pkt-line whose length exceeds 65524
69 +(65520 bytes of payload + 4 bytes of length data).
70 +
71 +Implementations SHOULD NOT send an empty pkt-line ("0004").
72 +
73 +A pkt-line with a length field of 0 ("0000"), called a flush-pkt,
74 +is a special case and MUST be handled differently than an empty
75 +pkt-line ("0004").
76 +
77 +----
78 + pkt-line = data-pkt / flush-pkt
79 +
80 + data-pkt = pkt-len pkt-payload
81 + pkt-len = 4*(HEXDIG)
82 + pkt-payload = (pkt-len - 4)*(OCTET)
83 +
84 + flush-pkt = "0000"
85 +----
86 +
87 +Examples (as C-style strings):
88 +
89 +----
90 + pkt-line actual value
91 + ---------------------------------
92 + "0006a\n" "a\n"
93 + "0005a" "a"
94 + "000bfoobar\n" "foobar\n"
95 + "0004" ""
96 +----