@matheusbsbs / matheus-bagliano / commits / 8b8d9a2298

protocol v2: add server-side "bundle-uri" skeleton

Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". This change only performs the basic boilerplate of advertising a new protocol v2 capability. The new 'bundle-uri' capability allows a client to request a list of bundles. Right now, the server only returns a flush packet, which corresponds to an empty advertisement. The bundle.* config namespace describes which key-value pairs will be communicated across this interface in future updates. The critical bit right now is that the new boolean uploadPack.adverstiseBundleURIs config value signals whether or not this capability should be advertised at all. An earlier version of this patch [1] used a different transfer format than the "key=value" pairs in the current implementation. The change was made to unify the protocol v2 command with the bundle lists provided by independent bundle servers. Further, the standard allows for the server to advertise a URI that contains a bundle list. This allows users automatically discovering bundle providers that are loosely associated with the origin server, but without the origin server knowing exactly which bundles are currently available. [1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/ The very-deep headings needed to be modified to stop at level 4 due to documentation build issues. These were not recognized in earlier builds since the file was previously in the Documentation/technical/ directory and was built in a different way. With its current location, the heavily-nested details were causing build issues and they are now replaced with a bulletted list of details. Co-authored-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Ævar Arnfjörð Bjarmason committed Dec 22, 2022 at 15:14 UTC 8b8d9a229888adb737851c4d7eeaa9a50e37afe1

5 files changed +289 -1

Documentation/gitprotocol-v2.txt

+201

@@ -578,6 +578,207 @@ and associated requested information, each separated by a single space.

578

579 obj-info = obj-id SP obj-size

580

581 +bundle-uri

582 +~~~~~~~~~~

583 +

584 +If the 'bundle-uri' capability is advertised, the server supports the

585 +`bundle-uri' command.

586 +

587 +The capability is currently advertised with no value (i.e. not

588 +"bundle-uri=somevalue"), a value may be added in the future for

589 +supporting command-wide extensions. Clients MUST ignore any unknown

590 +capability values and proceed with the 'bundle-uri` dialog they

591 +support.

592 +

593 +The 'bundle-uri' command is intended to be issued before `fetch` to

594 +get URIs to bundle files (see linkgit:git-bundle[1]) to "seed" and

595 +inform the subsequent `fetch` command.

596 +

597 +The client CAN issue `bundle-uri` before or after any other valid

598 +command. To be useful to clients it's expected that it'll be issued

599 +after an `ls-refs` and before `fetch`, but CAN be issued at any time

600 +in the dialog.

601 +

602 +DISCUSSION of bundle-uri

603 +^^^^^^^^^^^^^^^^^^^^^^^^

604 +

605 +The intent of the feature is optimize for server resource consumption

606 +in the common case by changing the common case of fetching a very

607 +large PACK during linkgit:git-clone[1] into a smaller incremental

608 +fetch.

609 +

610 +It also allows servers to achieve better caching in combination with

611 +an `uploadpack.packObjectsHook` (see linkgit:git-config[1]).

612 +

613 +By having new clones or fetches be a more predictable and common

614 +negotiation against the tips of recently produces *.bundle file(s).

615 +Servers might even pre-generate the results of such negotiations for

616 +the `uploadpack.packObjectsHook` as new pushes come in.

617 +

618 +One way that servers could take advantage of these bundles is that the

619 +server would anticipate that fresh clones will download a known bundle,

620 +followed by catching up to the current state of the repository using ref

621 +tips found in that bundle (or bundles).

622 +

623 +PROTOCOL for bundle-uri

624 +^^^^^^^^^^^^^^^^^^^^^^^

625 +

626 +A `bundle-uri` request takes no arguments, and as noted above does not

627 +currently advertise a capability value. Both may be added in the

628 +future.

629 +

630 +When the client issues a `command=bundle-uri` request, the response is a

631 +list of key-value pairs provided as packet lines with value

632 +`<key>=<value>`. Each `<key>` should be interpreted as a config key from

633 +the `bundle.*` namespace to construct a list of bundles. These keys are

634 +grouped by a `bundle.<id>.` subsection, where each key corresponding to a

635 +given `<id>` contributes attributes to the bundle defined by that `<id>`.

636 +See linkgit:git-config[1] for the specific details of these keys and how

637 +the Git client will interpret their values.

638 +

639 +Clients MUST parse the line according to the above format, lines that do

640 +not conform to the format SHOULD be discarded. The user MAY be warned in

641 +such a case.

642 +

643 +bundle-uri CLIENT AND SERVER EXPECTATIONS

644 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

645 +

646 +URI CONTENTS::

647 +The content at the advertised URIs MUST be one of two types.

648 ++

649 +The advertised URI may contain a bundle file that `git bundle verify`

650 +would accept. I.e. they MUST contain one or more reference tips for

651 +use by the client, MUST indicate prerequisites (in any) with standard

652 +"-" prefixes, and MUST indicate their "object-format", if

653 +applicable.

654 ++

655 +The advertised URI may alternatively contain a plaintext file that `git

656 +config --list` would accept (with the `--file` option). The key-value

657 +pairs in this list are in the `bundle.*` namespace (see

658 +linkgit:git-config[1]).

659 +

660 +bundle-uri CLIENT ERROR RECOVERY::

661 +A client MUST above all gracefully degrade on errors, whether that

662 +error is because of bad missing/data in the bundle URI(s), because

663 +that client is too dumb to e.g. understand and fully parse out bundle

664 +headers and their prerequisite relationships, or something else.

665 ++

666 +Server operators should feel confident in turning on "bundle-uri" and

667 +not worry if e.g. their CDN goes down that clones or fetches will run

668 +into hard failures. Even if the server bundle bundle(s) are

669 +incomplete, or bad in some way the client should still end up with a

670 +functioning repository, just as if it had chosen not to use this

671 +protocol extension.

672 ++

673 +All subsequent discussion on client and server interaction MUST keep

674 +this in mind.

675 +

676 +bundle-uri SERVER TO CLIENT::

677 +The ordering of the returned bundle uris is not significant. Clients

678 +MUST parse their headers to discover their contained OIDS and

679 +prerequisites. A client MUST consider the content of the bundle(s)

680 +themselves and their header as the ultimate source of truth.

681 ++

682 +A server MAY even return bundle(s) that don't have any direct

683 +relationship to the repository being cloned (either through accident,

684 +or intentional "clever" configuration), and expect a client to sort

685 +out what data they'd like from the bundle(s), if any.

686 +

687 +bundle-uri CLIENT TO SERVER::

688 +The client SHOULD provide reference tips found in the bundle header(s)

689 +as 'have' lines in any subsequent `fetch` request. A client MAY also

690 +ignore the bundle(s) entirely if doing so is deemed worse for some

691 +reason, e.g. if the bundles can't be downloaded, it doesn't like the

692 +tips it finds etc.

693 +

694 +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION::

695 +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s)

696 +of the bundle(s) the client finds that the ref tips it wants can be

697 +retrieved entirely from advertised bundle(s), the client MAY disconnect

698 +from the Git server. The results of such a 'clone' or 'fetch' should be

699 +indistinguishable from the state attained without using bundle-uri.

700 +

701 +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY::

702 +A client MAY perform an early disconnect while still downloading the

703 +bundle(s) (having streamed and parsed their headers). In such a case

704 +the client MUST gracefully recover from any errors related to

705 +finishing the download and validation of the bundle(s).

706 ++

707 +I.e. a client might need to re-connect and issue a 'fetch' command,

708 +and possibly fall back to not making use of 'bundle-uri' at all.

709 ++

710 +This "MAY" behavior is specified as such (and not a "SHOULD") on the

711 +assumption that a server advertising bundle uris is more likely than

712 +not to be serving up a relatively large repository, and to be pointing

713 +to URIs that have a good chance of being in working order. A client

714 +MAY e.g. look at the payload size of the bundles as a heuristic to see

715 +if an early disconnect is worth it, should falling back on a full

716 +"fetch" dialog be necessary.

717 +

718 +WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION::

719 +A client SHOULD commence a negotiation of a PACK from the server via

720 +the "fetch" command using the OID tips found in advertised bundles,

721 +even if's still in the process of downloading those bundle(s).

722 ++

723 +This allows for aggressive early disconnects from any interactive

724 +server dialog. The client blindly trusts that the advertised OID tips

725 +are relevant, and issues them as 'have' lines, it then requests any

726 +tips it would like (usually from the "ls-refs" advertisement) via

727 +'want' lines. The server will then compute a (hopefully small) PACK

728 +with the expected difference between the tips from the bundle(s) and

729 +the data requested.

730 ++

731 +The only connection the client then needs to keep active is to the

732 +concurrently downloading static bundle(s), when those and the

733 +incremental PACK are retrieved they should be inflated and

734 +validated. Any errors at this point should be gracefully recovered

735 +from, see above.

736 +

737 +bundle-uri PROTOCOL FEATURES

738 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^

739 +

740 +The client constructs a bundle list from the `<key>=<value>` pairs

741 +provided by the server. These pairs are part of the `bundle.*` namespace

742 +as documented in linkgit:git-config[1]. In this section, we discuss some

743 +of these keys and describe the actions the client will do in response to

744 +this information.

745 +

746 +In particular, the `bundle.version` key specifies an integer value. The

747 +only accepted value at the moment is `1`, but if the client sees an

748 +unexpected value here then the client MUST ignore the bundle list.

749 +

750 +As long as `bundle.version` is understood, all other unknown keys MAY be

751 +ignored by the client. The server will guarantee compatibility with older

752 +clients, though newer clients may be better able to use the extra keys to

753 +minimize downloads.

754 +

755 +Any backwards-incompatible addition of pre-URI key-value will be

756 +guarded by a new `bundle.version` value or values in 'bundle-uri'

757 +capability advertisement itself, and/or by new future `bundle-uri`

758 +request arguments.

759 +

760 +Some example key-value pairs that are not currently implemented but could

761 +be implemented in the future include:

762 +

763 + * Add a "hash=<val>" or "size=<bytes>" advertise the expected hash or

764 + size of the bundle file.

765 +

766 + * Advertise that one or more bundle files are the same (to e.g. have

767 + clients round-robin or otherwise choose one of N possible files).

768 +

769 + * A "oid=<OID>" shortcut and "prerequisite=<OID>" shortcut. For

770 + expressing the common case of a bundle with one tip and no

771 + prerequisites, or one tip and one prerequisite.

772 ++

773 +This would allow for optimizing the common case of servers who'd like

774 +to provide one "big bundle" containing only their "main" branch,

775 +and/or incremental updates thereof.

776 ++

777 +A client receiving such a a response MAY assume that they can skip

778 +retrieving the header from a bundle at the indicated URI, and thus

779 +save themselves and the server(s) the request(s) needed to inspect the

780 +headers of that bundle or bundles.

781 +

782 GIT

783 ---

784 Part of the linkgit:git[1] suite

bundle-uri.c

+36

@@ -563,6 +563,42 @@ cleanup:

563 return result;

564 }

565

566 +/**

567 + * API for serve.c.

568 + */

569 +

570 +int bundle_uri_advertise(struct repository *r, struct strbuf *value UNUSED)

571 +{

572 + static int advertise_bundle_uri = -1;

573 +

574 + if (advertise_bundle_uri != -1)

575 + goto cached;

576 +

577 + advertise_bundle_uri = 0;

578 + repo_config_get_maybe_bool(r, "uploadpack.advertisebundleuris", &advertise_bundle_uri);

579 +

580 +cached:

581 + return advertise_bundle_uri;

582 +}

583 +

584 +int bundle_uri_command(struct repository *r,

585 + struct packet_reader *request)

586 +{

587 + struct packet_writer writer;

588 + packet_writer_init(&writer, 1);

589 +

590 + while (packet_reader_read(request) == PACKET_READ_NORMAL)

591 + die(_("bundle-uri: unexpected argument: '%s'"), request->line);

592 + if (request->status != PACKET_READ_FLUSH)

593 + die(_("bundle-uri: expected flush after arguments"));

594 +

595 + /* TODO: Implement the communication */

596 +

597 + packet_writer_flush(&writer);

598 +

599 + return 0;

600 +}

601 +

602 /**

603 * General API for {transport,connect}.c etc.

604 */

bundle-uri.h

@@ -4,6 +4,7 @@

4 #include "hashmap.h"

5 #include "strbuf.h"

7 +struct packet_reader;

8 struct repository;

9 struct string_list;

@@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri,

93 */

94 int fetch_bundle_uri(struct repository *r, const char *uri);

96 +/**

97 + * API for serve.c.

98 + */

99 +int bundle_uri_advertise(struct repository *r, struct strbuf *value);

100 +int bundle_uri_command(struct repository *r, struct packet_reader *request);

101 +

102 /**

103 * General API for {transport,connect}.c etc.

104 */

serve.c

@@ -7,6 +7,7 @@

7 #include "protocol-caps.h"

8 #include "serve.h"

9 #include "upload-pack.h"

10 +#include "bundle-uri.h"

12 static int advertise_sid = -1;

13 static int client_hash_algo = GIT_HASH_SHA1;

@@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = {

136 .advertise = always_advertise,

137 .command = cap_object_info,

138 },

139 + {

140 + .name = "bundle-uri",

141 + .advertise = bundle_uri_advertise,

142 + .command = bundle_uri_command,

143 + },

144 };

145

146 void protocol_v2_advertise_capabilities(void)

t/t5701-git-serve.sh

+39 -1

@@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' '

13 wrong_algo sha1:sha256

14 wrong_algo sha256:sha1

15 EOF

16 - cat >expect <<-EOF &&

16 + cat >expect.base <<-EOF &&

17 version 2

18 agent=git/$(git version | cut -d" " -f3)

19 ls-refs=unborn

@@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' '

21 server-option

22 object-format=$(test_oid algo)

23 object-info

24 + EOF

25 + cat >expect.trailer <<-EOF &&

26 0000

27 EOF

28 + cat expect.base expect.trailer >expect &&

30 GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \

31 --advertise-capabilities >out &&

@@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' '

345 test_cmp expect actual

346 '

347

348 +test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' '

349 + test_config uploadpack.advertiseBundleURIs true &&

350 +

351 + cat >expect.extra <<-EOF &&

352 + bundle-uri

353 + EOF

354 + cat expect.base \

355 + expect.extra \

356 + expect.trailer >expect &&

357 +

358 + GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \

359 + --advertise-capabilities >out &&

360 + test-tool pkt-line unpack <out >actual &&

361 + test_cmp expect actual

362 +'

363 +

364 +test_expect_success 'basics of bundle-uri: dies if not enabled' '

365 + test-tool pkt-line pack >in <<-EOF &&

366 + command=bundle-uri

367 + 0000

368 + EOF

369 +

370 + cat >err.expect <<-\EOF &&

371 + fatal: invalid command '"'"'bundle-uri'"'"'

372 + EOF

373 +

374 + cat >expect <<-\EOF &&

375 + ERR serve: invalid command '"'"'bundle-uri'"'"'

376 + EOF

377 +

378 + test_must_fail test-tool serve-v2 --stateless-rpc <in >out 2>err.actual &&

379 + test_cmp err.expect err.actual &&

380 + test_must_be_empty out

381 +'

382 +

383 test_done