@matheusbsbs / matheus-bagliano / commits / ed10cb952d

serve: introduce git-serve

Introduce git-serve, the base server for protocol version 2. Protocol version 2 is intended to be a replacement for Git's current wire protocol. The intention is that it will be a simpler, less wasteful protocol which can evolve over time. Protocol version 2 improves upon version 1 by eliminating the initial ref advertisement. In its place a server will export a list of capabilities and commands which it supports in a capability advertisement. A client can then request that a particular command be executed by providing a number of capabilities and command specific parameters. At the completion of a command, a client can request that another command be executed or can terminate the connection by sending a flush packet. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Brandon Williams committed Mar 15, 2018 at 10:31 UTC ed10cb952d315ae47d2f7f6f29333a9ee227bf6b
10 files changed +528
.gitignore
+1
@@ -140,6 +140,7 @@
140 /git-rm
141 /git-send-email
142 /git-send-pack
143 +/git-serve
144 /git-sh-i18n
145 /git-sh-i18n--envsubst
146 /git-sh-setup
Documentation/Makefile
+1
@@ -77,6 +77,7 @@ TECH_DOCS += technical/pack-heuristics
77 TECH_DOCS += technical/pack-protocol
78 TECH_DOCS += technical/protocol-capabilities
79 TECH_DOCS += technical/protocol-common
80 +TECH_DOCS += technical/protocol-v2
81 TECH_DOCS += technical/racy-git
82 TECH_DOCS += technical/send-pack-pipeline
83 TECH_DOCS += technical/shallow
Documentation/technical/protocol-v2.txt new
+170
@@ -0,0 +1,170 @@
1 + Git Wire Protocol, Version 2
2 +==============================
3 +
4 +This document presents a specification for a version 2 of Git's wire
5 +protocol. Protocol v2 will improve upon v1 in the following ways:
6 +
7 + * Instead of multiple service names, multiple commands will be
8 + supported by a single service
9 + * Easily extendable as capabilities are moved into their own section
10 + of the protocol, no longer being hidden behind a NUL byte and
11 + limited by the size of a pkt-line
12 + * Separate out other information hidden behind NUL bytes (e.g. agent
13 + string as a capability and symrefs can be requested using 'ls-refs')
14 + * Reference advertisement will be omitted unless explicitly requested
15 + * ls-refs command to explicitly request some refs
16 + * Designed with http and stateless-rpc in mind. With clear flush
17 + semantics the http remote helper can simply act as a proxy
18 +
19 +In protocol v2 communication is command oriented. When first contacting a
20 +server a list of capabilities will advertised. Some of these capabilities
21 +will be commands which a client can request be executed. Once a command
22 +has completed, a client can reuse the connection and request that other
23 +commands be executed.
24 +
25 + Packet-Line Framing
26 +---------------------
27 +
28 +All communication is done using packet-line framing, just as in v1. See
29 +`Documentation/technical/pack-protocol.txt` and
30 +`Documentation/technical/protocol-common.txt` for more information.
31 +
32 +In protocol v2 these special packets will have the following semantics:
33 +
34 + * '0000' Flush Packet (flush-pkt) - indicates the end of a message
35 + * '0001' Delimiter Packet (delim-pkt) - separates sections of a message
36 +
37 + Initial Client Request
38 +------------------------
39 +
40 +In general a client can request to speak protocol v2 by sending
41 +`version=2` through the respective side-channel for the transport being
42 +used which inevitably sets `GIT_PROTOCOL`. More information can be
43 +found in `pack-protocol.txt` and `http-protocol.txt`. In all cases the
44 +response from the server is the capability advertisement.
45 +
46 + Git Transport
47 +~~~~~~~~~~~~~~~
48 +
49 +When using the git:// transport, you can request to use protocol v2 by
50 +sending "version=2" as an extra parameter:
51 +
52 + 003egit-upload-pack /project.git\0host=myserver.com\0\0version=2\0
53 +
54 + SSH and File Transport
55 +~~~~~~~~~~~~~~~~~~~~~~~~
56 +
57 +When using either the ssh:// or file:// transport, the GIT_PROTOCOL
58 +environment variable must be set explicitly to include "version=2".
59 +
60 + HTTP Transport
61 +~~~~~~~~~~~~~~~~
62 +
63 +When using the http:// or https:// transport a client makes a "smart"
64 +info/refs request as described in `http-protocol.txt` and requests that
65 +v2 be used by supplying "version=2" in the `Git-Protocol` header.
66 +
67 + C: Git-Protocol: version=2
68 + C:
69 + C: GET $GIT_URL/info/refs?service=git-upload-pack HTTP/1.0
70 +
71 +A v2 server would reply:
72 +
73 + S: 200 OK
74 + S: <Some headers>
75 + S: ...
76 + S:
77 + S: 000eversion 2\n
78 + S: <capability-advertisement>
79 +
80 +Subsequent requests are then made directly to the service
81 +`$GIT_URL/git-upload-pack`. (This works the same for git-receive-pack).
82 +
83 + Capability Advertisement
84 +--------------------------
85 +
86 +A server which decides to communicate (based on a request from a client)
87 +using protocol version 2, notifies the client by sending a version string
88 +in its initial response followed by an advertisement of its capabilities.
89 +Each capability is a key with an optional value. Clients must ignore all
90 +unknown keys. Semantics of unknown values are left to the definition of
91 +each key. Some capabilities will describe commands which can be requested
92 +to be executed by the client.
93 +
94 + capability-advertisement = protocol-version
95 + capability-list
96 + flush-pkt
97 +
98 + protocol-version = PKT-LINE("version 2" LF)
99 + capability-list = *capability
100 + capability = PKT-LINE(key[=value] LF)
101 +
102 + key = 1*(ALPHA | DIGIT | "-_")
103 + value = 1*(ALPHA | DIGIT | " -_.,?\/{}[]()<>!@#$%^&*+=:;")
104 +
105 + Command Request
106 +-----------------
107 +
108 +After receiving the capability advertisement, a client can then issue a
109 +request to select the command it wants with any particular capabilities
110 +or arguments. There is then an optional section where the client can
111 +provide any command specific parameters or queries. Only a single
112 +command can be requested at a time.
113 +
114 + request = empty-request | command-request
115 + empty-request = flush-pkt
116 + command-request = command
117 + capability-list
118 + [command-args]
119 + flush-pkt
120 + command = PKT-LINE("command=" key LF)
121 + command-args = delim-pkt
122 + *command-specific-arg
123 +
124 + command-specific-args are packet line framed arguments defined by
125 + each individual command.
126 +
127 +The server will then check to ensure that the client's request is
128 +comprised of a valid command as well as valid capabilities which were
129 +advertised. If the request is valid the server will then execute the
130 +command. A server MUST wait till it has received the client's entire
131 +request before issuing a response. The format of the response is
132 +determined by the command being executed, but in all cases a flush-pkt
133 +indicates the end of the response.
134 +
135 +When a command has finished, and the client has received the entire
136 +response from the server, a client can either request that another
137 +command be executed or can terminate the connection. A client may
138 +optionally send an empty request consisting of just a flush-pkt to
139 +indicate that no more requests will be made.
140 +
141 + Capabilities
142 +--------------
143 +
144 +There are two different types of capabilities: normal capabilities,
145 +which can be used to to convey information or alter the behavior of a
146 +request, and commands, which are the core actions that a client wants to
147 +perform (fetch, push, etc).
148 +
149 +Protocol version 2 is stateless by default. This means that all commands
150 +must only last a single round and be stateless from the perspective of the
151 +server side, unless the client has requested a capability indicating that
152 +state should be maintained by the server. Clients MUST NOT require state
153 +management on the server side in order to function correctly. This
154 +permits simple round-robin load-balancing on the server side, without
155 +needing to worry about state management.
156 +
157 + agent
158 +~~~~~~~
159 +
160 +The server can advertise the `agent` capability with a value `X` (in the
161 +form `agent=X`) to notify the client that the server is running version
162 +`X`. The client may optionally send its own agent string by including
163 +the `agent` capability with a value `Y` (in the form `agent=Y`) in its
164 +request to the server (but it MUST NOT do so if the server did not
165 +advertise the agent capability). The `X` and `Y` strings may contain any
166 +printable ASCII characters except space (i.e., the byte range 32 < x <
167 +127), and are typically of the form "package/version" (e.g.,
168 +"git/1.8.3.1"). The agent strings are purely informative for statistics
169 +and debugging purposes, and MUST NOT be used to programmatically assume
170 +the presence or absence of particular features.
Makefile
+2
@@ -876,6 +876,7 @@ LIB_OBJS += revision.o
876 LIB_OBJS += run-command.o
877 LIB_OBJS += send-pack.o
878 LIB_OBJS += sequencer.o
879 +LIB_OBJS += serve.o
880 LIB_OBJS += server-info.o
881 LIB_OBJS += setup.o
882 LIB_OBJS += sha1-array.o
@@ -1009,6 +1010,7 @@ BUILTIN_OBJS += builtin/rev-parse.o
1010 BUILTIN_OBJS += builtin/revert.o
1011 BUILTIN_OBJS += builtin/rm.o
1012 BUILTIN_OBJS += builtin/send-pack.o
1013 +BUILTIN_OBJS += builtin/serve.o
1014 BUILTIN_OBJS += builtin/shortlog.o
1015 BUILTIN_OBJS += builtin/show-branch.o
1016 BUILTIN_OBJS += builtin/show-ref.o
builtin.h
+1
@@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);
215 extern int cmd_revert(int argc, const char **argv, const char *prefix);
216 extern int cmd_rm(int argc, const char **argv, const char *prefix);
217 extern int cmd_send_pack(int argc, const char **argv, const char *prefix);
218 +extern int cmd_serve(int argc, const char **argv, const char *prefix);
219 extern int cmd_shortlog(int argc, const char **argv, const char *prefix);
220 extern int cmd_show(int argc, const char **argv, const char *prefix);
221 extern int cmd_show_branch(int argc, const char **argv, const char *prefix);
builtin/serve.c new
+30
@@ -0,0 +1,30 @@
1 +#include "cache.h"
2 +#include "builtin.h"
3 +#include "parse-options.h"
4 +#include "serve.h"
5 +
6 +static char const * const serve_usage[] = {
7 + N_("git serve [<options>]"),
8 + NULL
9 +};
10 +
11 +int cmd_serve(int argc, const char **argv, const char *prefix)
12 +{
13 + struct serve_options opts = SERVE_OPTIONS_INIT;
14 +
15 + struct option options[] = {
16 + OPT_BOOL(0, "stateless-rpc", &opts.stateless_rpc,
17 + N_("quit after a single request/response exchange")),
18 + OPT_BOOL(0, "advertise-capabilities", &opts.advertise_capabilities,
19 + N_("exit immediately after advertising capabilities")),
20 + OPT_END()
21 + };
22 +
23 + /* ignore all unknown cmdline switches for now */
24 + argc = parse_options(argc, argv, prefix, options, serve_usage,
25 + PARSE_OPT_KEEP_DASHDASH |
26 + PARSE_OPT_KEEP_UNKNOWN);
27 + serve(&opts);
28 +
29 + return 0;
30 +}
git.c
+1
@@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {
461 { "revert", cmd_revert, RUN_SETUP | NEED_WORK_TREE },
462 { "rm", cmd_rm, RUN_SETUP },
463 { "send-pack", cmd_send_pack, RUN_SETUP },
464 + { "serve", cmd_serve, RUN_SETUP },
465 { "shortlog", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },
466 { "show", cmd_show, RUN_SETUP },
467 { "show-branch", cmd_show_branch, RUN_SETUP },
serve.c new
+247
@@ -0,0 +1,247 @@
1 +#include "cache.h"
2 +#include "repository.h"
3 +#include "config.h"
4 +#include "pkt-line.h"
5 +#include "version.h"
6 +#include "argv-array.h"
7 +#include "serve.h"
8 +
9 +static int agent_advertise(struct repository *r,
10 + struct strbuf *value)
11 +{
12 + if (value)
13 + strbuf_addstr(value, git_user_agent_sanitized());
14 + return 1;
15 +}
16 +
17 +struct protocol_capability {
18 + /*
19 + * The name of the capability. The server uses this name when
20 + * advertising this capability, and the client uses this name to
21 + * specify this capability.
22 + */
23 + const char *name;
24 +
25 + /*
26 + * Function queried to see if a capability should be advertised.
27 + * Optionally a value can be specified by adding it to 'value'.
28 + * If a value is added to 'value', the server will advertise this
29 + * capability as "<name>=<value>" instead of "<name>".
30 + */
31 + int (*advertise)(struct repository *r, struct strbuf *value);
32 +
33 + /*
34 + * Function called when a client requests the capability as a command.
35 + * The function will be provided the capabilities requested via 'keys'
36 + * as well as a struct packet_reader 'request' which the command should
37 + * use to read the command specific part of the request. Every command
38 + * MUST read until a flush packet is seen before sending a response.
39 + *
40 + * This field should be NULL for capabilities which are not commands.
41 + */
42 + int (*command)(struct repository *r,
43 + struct argv_array *keys,
44 + struct packet_reader *request);
45 +};
46 +
47 +static struct protocol_capability capabilities[] = {
48 + { "agent", agent_advertise, NULL },
49 +};
50 +
51 +static void advertise_capabilities(void)
52 +{
53 + struct strbuf capability = STRBUF_INIT;
54 + struct strbuf value = STRBUF_INIT;
55 + int i;
56 +
57 + for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
58 + struct protocol_capability *c = &capabilities[i];
59 +
60 + if (c->advertise(the_repository, &value)) {
61 + strbuf_addstr(&capability, c->name);
62 +
63 + if (value.len) {
64 + strbuf_addch(&capability, '=');
65 + strbuf_addbuf(&capability, &value);
66 + }
67 +
68 + strbuf_addch(&capability, '\n');
69 + packet_write(1, capability.buf, capability.len);
70 + }
71 +
72 + strbuf_reset(&capability);
73 + strbuf_reset(&value);
74 + }
75 +
76 + packet_flush(1);
77 + strbuf_release(&capability);
78 + strbuf_release(&value);
79 +}
80 +
81 +static struct protocol_capability *get_capability(const char *key)
82 +{
83 + int i;
84 +
85 + if (!key)
86 + return NULL;
87 +
88 + for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
89 + struct protocol_capability *c = &capabilities[i];
90 + const char *out;
91 + if (skip_prefix(key, c->name, &out) && (!*out || *out == '='))
92 + return c;
93 + }
94 +
95 + return NULL;
96 +}
97 +
98 +static int is_valid_capability(const char *key)
99 +{
100 + const struct protocol_capability *c = get_capability(key);
101 +
102 + return c && c->advertise(the_repository, NULL);
103 +}
104 +
105 +static int is_command(const char *key, struct protocol_capability **command)
106 +{
107 + const char *out;
108 +
109 + if (skip_prefix(key, "command=", &out)) {
110 + struct protocol_capability *cmd = get_capability(out);
111 +
112 + if (*command)
113 + die("command '%s' requested after already requesting command '%s'",
114 + out, (*command)->name);
115 + if (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)
116 + die("invalid command '%s'", out);
117 +
118 + *command = cmd;
119 + return 1;
120 + }
121 +
122 + return 0;
123 +}
124 +
125 +int has_capability(const struct argv_array *keys, const char *capability,
126 + const char **value)
127 +{
128 + int i;
129 + for (i = 0; i < keys->argc; i++) {
130 + const char *out;
131 + if (skip_prefix(keys->argv[i], capability, &out) &&
132 + (!*out || *out == '=')) {
133 + if (value) {
134 + if (*out == '=')
135 + out++;
136 + *value = out;
137 + }
138 + return 1;
139 + }
140 + }
141 +
142 + return 0;
143 +}
144 +
145 +enum request_state {
146 + PROCESS_REQUEST_KEYS,
147 + PROCESS_REQUEST_DONE,
148 +};
149 +
150 +static int process_request(void)
151 +{
152 + enum request_state state = PROCESS_REQUEST_KEYS;
153 + struct packet_reader reader;
154 + struct argv_array keys = ARGV_ARRAY_INIT;
155 + struct protocol_capability *command = NULL;
156 +
157 + packet_reader_init(&reader, 0, NULL, 0,
158 + PACKET_READ_CHOMP_NEWLINE |
159 + PACKET_READ_GENTLE_ON_EOF);
160 +
161 + /*
162 + * Check to see if the client closed their end before sending another
163 + * request. If so we can terminate the connection.
164 + */
165 + if (packet_reader_peek(&reader) == PACKET_READ_EOF)
166 + return 1;
167 + reader.options = PACKET_READ_CHOMP_NEWLINE;
168 +
169 + while (state != PROCESS_REQUEST_DONE) {
170 + switch (packet_reader_peek(&reader)) {
171 + case PACKET_READ_EOF:
172 + BUG("Should have already died when seeing EOF");
173 + case PACKET_READ_NORMAL:
174 + /* collect request; a sequence of keys and values */
175 + if (is_command(reader.line, &command) ||
176 + is_valid_capability(reader.line))
177 + argv_array_push(&keys, reader.line);
178 + else
179 + die("unknown capability '%s'", reader.line);
180 +
181 + /* Consume the peeked line */
182 + packet_reader_read(&reader);
183 + break;
184 + case PACKET_READ_FLUSH:
185 + /*
186 + * If no command and no keys were given then the client
187 + * wanted to terminate the connection.
188 + */
189 + if (!keys.argc)
190 + return 1;
191 +
192 + /*
193 + * The flush packet isn't consume here like it is in
194 + * the other parts of this switch statement. This is
195 + * so that the command can read the flush packet and
196 + * see the end of the request in the same way it would
197 + * if command specific arguments were provided after a
198 + * delim packet.
199 + */
200 + state = PROCESS_REQUEST_DONE;
201 + break;
202 + case PACKET_READ_DELIM:
203 + /* Consume the peeked line */
204 + packet_reader_read(&reader);
205 +
206 + state = PROCESS_REQUEST_DONE;
207 + break;
208 + }
209 + }
210 +
211 + if (!command)
212 + die("no command requested");
213 +
214 + command->command(the_repository, &keys, &reader);
215 +
216 + argv_array_clear(&keys);
217 + return 0;
218 +}
219 +
220 +/* Main serve loop for protocol version 2 */
221 +void serve(struct serve_options *options)
222 +{
223 + if (options->advertise_capabilities || !options->stateless_rpc) {
224 + /* serve by default supports v2 */
225 + packet_write_fmt(1, "version 2\n");
226 +
227 + advertise_capabilities();
228 + /*
229 + * If only the list of capabilities was requested exit
230 + * immediately after advertising capabilities
231 + */
232 + if (options->advertise_capabilities)
233 + return;
234 + }
235 +
236 + /*
237 + * If stateless-rpc was requested then exit after
238 + * a single request/response exchange
239 + */
240 + if (options->stateless_rpc) {
241 + process_request();
242 + } else {
243 + for (;;)
244 + if (process_request())
245 + break;
246 + }
247 +}
serve.h new
+15
@@ -0,0 +1,15 @@
1 +#ifndef SERVE_H
2 +#define SERVE_H
3 +
4 +struct argv_array;
5 +extern int has_capability(const struct argv_array *keys, const char *capability,
6 + const char **value);
7 +
8 +struct serve_options {
9 + unsigned advertise_capabilities;
10 + unsigned stateless_rpc;
11 +};
12 +#define SERVE_OPTIONS_INIT { 0 }
13 +extern void serve(struct serve_options *options);
14 +
15 +#endif /* SERVE_H */
t/t5701-git-serve.sh new
+60
@@ -0,0 +1,60 @@
1 +#!/bin/sh
2 +
3 +test_description='test git-serve and server commands'
4 +
5 +. ./test-lib.sh
6 +
7 +test_expect_success 'test capability advertisement' '
8 + cat >expect <<-EOF &&
9 + version 2
10 + agent=git/$(git version | cut -d" " -f3)
11 + 0000
12 + EOF
13 +
14 + git serve --advertise-capabilities >out &&
15 + test-pkt-line unpack <out >actual &&
16 + test_cmp actual expect
17 +'
18 +
19 +test_expect_success 'stateless-rpc flag does not list capabilities' '
20 + # Empty request
21 + test-pkt-line pack >in <<-EOF &&
22 + 0000
23 + EOF
24 + git serve --stateless-rpc >out <in &&
25 + test_must_be_empty out &&
26 +
27 + # EOF
28 + git serve --stateless-rpc >out &&
29 + test_must_be_empty out
30 +'
31 +
32 +test_expect_success 'request invalid capability' '
33 + test-pkt-line pack >in <<-EOF &&
34 + foobar
35 + 0000
36 + EOF
37 + test_must_fail git serve --stateless-rpc 2>err <in &&
38 + test_i18ngrep "unknown capability" err
39 +'
40 +
41 +test_expect_success 'request with no command' '
42 + test-pkt-line pack >in <<-EOF &&
43 + agent=git/test
44 + 0000
45 + EOF
46 + test_must_fail git serve --stateless-rpc 2>err <in &&
47 + test_i18ngrep "no command requested" err
48 +'
49 +
50 +test_expect_success 'request invalid command' '
51 + test-pkt-line pack >in <<-EOF &&
52 + command=foo
53 + agent=git/test
54 + 0000
55 + EOF
56 + test_must_fail git serve --stateless-rpc 2>err <in &&
57 + test_i18ngrep "invalid command" err
58 +'
59 +
60 +test_done