@matheusbsbs / matheus-bagliano / commits / 0d0bac67ce

transport: drop support for git-over-rsync

The git-over-rsync protocol is inefficient and broken, and has been for a long time. It transfers way more objects than it needs (grabbing all of the remote's "objects/", regardless of which objects we need). It does its own ad-hoc parsing of loose and packed refs from the remote, but doesn't properly override packed refs with loose ones, leading to garbage results (e.g., expecting the other side to have an object pointed to by a stale packed-refs entry, or complaining that the other side has two copies of the refs[1]). This latter breakage means that nobody could have successfully pulled from a moderately active repository since cd547b4 (fetch/push: readd rsync support, 2007-10-01). We never made an official deprecation notice in the release notes for git's rsync protocol, but the tutorial has marked it as such since 914328a (Update tutorial., 2005-08-30). And on the mailing list as far back as Oct 2005, we can find Junio mentioning it as having "been deprecated for quite some time."[2,3,4]. So it was old news then; cogito had deprecated the transport in July of 2005[5] (though it did come back briefly when Linus broke git-http-pull!). Of course some people professed their love of rsync through 2006, but Linus clarified in his usual gentle manner[6]: > Thanks! This is why I still use rsync, even though > everybody and their mother tells me "Linus says rsync is > deprecated." No. You're using rsync because you're actively doing something _wrong_. The deprecation sentiment was reinforced in 2008, with a mention that cloning via rsync is broken (with no fix)[7]. Even the commit porting rsync over to C from shell (cd547b4) lists it as deprecated! So between the 10 years of informal warnings, and the fact that it has been severely broken since 2007, it's probably safe to simply remove it without further deprecation warnings. [1] http://article.gmane.org/gmane.comp.version-control.git/285101 [2] http://article.gmane.org/gmane.comp.version-control.git/10093 [3] http://article.gmane.org/gmane.comp.version-control.git/17734 [4] http://article.gmane.org/gmane.comp.version-control.git/18911 [5] http://article.gmane.org/gmane.comp.version-control.git/5617 [6] http://article.gmane.org/gmane.comp.version-control.git/19354 [7] http://article.gmane.org/gmane.comp.version-control.git/103635 Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Jeff King committed Jan 30, 2016 at 02:21 UTC 0d0bac67ce3b3f2301702573f6acc100798d7edd

10 files changed +10 -395

Documentation/config.txt

+1 -1

@@ -2074,7 +2074,7 @@ pack.indexVersion::

2074 larger than 2 GB.

2075 +

2076 If you have an old Git that does not understand the version 2 `*.idx` file,

2077 -cloning or fetching over a non native protocol (e.g. "http" and "rsync")

2077 +cloning or fetching over a non native protocol (e.g. "http")

2078 that will copy both `*.pack` file and corresponding `*.idx` file from the

2079 other side may give you a repository that cannot be accessed with your

2080 older version of Git. If the `*.pack` file is smaller than 2 GB, however,

Documentation/git-bundle.txt

+1 -1

@@ -20,7 +20,7 @@ DESCRIPTION

20 Some workflows require that one or more branches of development on one

21 machine be replicated on another machine, but the two machines cannot

22 be directly connected, and therefore the interactive Git protocols (git,

23 -ssh, rsync, http) cannot be used. This command provides support for

23 +ssh, http) cannot be used. This command provides support for

24 'git fetch' and 'git pull' to operate by packaging objects and references

25 in an archive at the originating machine, then importing those into

26 another repository using 'git fetch' and 'git pull'

Documentation/git-clone.txt

+1 -2

@@ -115,8 +115,7 @@ objects from the source repository into a pack in the cloned repository.

115 --quiet::

116 -q::

117 Operate quietly. Progress is not reported to the standard

118 - error stream. This flag is also passed to the `rsync'

119 - command when given.

118 + error stream.

119

120 --verbose::

121 -v::

Documentation/git-repack.txt

+1 -1

@@ -133,7 +133,7 @@ By default, the command passes `--delta-base-offset` option to

133 'git pack-objects'; this typically results in slightly smaller packs,

134 but the generated packs are incompatible with versions of Git older than

135 version 1.4.4. If you need to share your repository with such ancient Git

136 -versions, either directly or via the dumb http or rsync protocol, then you

136 +versions, either directly or via the dumb http protocol, then you

137 need to set the configuration variable `repack.UseDeltaBaseOffset` to

138 "false" and repack. Access from old Git versions over the native protocol

139 is unaffected by this option as the conversion is performed on the fly

Documentation/git.txt

-2

@@ -1123,8 +1123,6 @@ of clones and fetches.

1123 - `ssh`: git over ssh (including `host:path` syntax,

1124 `git+ssh://`, etc).

1125

1126 - - `rsync`: git over rsync

1127 -

1126 - `http`: git over http, both "smart http" and "dumb http".

1127 Note that this does _not_ include `https`; if you want both,

1128 you should specify both as `http:https`.

Documentation/gitcore-tutorial.txt

+2 -16

@@ -710,7 +710,7 @@ files).

710 Again, this can all be simplified with

711

712 ----------------

713 -$ git clone rsync://rsync.kernel.org/pub/scm/git/git.git/ my-git

713 +$ git clone git://git.kernel.org/pub/scm/git/git.git/ my-git

714 $ cd my-git

715 $ git checkout

716 ----------------

@@ -1011,20 +1011,6 @@ $ git fetch <remote-repository>

1011 One of the following transports can be used to name the

1012 repository to download from:

1013

1014 -Rsync::

1015 - `rsync://remote.machine/path/to/repo.git/`

1016 -+

1017 -Rsync transport is usable for both uploading and downloading,

1018 -but is completely unaware of what git does, and can produce

1019 -unexpected results when you download from the public repository

1020 -while the repository owner is uploading into it via `rsync`

1021 -transport. Most notably, it could update the files under

1022 -`refs/` which holds the object name of the topmost commits

1023 -before uploading the files in `objects/` -- the downloader would

1024 -obtain head commit object name while that object itself is still

1025 -not available in the repository. For this reason, it is

1026 -considered deprecated.

1027 -

1014 SSH::

1015 `remote.machine:/path/to/repo.git/` or

1016 +

@@ -1430,7 +1416,7 @@ while, depending on how active your project is.

1416

1417 When a repository is synchronized via `git push` and `git pull`

1418 objects packed in the source repository are usually stored

1433 -unpacked in the destination, unless rsync transport is used.

1419 +unpacked in the destination.

1420 While this allows you to use different packing strategies on

1421 both ends, it also means you may need to repack both

1422 repositories every once in a while.

Documentation/gittutorial.txt

+1 -1

@@ -451,7 +451,7 @@ perform clones and pulls using the ssh protocol:

451 bob$ git clone alice.org:/home/alice/project myrepo

452 -------------------------------------

453

454 -Alternatively, Git has a native protocol, or can use rsync or http;

454 +Alternatively, Git has a native protocol, or can use http;

455 see linkgit:git-pull[1] for details.

456

457 Git can also be used in a CVS-like mode, with a central repository

Documentation/urls.txt

+2 -4

@@ -7,9 +7,8 @@ Depending on the transport protocol, some of this information may be

7 absent.

9 Git supports ssh, git, http, and https protocols (in addition, ftp,

10 -and ftps can be used for fetching and rsync can be used for fetching

11 -and pushing, but these are inefficient and deprecated; do not use

12 -them).

10 +and ftps can be used for fetching, but this is inefficient and

11 +deprecated; do not use it).

13 The native transport (i.e. git:// URL) does no authentication and

14 should be used with caution on unsecured networks.

@@ -20,7 +19,6 @@ The following syntaxes may be used with them:

19 - git://host.xz{startsb}:port{endsb}/path/to/repo.git/

20 - http{startsb}s{endsb}://host.xz{startsb}:port{endsb}/path/to/repo.git/

21 - ftp{startsb}s{endsb}://host.xz{startsb}:port{endsb}/path/to/repo.git/

23 -- rsync://host.xz/path/to/repo.git/

23 An alternative scp-like syntax may also be used with the ssh protocol:

t/t5510-fetch.sh

-36

@@ -314,42 +314,6 @@ test_expect_success 'bundle should be able to create a full history' '

314

315 '

316

317 -! rsync --help > /dev/null 2> /dev/null &&

318 -say 'Skipping rsync tests because rsync was not found' || {

319 -test_expect_success 'fetch via rsync' '

320 - git pack-refs &&

321 - mkdir rsynced &&

322 - (cd rsynced &&

323 - git init --bare &&

324 - git fetch "rsync:../.git" master:refs/heads/master &&

325 - git gc --prune &&

326 - test $(git rev-parse master) = $(cd .. && git rev-parse master) &&

327 - git fsck --full)

328 -'

329 -

330 -test_expect_success 'push via rsync' '

331 - mkdir rsynced2 &&

332 - (cd rsynced2 &&

333 - git init) &&

334 - (cd rsynced &&

335 - git push "rsync:../rsynced2/.git" master) &&

336 - (cd rsynced2 &&

337 - git gc --prune &&

338 - test $(git rev-parse master) = $(cd .. && git rev-parse master) &&

339 - git fsck --full)

340 -'

341 -

342 -test_expect_success 'push via rsync' '

343 - mkdir rsynced3 &&

344 - (cd rsynced3 &&

345 - git init) &&

346 - git push --all "rsync:rsynced3/.git" &&

347 - (cd rsynced3 &&

348 - test $(git rev-parse master) = $(cd .. && git rev-parse master) &&

349 - git fsck --full)

350 -'

351 -}

352 -

317 test_expect_success 'fetch with a non-applying branch.<name>.merge' '

318 git config branch.master.remote yeti &&

319 git config branch.master.merge refs/heads/bigfoot &&

transport.c

+1 -331

@@ -17,133 +17,6 @@

17 #include "sha1-array.h"

18 #include "sigchain.h"

20 -/* rsync support */

21 -

22 -/*

23 - * We copy packed-refs and refs/ into a temporary file, then read the

24 - * loose refs recursively (sorting whenever possible), and then inserting

25 - * those packed refs that are not yet in the list (not validating, but

26 - * assuming that the file is sorted).

27 - *

28 - * Appears refactoring this from refs.c is too cumbersome.

29 - */

30 -

31 -static int str_cmp(const void *a, const void *b)

32 -{

33 - const char *s1 = a;

34 - const char *s2 = b;

35 -

36 - return strcmp(s1, s2);

37 -}

38 -

39 -/* path->buf + name_offset is expected to point to "refs/" */

40 -

41 -static int read_loose_refs(struct strbuf *path, int name_offset,

42 - struct ref **tail)

43 -{

44 - DIR *dir = opendir(path->buf);

45 - struct dirent *de;

46 - struct {

47 - char **entries;

48 - int nr, alloc;

49 - } list;

50 - int i, pathlen;

51 -

52 - if (!dir)

53 - return -1;

54 -

55 - memset (&list, 0, sizeof(list));

56 -

57 - while ((de = readdir(dir))) {

58 - if (is_dot_or_dotdot(de->d_name))

59 - continue;

60 - ALLOC_GROW(list.entries, list.nr + 1, list.alloc);

61 - list.entries[list.nr++] = xstrdup(de->d_name);

62 - }

63 - closedir(dir);

64 -

65 - /* sort the list */

66 -

67 - qsort(list.entries, list.nr, sizeof(char *), str_cmp);

68 -

69 - pathlen = path->len;

70 - strbuf_addch(path, '/');

71 -

72 - for (i = 0; i < list.nr; i++, strbuf_setlen(path, pathlen + 1)) {

73 - strbuf_addstr(path, list.entries[i]);

74 - if (read_loose_refs(path, name_offset, tail)) {

75 - int fd = open(path->buf, O_RDONLY);

76 - char buffer[40];

77 - struct ref *next;

78 -

79 - if (fd < 0)

80 - continue;

81 - next = alloc_ref(path->buf + name_offset);

82 - if (read_in_full(fd, buffer, 40) != 40 ||

83 - get_oid_hex(buffer, &next->old_oid)) {

84 - close(fd);

85 - free(next);

86 - continue;

87 - }

88 - close(fd);

89 - (*tail)->next = next;

90 - *tail = next;

91 - }

92 - }

93 - strbuf_setlen(path, pathlen);

94 -

95 - for (i = 0; i < list.nr; i++)

96 - free(list.entries[i]);

97 - free(list.entries);

98 -

99 - return 0;

100 -}

101 -

102 -/* insert the packed refs for which no loose refs were found */

103 -

104 -static void insert_packed_refs(const char *packed_refs, struct ref **list)

105 -{

106 - FILE *f = fopen(packed_refs, "r");

107 - static char buffer[PATH_MAX];

108 -

109 - if (!f)

110 - return;

111 -

112 - for (;;) {

113 - int cmp = 0; /* assigned before used */

114 - int len;

115 -

116 - if (!fgets(buffer, sizeof(buffer), f)) {

117 - fclose(f);

118 - return;

119 - }

120 -

121 - if (!isxdigit(buffer[0]))

122 - continue;

123 - len = strlen(buffer);

124 - if (len && buffer[len - 1] == '\n')

125 - buffer[--len] = '\0';

126 - if (len < 41)

127 - continue;

128 - while ((*list)->next &&

129 - (cmp = strcmp(buffer + 41,

130 - (*list)->next->name)) > 0)

131 - list = &(*list)->next;

132 - if (!(*list)->next || cmp < 0) {

133 - struct ref *next = alloc_ref(buffer + 41);

134 - buffer[40] = '\0';

135 - if (get_oid_hex(buffer, &next->old_oid)) {

136 - warning ("invalid SHA-1: %s", buffer);

137 - free(next);

138 - continue;

139 - }

140 - next->next = (*list)->next;

141 - (*list)->next = next;

142 - list = &(*list)->next;

143 - }

144 - }

145 -}

146 -

20 static void set_upstreams(struct transport *transport, struct ref *refs,

21 int pretend)

22 {

@@ -192,205 +65,6 @@ static void set_upstreams(struct transport *transport, struct ref *refs,

65 }

66 }

195 -static const char *rsync_url(const char *url)

196 -{

197 - if (!starts_with(url, "rsync://"))

198 - skip_prefix(url, "rsync:", &url);

199 - return url;

200 -}

201 -

202 -static struct ref *get_refs_via_rsync(struct transport *transport, int for_push)

203 -{

204 - struct strbuf buf = STRBUF_INIT, temp_dir = STRBUF_INIT;

205 - struct ref dummy = {NULL}, *tail = &dummy;

206 - struct child_process rsync = CHILD_PROCESS_INIT;

207 - const char *args[5];

208 - int temp_dir_len;

209 -

210 - if (for_push)

211 - return NULL;

212 -

213 - /* copy the refs to the temporary directory */

214 -

215 - strbuf_addstr(&temp_dir, git_path("rsync-refs-XXXXXX"));

216 - if (!mkdtemp(temp_dir.buf))

217 - die_errno ("Could not make temporary directory");

218 - temp_dir_len = temp_dir.len;

219 -

220 - strbuf_addstr(&buf, rsync_url(transport->url));

221 - strbuf_addstr(&buf, "/refs");

222 -

223 - rsync.argv = args;

224 - rsync.stdout_to_stderr = 1;

225 - args[0] = "rsync";

226 - args[1] = (transport->verbose > 1) ? "-rv" : "-r";

227 - args[2] = buf.buf;

228 - args[3] = temp_dir.buf;

229 - args[4] = NULL;

230 -

231 - if (run_command(&rsync))

232 - die ("Could not run rsync to get refs");

233 -

234 - strbuf_reset(&buf);

235 - strbuf_addstr(&buf, rsync_url(transport->url));

236 - strbuf_addstr(&buf, "/packed-refs");

237 -

238 - args[2] = buf.buf;

239 -

240 - if (run_command(&rsync))

241 - die ("Could not run rsync to get refs");

242 -

243 - /* read the copied refs */

244 -

245 - strbuf_addstr(&temp_dir, "/refs");

246 - read_loose_refs(&temp_dir, temp_dir_len + 1, &tail);

247 - strbuf_setlen(&temp_dir, temp_dir_len);

248 -

249 - tail = &dummy;

250 - strbuf_addstr(&temp_dir, "/packed-refs");

251 - insert_packed_refs(temp_dir.buf, &tail);

252 - strbuf_setlen(&temp_dir, temp_dir_len);

253 -

254 - if (remove_dir_recursively(&temp_dir, 0))

255 - warning ("Error removing temporary directory %s.",

256 - temp_dir.buf);

257 -

258 - strbuf_release(&buf);

259 - strbuf_release(&temp_dir);

260 -

261 - return dummy.next;

262 -}

263 -

264 -static int fetch_objs_via_rsync(struct transport *transport,

265 - int nr_objs, struct ref **to_fetch)

266 -{

267 - struct child_process rsync = CHILD_PROCESS_INIT;

268 -

269 - rsync.stdout_to_stderr = 1;

270 - argv_array_push(&rsync.args, "rsync");

271 - argv_array_push(&rsync.args, (transport->verbose > 1) ? "-rv" : "-r");

272 - argv_array_push(&rsync.args, "--ignore-existing");

273 - argv_array_push(&rsync.args, "--exclude");

274 - argv_array_push(&rsync.args, "info");

275 - argv_array_pushf(&rsync.args, "%s/objects/", rsync_url(transport->url));

276 - argv_array_push(&rsync.args, get_object_directory());

277 -

278 - /* NEEDSWORK: handle one level of alternates */

279 - return run_command(&rsync);

280 -}

281 -

282 -static int write_one_ref(const char *name, const struct object_id *oid,

283 - int flags, void *data)

284 -{

285 - struct strbuf *buf = data;

286 - int len = buf->len;

287 -

288 - /* when called via for_each_ref(), flags is non-zero */

289 - if (flags && !starts_with(name, "refs/heads/") &&

290 - !starts_with(name, "refs/tags/"))

291 - return 0;

292 -

293 - strbuf_addstr(buf, name);

294 - if (safe_create_leading_directories(buf->buf) ||

295 - write_file_gently(buf->buf, "%s", oid_to_hex(oid)))

296 - return error("problems writing temporary file %s: %s",

297 - buf->buf, strerror(errno));

298 - strbuf_setlen(buf, len);

299 - return 0;

300 -}

301 -

302 -static int write_refs_to_temp_dir(struct strbuf *temp_dir,

303 - int refspec_nr, const char **refspec)

304 -{

305 - int i;

306 -

307 - for (i = 0; i < refspec_nr; i++) {

308 - struct object_id oid;

309 - char *ref;

310 -

311 - if (dwim_ref(refspec[i], strlen(refspec[i]), oid.hash, &ref) != 1)

312 - return error("Could not get ref %s", refspec[i]);

313 -

314 - if (write_one_ref(ref, &oid, 0, temp_dir)) {

315 - free(ref);

316 - return -1;

317 - }

318 - free(ref);

319 - }

320 - return 0;

321 -}

322 -

323 -static int rsync_transport_push(struct transport *transport,

324 - int refspec_nr, const char **refspec, int flags)

325 -{

326 - struct strbuf buf = STRBUF_INIT, temp_dir = STRBUF_INIT;

327 - int result = 0, i;

328 - struct child_process rsync = CHILD_PROCESS_INIT;

329 - const char *args[10];

330 -

331 - if (flags & TRANSPORT_PUSH_MIRROR)

332 - return error("rsync transport does not support mirror mode");

333 -

334 - /* first push the objects */

335 -

336 - strbuf_addstr(&buf, rsync_url(transport->url));

337 - strbuf_addch(&buf, '/');

338 -

339 - rsync.argv = args;

340 - rsync.stdout_to_stderr = 1;

341 - i = 0;

342 - args[i++] = "rsync";

343 - args[i++] = "-a";

344 - if (flags & TRANSPORT_PUSH_DRY_RUN)

345 - args[i++] = "--dry-run";

346 - if (transport->verbose > 1)

347 - args[i++] = "-v";

348 - args[i++] = "--ignore-existing";

349 - args[i++] = "--exclude";

350 - args[i++] = "info";

351 - args[i++] = get_object_directory();

352 - args[i++] = buf.buf;

353 - args[i++] = NULL;

354 -

355 - if (run_command(&rsync))

356 - return error("Could not push objects to %s",

357 - rsync_url(transport->url));

358 -

359 - /* copy the refs to the temporary directory; they could be packed. */

360 -

361 - strbuf_addstr(&temp_dir, git_path("rsync-refs-XXXXXX"));

362 - if (!mkdtemp(temp_dir.buf))

363 - die_errno ("Could not make temporary directory");

364 - strbuf_addch(&temp_dir, '/');

365 -

366 - if (flags & TRANSPORT_PUSH_ALL) {

367 - if (for_each_ref(write_one_ref, &temp_dir))

368 - return -1;

369 - } else if (write_refs_to_temp_dir(&temp_dir, refspec_nr, refspec))

370 - return -1;

371 -

372 - i = 2;

373 - if (flags & TRANSPORT_PUSH_DRY_RUN)

374 - args[i++] = "--dry-run";

375 - if (!(flags & TRANSPORT_PUSH_FORCE))

376 - args[i++] = "--ignore-existing";

377 - args[i++] = temp_dir.buf;

378 - args[i++] = rsync_url(transport->url);

379 - args[i++] = NULL;

380 - if (run_command(&rsync))

381 - result = error("Could not push to %s",

382 - rsync_url(transport->url));

383 -

384 - if (remove_dir_recursively(&temp_dir, 0))

385 - warning ("Could not remove temporary directory %s.",

386 - temp_dir.buf);

387 -

388 - strbuf_release(&buf);

389 - strbuf_release(&temp_dir);

390 -

391 - return result;

392 -}

393 -

68 struct bundle_transport_data {

69 int fd;

70 struct bundle_header header;

@@ -984,11 +658,7 @@ struct transport *transport_get(struct remote *remote, const char *url)

658 if (helper) {

659 transport_helper_init(ret, helper);

660 } else if (starts_with(url, "rsync:")) {

987 - transport_check_allowed("rsync");

988 - ret->get_refs_list = get_refs_via_rsync;

989 - ret->fetch = fetch_objs_via_rsync;

990 - ret->push = rsync_transport_push;

991 - ret->smart_options = NULL;

661 + die("git-over-rsync is no longer supported");

662 } else if (url_is_local_not_ssh(url) && is_file(url) && is_bundle(url, 1)) {

663 struct bundle_transport_data *data = xcalloc(1, sizeof(*data));

664 transport_check_allowed("file");